RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch
Authors: Yiqin Tan
Pihe Hu
Ling Pan
Jiatai Huang
Longbo Huang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Deep Reinforcement Learning
Lottery Ticket Hypothesis
Model Compression
Value Learning

Abstract: Training deep reinforcement learning (DRL) models usually requires high computation costs. Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment. However, existing methods that generate small models mainly adopt the knowledge distillation-based approach by iteratively training a dense network. As a result, the training process still demands massive computing resources. Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity in bootstrap training. In this work, we propose a novel sparse DRL training framework, “the Rigged Reinforcement Learning Lottery” (RLx2), which builds upon gradient-based topology evolution and is capable of training a sparse DRL model based entirely on a sparse network. Specifically, RLx2 introduces a novel multi-step TD target mechanism with a dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration in sparse models. It also reaches state-of-the-art sparse training performance in several tasks, showing $7.5\times$-$20\times$ model compression with less than $3\%$ performance degradation and up to $20\times$ and $50\times$ FLOPs reduction for training and inference, respectively.

Published as a conference paper at ICLR 2023
RLx2: TRAINING A SPARSE DEEP REINFORCEMENT LEARNING MODEL FROM SCRATCH
Yigin Tan", Pihe Hu", Ling Pan, Jiatai Huang, Longbo Huang! Institute for Interdisciplinary Institute for Interdisciplinary Information Sciences Tsinghua University, Beijing, China {tyq22, hph1 @ma 1s. ts nnghua www.wa.ed..co longbohuarggessinghue .dd...c.eeddcccmo
ABSTRACT
Training deep reinforcement learning (DRL) models usually requires high com- putation costs. Therefore, compressing DRL models possesses immense potential for training acceleration and model deployment. However, existing methods that generate small models mainly adopt the knowledge iteratively training a dense network. As a result, the training process still demands massive computing resources. Indeed, sparse training from scratch in DRL has not been well explored and is particularly challenging due to non-stationarity bootstrap training. In this work, propose novel sparse DRL training frame- work, "the Rigged Reinforcement Learning Lottery" (RLx2), which builds upon gradient-based topology evolution and is capable of training DRL model based entirely on sparse networks. Specifically, RLx2 introduces a novel delayed multi- step TD target mechanism with dynamic-capacity replay buffer achieve robust value learning and efficient topology exploration in sparse models. It also reaches state-of-ther oftthh--tttttttttt sparse training performance in several tasks, showing x model compression with less than 3% performance degradation and up to 20 and 50x FLOPs reduction for training and inference, respectively.
ditilllation-baaee approach by
INTRODUCTION
Deep reinforcement learning (DRL) has found successful upplications in many important areas, c.g., games (Silver et al. 2017), robotics(Gu et al. 2017) and nuclear fusion (Degrave et al., 2022). How- ever, training DRL model demands heavy computational resources. For instance, AlphaGo-Zero for Go games (Silver et al. 2017). which defeats all Go-AIs and human experts, requires more than 40 days of training time on four tensor processing units (TPUs). The heavy resource requirement results in expensive consumption and hinders the application of DRL on resource- resource-limited devices.
Sparse networks, initially proposed in deep supervised learning, have demonstrated great potential for model compression and training acceleration of deep reinforcement learning. Specifically, in deep supervised learning, the state-of-the-an sparse training frameworks, e.g.. SET (Mocanu al.) 2018) and Rigl. (Evci et al. 2020), can train a 90%-sparse network (i.e., the resulting network size 10% of the original network) from scratch without performance degradation. On the DRL. side, existing works including [uusu et al. (2016); [Schmitt et al. (2018); Zhang et al. (2019) suc- ceeded generating ultimately sparse DRL networks. Yet, their approaches still require iteratively training dense networks, e.g., pre-trained dense teachers may be needed. As result, the training cost for DRL remains prohibitively high, and existing methods cannot be directly implemented on resource-limited devices, leading to low flexibility in adapting the compressed DRL models to new environments, i.e., on-device models have to be retrained at large servers and re-deployed.
Training sparse DRL model from seratch, if done perfectly, has the potential significantly duce computation expenditure and enable efficient deployment on resource-limited devices, and achieves excellent flexibility
model adaptation. However, training an ultra sparse network (e.g., 90% sparsity) from scratch in DRL is challenging due to the non-stationarity in bootstrap train- ing. Specifically, in DRL, the learning target not fixed but evolves in a bootstrap way (Tesauro
"Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2023
et al. 1995), and the distribution of the training data can also be non-stationary (Desai al. 2019) Moreover, using a sparse network structure means searching in a smaller hypothesis space, which further reduces the learning target's confidence. As result, improper sparsification cause irre- versible damage to the learning path (Igl et al. 2021), resulting in poor performance. Indeed, recent works (Sokar et al.) 202T; Graesser al. 2022) show that direct adoption of dynamic sparse training (DST) framework in DRL still fails to achieve good compression of the model for different environments uniformly. Therefore, the following interesting question remains open:
Can an efficient DRL agent be trained from scratch with ultra-sparse network throughout?
In this paper, we give affirmative answer to the problem and propose a novel sparse training framework, "the Rigged Reinforcement Learning Lottery" (RLx2), for off-policy RL, which the first algorithm to achieve sparse training throughout using sparsity of more than 90% with only minimal performance loss. RIx2 is inspired by the gradient-based topology evolution criteria in Rigl. (Evei et al., (220)) for supervised learning. However, direct application of Rigl. does not achieve high sparsity. because sparse DRL models suffer from unreliable value estimation due to limited hypothesis space, which further disturbs topology evolution. Thus, RLx2 is equipped with a delayed multi-step Temporal Difference (TD) target mechanism and a novel dynamic-capacity replay buffer to achieve robust value learning and efficient topology exploration. These two new components address the value estimation problem under sparse topology, and together with RigL, achieve superior sparse-training performance.
The main contributions of the paper are summarized as follows.
We investigate the fundamental obstacles in training sparse DRL agent from scratch, and discover two key factors for achieving good performance under sparse networks, namely robust value estimation and efficient topology exploration.
Motivated by our findings, we propose RLx2, the first framework that enables DRL training based entirely on sparse networks. RLx2 possesses two key functions. i.e., a gradient- based search scheme for efficient topology exploration, and a delayed multi-step TD target mechanism with a dlynamic-capaciy replay buffer for robust value learning.
Through extensive experiments, we demonstrate the state-of-the-ar sparse training perfor- mance of RI.22 with two popular DRI. algorithms, TD3 (Fujimoto et al.) (018)) and SAC (Haarnoja et al. 2018), on several MuJoCo (Todorov et al., (2012) continuous control tasks. Our results show up to x model compression. RLx2 also achieves 20x acceleration in training and 50x in inference in terms of floating- soiiin operations (FLOPs).
RELATED WORKS
We discuss the related works on training sparse models in deep supervised learning and reinforce- ment learning below. We also provide a comprehensive performance comparison in Table [
Sparse Models in Deep Supervised Learning Han et al. (2155 2016); Srinivas al. (207)) Zhu
& Guptal (001)) focus on finding sparse network by pruning pre-trained dense networks Iterative Magnitude Pruning (IMP) in Han al. (2016) achieves a sparsity of more than 90%. Techniques including neuron characteristic (Hu et al. 2016)) dynamic network surgery (Guo et al. (2066), deriva- tives (Dong al. 2017 Molchanov et al. [2019)), regularization (Louizos et al. [208]] Tartaglione et al. 2018), dropout (Molchanov et al. (017)) and weight reparameterization Schwarz et al. (2021) have also been applied network pruning. Another line of work focuses on the Lottery Ticket Hy- pothesis (LTH), first proposed in Frankle & Carbin (2019), which shows that training from sparse network from scratch is possible one finds a sparse "winning ticket" initialization in deep super-
vised lcarning. The LTH is also validated in other deep learning models (Chen et al. 2020 Brix
ctt al. 2020 Chen et al. 2021).
Many works BBllec et al. 2017 Mocanu al. 2018 Mostafa & Wang 2019 Dettmers & Zettle-
moyer 2019 Evci et al. 2020) also try to train a sparse neural network from scratch without having to prantrd dense models. These works adjust structures of sparse networks during training, including Deep Rewiring (DeepR) (Bellec et al. 2017), Sparse Evolutionary Training (SET) (Mo-)
canu et al.] 2018), Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang 2019) Sparse
2

---

Published as a conference paper at ICLR 2023
Networks from Scratch (SNFS) (Dettmers & Zettlemoyer 2019), and Rigged Lottery (RigL) (Evci)
et al. 2020). Single- Network Pruning (SNIP) (Lee et al. 2019) and Gradient Signal Preserva- tion GraSP) (Wang et al. 2020) focus finding static sparse networks before training.
Table 1: Comparison of different sparse training techniques in DRL. Here ST and TA stand for "sparse throughout training" and "training acceleration", respectively. The shown sparsity is the maximum sparsity level without performance degradation under the algorithms. t: There are multi- ple method combinations in (Graesser et al. 2022), where "TE" stands for two topology evolution schemes: SET and RigL, "RL" refers to two RL algorithms: TD3 and SAC.
Name
Paradigm
Scenario
ST
TA
Sparsity
PoPS (Livne & Cohen 2020)
IMP
Online
No
No
99%
LTH-RL Yu ct al. 2020)
IMP
Online
Yes
No
99%
LTH-IL Vischer et al.
(0222)
IMP
Online
Yes
No
95%
SSP Arnob et al. 2021
Single-shot pruning
Offline
Yes
Yes
95%
GST Lee et al. (02TT)
Gradual pruning
Online
No
No
70%
DST Sokar el al. 2021
Topology Evolution
Online
Yes
Yes
50%
TE-RL Graesser et al. 2022
Topology Evolution
Online
Yes
Yes
90%
RLx2 (Ours)
Topology Evolution
Online Yes
Yes
95%
Sparse Models in DRL Evei al. (2020); Sokar et al. (2021) show that finding a sparse model in
DRL is difficult due to training instability. Existing works (Rusu et al., 2016)] [Schmitt et al.) 2018 2019) leverage knowledge distillation with static data avoid unstable training and obtain small dense agents. Policy Pruning and Shrinking (PoPs) in Livne & Cohen 2020) obtains sparse DRL agent with iterative policy pruning (similar to IMP). LTH in DRL is firstly investigated in eu al. (2020), and then Vischer et al. 2022) shows that a sparse winning ticket can also be found by behavior cloning (BC). Another line of works (Lee al. 202T, Sokar et al.) 2021 Amob al., 2021) attempts to train a sparse neural network from scratch without pre- pre-training dense teacher.
Zhang et al.
Group Sparse Training (GST) [ee et al. (202T) utilizes block-cireuit compression and pruning- Sokar et al. 2021) proposes using SET in topology evolution in DRL and achieves 50% sparsity. Arnob et al. (021) proposes single-shot pruning (SSP) for offline RL. Grresssr et al. (2022) finds that pruning often obtains the best results and plain dynamic sparse training methods, including SET and RigL,, improves over static sparse training significantly. However, existing works either demands massive computing resources, e.g. pruning- based methods (Rusu et al. 2016 Schmitt et al., 2018 Zhang et al. 2019 Livne & Cohen 2020). or fail in ultra sparse models, e.g. DST-based methods (Sokar et al.
2021 Graesser al. 2022). In this paper, we further improve the performance of DST
by introducing a delayed multi-step TD target mechanism with a dynamic-capacity replay buffer, which effectively addresses the unreliability of -xxe-topology models during sparse training.
3 DEEP REINFORCEMENT LEARNING PRELIMINARIES
In reinforcement learning, agent interacts with an unknown environment to learn an optimal pol- icy. The learning process is formulated as a Markov decision process (MDP) M = (S, r, P,7). where S is the state space, A is the action space, is the reward function, P denotes the transition matrix, and y stands for the discount factor. Specifically, at time slot given the current state s, E S, the agent selects an action 0 A by policy S A, which then incurs reward r(st, at).
Denote the Q function associated with the policy æ for state-action pair (s, a) as
(x) Esg
E "rsn 5,
In actor-critic methods Siiver et al. 2014), the policy T(s; Φ) is parameterized by policy (actor) network with weight parameter o. and the function Q(s, a; a; parameterized by value (critic) network with parameter 0. The goal of the agent is to find an optimal policy x² (s; Ø²) which
maximizes long-tem cumulative reward, i.e., J max, Ex 12/ > a:) (o, ag]-
There are various DRL methods for learning an efficient policy. In this paper, we focus on off-policy TD learning methods, including broad range of state-of-the-ard algorithms, e.g., TD3 (Fujimoto)
3

---

Published as a conference paper at ICLR 2023
al. 2018) and SAC (Haarnoja et al. 2018). Specifically, the critic network is updated by gradient descent to fit the one-step TD targets 71 generated by a target network ((s, a; 8"), i.e.
Ti = r(s, () 7 (8 a':
(2)
for each state-action pair (s,a), where a' ()); The loss function of the value network is defined as the expected squared loss between the current value network and TD targets:
(()) = (()) (Q(s.a;0) Ti1².
(3)
The policy IT (s; 6) is updated by the deterministic policy gradient algorithm in Silver al. (2014):
VoJ(o) (x)) VaQu (; [oT(s;)
RLx2: RIGGING THE LOTTERY IN DRL
In this section, we present the RLx2 algorithm, which capable of training a sparse DRL model from scratch. An overview of the RLx2 framework [ To motivate the design of RLx2, we present a comparison of four sparse DRL training methods using TD3 with different topology update schemes InvertedPenduluum-v2 simple control task from MuJoCo, in Figure
actor-critic architecture is shown in Fig-
Topolagy Evalution
Topolagy Evalution
Grow
Prune
Grow
Prune
Performance (%%
80
Critic
Actor
Replay Buffer (Dynami C-aize)
Interaction
40
f== (Q TF
la = T(s)
0
RigL RigL+Q- Rix2
Figure 2: Performance compari- son for four sparse training meth- ods, i.c., SS, RigL, RigL+Q" and RLx2. The results show that both efficient topology evolution and ro- bust value estimation are critical.
Figure 1: The RLx2 framework contains three key components, i.e. multi-step TD target mechanism, dynamic-capacit replay buffer and gradient-bart based topology evolution.
From the results, make the following important observations. (i) Topology evolution essential. It can be seen that a random static sparse network (SS) leads to much worse performance than RigL. (ii) Robust value estimation is significant. This validated by the comparison between RigL and RigL+Q", both using the same topology adjustment scheme but with different Q-values.
Motivated by the above findings, RLx2 utilizes gradient-based topology adjustment, i.e., RigL (for topology evolution), and introduces delayed multi-step TD target mechanism with dynamic- capacity replay buffer (for robust value estimation). Below, we explain the key components of RLx2 in detail, to illustrate why RLx2 is capable of achieving robust value learning and efficient topology exploration simultaneously.
4.1 GRADIENT-BASER TOPOLOGY EVOLUTION
The topology evolution in RLx2 is conducted by adopting the RigL method (Evei al. 2020 Specifically, we compute the gradient values of the loss function with respect to link weights. Then, we dynamically grow connections (connecting neurons) with large gradients and remove existing links with the smallest absolute value of the weights. In this way, we obtain sparse mask that evolves by self-adjustment.
Four schemes: training with random static sparse network (SS): 2) training with RigL, (RigL): 3) dynamic sparse training guided by true Q-value, i.c., Q-values from fully trained expert critic with dense network (RigI++")) ); and dynamic sparse training guided by leamned Q-value with TD targets (RLx2).
4

---

Published as a conference paper at ICLR 2023
The pseudo-code of our scheme is given in Algo- rithm where @ is the element-wise wise multiplica- tion operator and Mo is the binary mask to rep- resent the sparse topology of the network O. The update fraction anneals during the training pro- cess according &. & (1 + cos( - where Co is the initial update fraction and Tend is the total number of iterations. Finding top-k links with maximum gradients in Line 10 can be effi- ciently implemented such that Algorithm[ owns
Algorithm 1 Topology Evolution
(Evei
et al., 2020)
1: N7: Number of parameters in layer
2: 0:: Parameters in layer I
3: Me.: Sparse mask of layer /
4: sy: Sparsity of layer
5: L: Loss function
6: G: Update fraction in training step
7: for each layer do
8: k = (r St
time complexity (((1- -) N) log N)) (detailed in Appendix|A A.I) where is the total sparsity. Be- sides, the topology adjustment happens very in- frequently during the training, i.e., every 10000 step in our setup. such that consumption of this step is negligible (detailed in Appendix C.3) Our topology evolution scheme can be imple- mented efficiently on resource-limited devices.
9: Idrop = Arg (o o Ma. (.))))))))
10: Igoow = Arg Too ao Vlop (1V k()
11: Update Ma, according to drop and grow
12:
01 +- 0g @ Mo,
13: end for
5000
R gl mask Rix2 mark
4.2 ROBUST VALUE LEARNING
bestape Rotum
4000
As discussed above, value function learning is crucial in sparse training. Specifically, we find that under sparse models, robust value learning not only serves to guarantee the efficiency of boot- strap training as in dense models, but also guides the gradient-based topology exploration of the sparse network during training.
3000
2200
1000
Figure compares the performance of the masks (i.e., sparse net- work topology) obtained by RigL and RLx2 (i.c., RigL + robust value lcarning) on Ant-v3. Here we use method similar to (Fran- kle & Carbin 2019) for evaluating the obtained sparse mask 1) first initialize a random sparse topology: 2) keep adjusting the topology during the training and obtain the final mask; 3) train a sparse agent with the obtained mask (the mask is fixed throughout this training phase, only the weights are restored
Figure 3: Sparse model com- parison in Ant-v3.
their initial values as in the first step at the beginning). It can be clearly observed that the mask by RLx2 significantly outperforms that by solely using RigL (Appendix C.4 provides details and experiments in other environments, where similar results are observed).
To achieve robust value estimation and properly guide the topology search, RLx2 utilizes two novel components: i) delayed multi-step TD targets to bootstrap value estimation; ii) a dynamic-capacity replay buffer to eliminate the potential data inconsistency due to policy change during training.
4.2.1
MULTI-STEP TD TARGET
In TD learning, a TD target is generated, and the value network will be iteratively updated by mini- mizing a squared loss induced by the TD target. Single-step methods generate the TD target by com- bining one-step reward and discounted target network output, i.e., Ti + ( n 1); 0)- However, sparse network parameter 0 0 Ma, obtained from its dense counterpart 0, will inevitably reside in a smaller hypothesis space due to using fewer parameters. This means that the output of the sparse value network 0 can be unreliable and may lead to inaccurate value estimation.
Denote the fitting error the value network as E(s,, a) = Q(s, 0) - (s) One sees that this error may be larger under a sparse model compared to that under dense network.
To overcome this issue, we adopt multi-step target, i.e., To ("*(st+n, (s)4m); 0), where the target combines an N-step sample and the output of the sparse value network after N-step, both appropriately discounted. By doing so, we reduce the expected error between the TD target and the true target. Specifically, Eq. shows the expected TD er- ror between multi-step TD target Tn and the true Q-value Q, associated with the target policy m conditioned on transitions from behavior policy b (see detailed derivation in Appendix (A.2) E, (s. (s. a)] Qn (E, o)]
Ek 2 +
(s. +y" Ez(e(sn, IT (sn)))
(4)
Policy inconsistency error
Network fitting error
Frankle & Carbin 2019 obtains the mask, i.c., the "lottery ticket", by pruning pretrained dense model. Our sparse mask the hinal mask obtained by dynamic sparse training.
5

---

Published as a conference paper at ICLR 2023
The multi-step target has been studied in existing works (Bertsekas & loffe 1996. Precup 2000
Munos et al. 2016) for improving TD learning. In our case, we also find that introducing a multi- step target reduces the network fitting error by multiplicative factor y", as shown in Eq. On the other hand, it has been observed, e.g., in Fedus et al.] (2020), that an immediate adoption of multi-step TD targets may cause larger policy inconsistency error (the first term in Eq. 4 Thus, we adopt a delayed scheme to suppress policy inconsistency and further improve value learning. Specifically, the early stage of training, we use one-step TD targets to better handle the quickly changing policy during this period, where a multi-step target may not be meaningful. Then, after several training epochs, when the policy change becomes less abrupt, We permanently switch to multi-step TD targets, to exploit its better approximation of the value function.
4.2.2 DYNAMIC-CAPACITY BUFFER
The second component of RLx2 for robust value learning is novel dynamic buffer scheme for controlling data inconsistency. Off-policy algorithms use a replay buffer to store collected data and train networks with sampled batches from the buffer. Their performances generally improve when larger replay capacities are used (Fedus et al.] 2020). However, off-policy algorithms with unlimited-sin replay buffers suffer from policy inconsistency due to the following two aspects.
ii) Inconsistent multi-step targets: In off-policy algorithms with multi-step TD targets, the value function is updated minimize the squared loss in Eq. (3) on transitions sampled from the replay buffer, i.e., the reward sequence Тı, rs+1,' :. Tt+ collected during training. However, the fact that the policy can evolve during training means that the data in the replay buffer, used for Monte- Carlo approximation of the current policy may be collected under a different behavior policy (Hernandez-Garcia & Sutton 2019 Fedus et al. 2020). As result, it may lead large policy inconsistency error in Eq. ④ causing inaccurate estimation.
Sample amounk
(ii) Mismatched training data: In practice, the agent mini- mizes the value loss ((0) with respect the sampled value in mini-batch B,, given by
0.6
0.4
0.2
[() = Be
Σ (s.a,)aBB
(Q(s;, a;;0) T
(5)
$0.20
Theckoki
0.15
50.10
Compared to Eq. (3), the difference between the distribu- tion of transitions the mini-batch B, and the true transi- tion distribution induced by the current policy also leads to a mismatch in the training objective (Fujimoto et al. 2019). Indeed, our analysis in Appendix A.4 shows that training performance closely connected to policy consistency.
20.05
3.0
Time steps
Figure 4: Dynamic buffer capacity & policy inconsistency
Motivated by our analysis, introduce ddnamically buffer to reduce the policy gap based on the policy distance of the collected data. The formal scheme is given in Algorithm [3] We introduce the following policy distance measure evaluate the inconsistency of data in the buffer, i.e.
1 K
Σ
|| Φ) l|2.
(6)
(+L,0))EOHK(())
where B denotes the current replay buffer, OldK(B) denotes the oldest K transitions in B, and ( σ) is the current policy. Here K is a hyperparameter. We calculate the D(B, σ) value every As steps. If D(B, Φ) gets above certain pre-specified threshold Dp, we start to pop items from B in First--n-First-Out (FIFO) order until this distance measure D becomes below the threshold.
visualization of the number of stored samples and the proposed policy distance metric during training is shown in Figure 4 We see that the policy distance oscillates in the early stage as the policy evolves, but is tightly controlled and does not violate the threshold condition to effectively address the off off-policyness issue. As the policy converges, the policy distance tends to decrease and converge (Appendix C.8 also shows that the performance is insensitive to the policy threshold Do).
EXPERIMENTS
In this section, we investigate the performance improvement of RLx2 Section and the impor- tance of each component in RLx2 in Section 5.2 In particular, pay extra attention to the role

---

Published as a conference paper at ICLR 2023
topology evolution plays in sparse training in Section 5.3 Our experiments conducted in four popular MuJoCo environments: HalfChectab-v3 (Hal.), Hopper-v3 (Hop.), Walker2d-v3 (Wal.), and Ant-v3 (Ant.) I for RLx2 with two off- policy algorithms, TD3 and SAC. Instantiations of RLx2 on TD3 and SAC are provided in Appendix B Each result is averaged over eight random seeds. The code is available at.ttts:: //g teggtthhub.... cwm/tygl 0224//0002222200 /
5.1 COMPARATIVE EVALUATION
Table summarizes the comparison results. In our experiments, we compare RLx2 with the fol- lowing baselines: (i) Tiny, which uses tiny dense networks with the same number of parameters as the sparse model in training. (ii) SS: using static sparse networks with random initialization. (iii) SET (Mocanu et al., 2018), which uses dynamic sparse training by dropping connections according to the magnitude and growing connections randomly. Please notice that the previous work (Sokar ct al. 2021) also adopts the SET algorithm for topology evolution in reinforcement learning. Our implementations reach better performance due to different hyperparameters. (iv) RigL (Evci et al., 2020), which uses dynamic sparse training by dropping and growing connections with magnitude and gradient criteria, respectively, the same as RLx2's topology evolution procedure.
Table Comparisons of RLx2 with sparse training baselines. Here "Sp." refers to the sparsity level (percentage of model size reduced), "Total Size" refers to the total parameters of both critic and actor networks (detailed calculation of training and inference FLOPs are given in Appendix C.3). The
right five columns show the final performance of different methods. The "Total size," "FLOP" and
"Performance" are all normalized w.r.t. the original large dense model (detailed in Appendix
Actor Sp.
Critic
Total
FLOPs (Train)
FLOPs (Test)
Tiny (%)
SS
SET
RigL (%)
RLx2
Alg.
Env.
Sp.
Size
(%)
(%)
(%)
Hal.
90%
85%
0.133x
0.138x
0.100x
86.3
77.1
92.6
90.8
99.8
Hop.
98%
95%
0.040x
0.043x
0.020x
64.5
67.7
66.5
90.6
97.0
TD3
Wal.
97%
95%
0.043x
0.045x
0.030x
60.8
42.9
39.3
35.7
98.1
Ant.
96%
88%
0.093x 0.100x
0.040x
16.5
49.6
62.5
68.5
103.9
Avg. 95%
91 %
0.077x 0.081x
0.048x
57.0
59.3
65.2
71.4
99.7
Hal. 90%
80%
0.180x
0.197x
0.100x
95.0
75.4
94.8
89.8
102.2
Hop.
98%
95%
0.044x
0.048x
0.020x
89.1
81.6
103.9
110.0
109.7
SAC
Wal.
90%
90%
0.100x
0.113x
0.100x
73.8
83.4
95.8
81.9
103.2
Ant
90%
75%
0.220x
0.239x
0.100x
49.6
49.3
79.8
90.9
105.6
Avg. 92%
85%
0.136x
0.149x
0.080x
76.9
72.4
93.6
93.2
105.2
Avg.
94%
88%
0.107x
0.115x
0.064x
67.0
65.9
79.4
82.3
101.8
In our experiments, we allow the actor and critic networks to take different sparsities. We define an ultimate compression ratio, i.e., the largest sparsity level under which the performance degradation under RLx2 is within +%3 of that under the original dense models. This can also be understood as the minimum size of the sparse model with the full performance of the original dense model. We present performance comparison results in Table based on the ultimate compression ratio. The per- formance of each algorithm is evaluated with the average reward per episode over the last 30 policy evaluations of the training policy evaluation is conducted every 5000 steps). Hyperparameters are fixed in all four environments for TD3 and SAC, respectively, which are presented in Appendix C.2
Performance Table 2 shows RLx2 performs best among all baselines in all four environments by large margin (except for close performance with RigL with SAC in Hopper). In addition, tiny dense (Tiny) and random static sparse networks (SS) performance are worst on average. SET and RigL are better yet fail to maintain the performance in Walker2d-v3 and Ant-v3, which means robust value learning is necessary under sparse training. To further validate the performance of RLx2, we compare the performance of different methods under different sparsity levels Hopper-v3 and Ant- v3 in Figure S] showing RLx2 has a significant performance gain over other baselines.
Model Compression RLx2 achieves superior compression ratios (the reciprocal of the total size), with minor performance degradation (less than 3%). Specifically, RLx2 with TD3 achieves 7.5x- 25x model compression, with the best compression ratios of 25x on Hopper-v3. The actor can be
AA complex environment with larger state space, Humanoid-v3. is also evaluated in Appendix C.9

---

Published as a conference paper at ICLR 2023
compressed for each environment by more than 96%, and the critic is compressed by 85%-95%. The results for SAC are similar. RLx2 with SAC achieves a 5x-20x model compression.
Dense
Tiny Dense Walker2d-v3
Static Sparse
SET
RigL
RLx2
Ant-v3
Performance (%)
Performance (%)
100
100
50
71.1
85.6
92.8 Sparsity (%)
95.7
97.1
53.3
76.7
90.7
93.0
95.3
Sparsity (%)
Figure 5: Performance comparison under different model sparsity.
Acceleration in FLOPs Different from knowledge hewwllede-diistllationBBB based methods, e.g., Livne & Cohen (2020): Vischer et al.] (2022): Lee et al.) (2021). RLx2 uses sparse network throughour training. Thus, it has an additional advantage of immensely accelerating training and saving com- putation, i.e., 12x training acceleration and 20x inference acceleration for RLx2-TD3, and 7x training acceleration and 12x inference acceleration for RLx2-SAC.
5.2 ABLATION STUDY
We conduct a comprehensive ablation study on the three critical components of RLx2 on TD3, i.e., topology evolution, multi-step TD target, and capacic- buffer, to examine the effect of each component in RLx2 and their robustness in hyperparameters. In addition, we provide the sensitivity analysis for algorithm hyper-parameters, e.g- initial mask update fraction, mask update interval, buffer adjustment interval, and buffer policy distance threshold, in Appendix (C.8
Topology evolution RLx2 drops and grows connections with magnitude and gradient criteria, respectively, which has been adopted in RigL (Evci et al. (020) for deep supervised learning. To validate the necessity of our topology evolution criteria, we compare RIx2 with three baselines, which replace the topology evolution scheme in RLx2 with Tiny, SS and SET, while keeping other components in RLx2 unchanged R Thy evolution e left partpolog of Table 3 shows that RigL as topology adjustment scheme (the resulting scheme is RLx2 when using RigL) performs best among the four baselines. We also observe that Tiny performs worst, which is consistent with the conclusion in existing works (Zhu & Gupta 2018)) that a sparse network may contain smaller hypothesis space and leads to performance loss, which necessitates topology evolution scheme.
Table 3: Ablation study on topology evolution and multi-step target, where the performance (%) is normalized with respect to the performance of dense models.
Topoloy
Evolution SET
Multi-step Target
Env.
Tiny
SS
RLx2
1-step
2-step
3-step
4-step
5-step
Hal.
93.3
86.1
100.1
99.8
96.5
101.7
99.8
98.8
97.0
Hop.
74.4
84.2
88.8
97.0
77.9
91.7
97.0
84.0
87.5
Wal.
84.1
83.8
89.4
98.1
73.9
93.7
98.1
99.1
99.3
Ant.
28.7
80.2
83.5
103.9
103.9
105.1
103.9
96.7
94.5
Avg.
70.1
83.6
90.4
99.7
88.1
98.1
99.7
94.6
94.6
Multi-step TD targets We also compare different step lengths in multi-step TD targets for RLx2 in the right part of Table 3 We find that multi-step TD targets with step length of obtain the maxi- mum performance. In particular, multi-step TD targets improve the performance dramatically in H3pper-v3 and Walker2d-v3, while the improvement in HalfCheetach-v3 and Ant-v3 is minor.
6000
5000
4:00
3000
2000
1000
We compare different buffer sizing schemes, including our dynamic scheme, different fixed-capacity buffers, and an unlimited buffer. Figure [ shows that our dynamic- capacity buffer performs best among all settings of the buffer. Smaller
Dynamic-capacity Buffer
0.0
1.0
1.5
2.5
2.0
Tins stops
Figure 6: Performance with different buffer schemes.
"Take Algorithm in Appendix B for example. Only lines 16 and 21 of "Topology Evolution())" are changed while other parts remain unchanged. We also regard static topology as special topology evolution.
8

---

Published as a conference paper at ICLR 2023
buffer capacity benefits the performance in the early stage but may reduce the final performance. This is because using a smaller buffer results in higher sample efficiency in the early stage of train- ing but fails in reaching high performance in the long term, whereas a large unlimited one may perform poorly in all stages.
5.3 WHY EVOLVE TOPOLOGY IN DRL?
Compared to dense networks, sparse networks have smaller hypothesis spaces. Even under the same sparsity, different sparse architectures correspond different hypothesis spaces. As FFaankle & Carbin (2019) has shown, some sparse architecture (e.g., the "winning ticket") performs better than random one. To emphasize the necessity of topology evolution sparse training, we compare different sparse network architectures in Figure 7 including the random ticket (topology sampled random and fixed throughout training), the winning ticket (topology from an RLx2 run and fixed throughout training), and a dynamic ticket (i.e., training using RLx2) under both reinforcement learning (RI.) and behavior cloning (BC)
From Figure 7(a)
we see that RLx2 achieves the best
6000
6000
Average return
Average return
4000
4000
2000
2000
Random Ticket
Random Ticket
Winning Ticket
Winning Ticket
0.0
1.0
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
Time steps
le6
Time steps
1e6
(a) Rigging the Lottery Ticket in RL training
(b) Rigging the Lottery Ticket in behavior cloning
Figure 7: Comparison of different sparse network architecture for training a sparse DRL agent in Ant-v3, where the sparsity is the same that in Table
performance, which comparable with that under the original dense model. Due to the potential data inconsistency problem in value learning and the smaller hypothesis search space under sparse networks, training with single fixed topology does not fully reap the benefit of high sparsity and can cause significantly degraded performance. That is why the winning ticket and random ticket both lead to significant performance loss compared to RLx2. On the other hand, Figure (7b) shows that BC tasks, the winning ticket and RLx2 perform almost the same as the dense model, while the random ticket performs worst. This indicates that appropriate fixed topology can indeed be sufficient to reach satisfactory performance BC, which is intuitive since BC adopts a supervised learning approach and eliminates non-stationarity due bootstrapping training. In conclusion, find that fixed winning ticket can perform as well as a dynamic topology that evolves during the training in behavior cloning, while RLx2 outperforms the winning ticket in RL training. This observation indicates that topology evolution not only helps find the winning ticket in sparse DRI. training but is also necessary for training sparse DRL. agent due to the extra non-stationary in bootstrapping training, compared to deep supervised learning.
6 CONCLUSION
This paper proposes sparse training framework, RLx2, for off-policy reinforcement learning (RL). RLx2 utilizes gradient-based evolution to enable efficient topology exploration and establishes ro- bust value learning using delayed multi-step TD target mechanism with dynamic-capacity replay buffer. RLx2 enables training an efficient DRL agent with minimal performance loss using an ultra- sparse network throughout training and removes the need for pre-training dense networks. Our extensive experiments on RLx2 with TD3 and SAC demonstrate state-of- state-of-the-art sparse training per- formance, showing 7.5 x model compression with less than 3% performance degradation and up to 20x and 50x FLOPs reduction in training and inference, respectively. It will be interesting future work to extend the RLx2 framework in more complex RL scenarios requiring more intense demand for computing resources, e.g., real-world problems instead of standard MuloCo environ- ments or multi-agent settings.
'In BC, the actor network trained under the guidance of well-trained expert instead of the critic network.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Experiment details (including an efficient implementation for RLx2, implementation details of the dynamic buffer, hyperparameters, and network architectures) included in Appendix C for repro- duction. The proof for our analysis of the dynamic buffer can be found in Appendix A.4 The code open-sourced in https: hatt::/ggittbbbu com wttppu/...ottttg////e// www.toggg/////rx
ACKNOWLEDGEMENTS
The work is supported by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108400 and 2020AAA0108403, the Tsinghua University Initiative Scientific Research Program, and Tsinghua Precision Medicine Foundation 10001020109.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference machine learning, pp. 22-222-32 31.31. PMLR, 2017.
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. Single-shot pruning for offline reinforcement learning. arXiv preprint 1rXiv:2112.15577 2021.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv: 2017.
Dimitri P Bertsekas and Sergey loffe. Temporal differences-based policy iteration and applications in neuro-dynamic programming. Lab. for Info. and Decision Systems Report LIDS-P-23453 MIT, Cambridge, MA, 14, 1996.
Christopher Brix, Parnia Bahar, and Hermann Ney. Successfully applying the stabilized lottery ticket hypothesis to the transformer architecture. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 3909-3915, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 13:15834 15846, 2020.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-1706 PMLR, 2021.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abhas Abdolmaleki, Diego de Las Casas, et al. Mag- netic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):414- 419, 2022.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv: 09//./ 2019.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 1rX1v:1907.04840, 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2953-2952. PMLR, 2020.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Con- ference Machine Learning, pp. 3061-3071. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2023
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International conference on learning representations, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off--olicy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 2052-2062. PMLR, 2019.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference Machine Learning, pp. 7766-7792. PMLR, 2022.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 1389-33966. IEEE, 2017.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, Advances in neural information processing systems, 29, 2016.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International conference on learning representations, 2016.
J. Fernando Hernandez- and Richard S. Sutton. Understanding multi-step deep reinforcement learning: A systematic study of the dqn target. CoRR, abs/1901.07510, 2019. URL http:
Hengyuan Hu, Rui Peng, Yu- WunWing Tai, and Chi- Chi-Keung Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv: 2016.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In International conference on learning representations, 2021.
Juhyoung Lee, Sangyeob Kim, Sangjin Kim, Wooyoung Jo, and Hoi-Jun Yoo. Gst: Group-sparse training for accelerating deep reinforcement learning. arXiv preprint 1rXiv:2101.09650, 2021.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr Snip: Single-shot network pruning based on connection sensitivity. In International conference on learning representations, 2019.
Dor Livne and Kobi Cohen. Pops: Policy pruning and shrinking for deep reinforcement learning. IEEE Journal of Selected Topics in Signal Processing, 14(4):789-801, 2020.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. In International conference on learning representations, 2018.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1112, 2018.
Dmitry Molchanov, Arseni Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 2498 2507. PMLR, 2017.
P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019a
11

---

Published as a conference paper at ICLR 2023
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11272, 2019b.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference Machine Learning, pp- 4646-4655. PMLR, 2019.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems, 29, 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series, pp. 80, 2000.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. In International conference on learming representations, 2016.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek. Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Jocl Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstart- ing deep reinforcement learning. arXiv preprint 180Xiv:80330000 2018.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter E Latham, and Yee Tch. Power- propagation: A sparsity inducing weight reparameterisation. Advances in neural information processing systems, 34:28889-28900 2021.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pp- 387-395. PMLR, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):554-35) 2017.
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. arXiv preprint arXiv:2106.04221,, 2021.
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 138-145, 2017.
Enzo Tartaglione, Skjalg Lepsøy, Attilio Fiandrotti, and Gianluca Francini. Learning sparse neural networks via -ensitivity-driven regularization. Advances neural information processing sys- tems, 31, 2018.
Gerald Tesauro et al. Temporal difference learning and td-gammon. Communications of the ACM, 38(3):58-60 68, 1995.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Mare Aurel Vischer, Robert Tjarko Lange, and Henning Sprekeler. On lottery tickets and minimal task representations in deep reinforcement learning. In International conference on learning representations, 2022.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 2020.
12

---

Published as a conference paper at ICLR 2023
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in and nlp. In International conference on learning representations. 2020.
Hongjie Zhang, Zhuocheng He, and Jing Li. Accelerating the deep reinforcement learning with neu- ral network compression. In 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. In Workshop at international conference on learning representations, 2018.
13

---

Published as a conference paper at ICLR 2023
Supplementary Materials
Additional Details for Section
15
A.1
Efficient Implementation for Algorithm
15
A.2
Derivation of Eq. (4) in Section
16
A.3 Algorithm of Dynamic-capaciry Buffer in Section 4.2.2
16
A.4
Detailed analysis of dynamic buffer
17
Details of RLx2 with TD3 and SAC
19
Experimental Details
21
Hardware Setup
Hyperparameter Settings for Reproduction
................................
21
Calculation of Model Size and FLOPs
Comparisons between Different Masks in Other Environments
25
Training Curves of Comparative Evaluation in Section 5.1
................................
25
C.6
Standard Deviations of Results in Table 2
26
Supplementary Results for Ablation Study
27
Sensitivity Analysis for Hyperparameters
27
Additional Results in Humanoid-v3
29
10
Supplementary Results for Investigation the Lottery Ticket Hypothesis
30
Visualization of Sparse Models
31
14

---

Published as a conference paper at ICLR 2023
A ADDITIONAL DETAILS FOR SECTION 4
This section provides additional details for Section including how to efficiently implement Algo-
rithm [ with limited resource, the derivation of Eq. in Section 4.2. and the full algorithm of Jynamic-capacity buffer in Section 4.2.2
A.1 EFFICIENT IMPLEMENTATION FOR ALGORITHM
For simplicity, in this section, we omit all indices of in the symbols that appeared in Algorithm I
Parameter Storing Suppose layer takes an (in) -dmmensionl vector x as the input, and out- puts an n (mut) feature vector y linear transformation. Then the layer's number of parame- ters is N = (in) x nn(out) A naive implementation will be to store both 0 E RR (n( and Me E {0, 1] (n(in) in two dense n (i())) matrices in the memory. In forward and back- ward propagations, one simply performs the dense-matrix denne-matrix-mullily------ multipll operation on 0 and x. using sparsity ratio close to 1, so the network is highly sparse. Also, the actual memory occupied by the model is always proportional N and irrelevant s. However, better way is store 0 in more compact manner, where only the non-zero indices (i.e., positions of the ones in Ma) and their values. As result, the weights of the layer now occupies @((1 s) memory, and the matrix- mwiiiimlultiteeeeeeeeee ation also just costs O((1 - s) + (fn) (out)). Such sparse matrix (or tensor of higher orders) structures are supported by many modern machine learning frameworks, c.g., torch. sparse in PyTorch. Many of them also support automatic gradient calculation and backward propagation.
However, this implementation cannot enjoy any speed-up even when
oper-
Link Dropping With this sparse representation of 0, it can be seen that the link dropping step of Algorithm (Line 9 can be done in O((1 N) log N) time by sorting all the weight entries in their absolute values and then picking the top- K items.
Link Growing It then suffices to figure out a way to implement the link growing step of Algo-
rithm (Line 10). Denote by L the scalar loss function of the whole neural network containing layer Assume that we have just performed backward propagation, where layer contributes only
0((1 [[[1=)-N-a (in) +n(ott) to the computation time, and g((2) = 3x g = By
aL and g.() 36
a7.
all have been computed. Since 0 is in a compact representation with (1 - s)N elements, now (0)
obtained by auto-grad also only contains (1 - s)N items, but the growing step Line 10 basically asks to collect the top- items in the whole dense gradient matrix with N elements.
According to the chain rule, for a link fz. between the i-th input neural and the j-th output neural, the partial derivative of with respect to Ozi is given by (here, we abuse the notation a little)
(0) 0L 9
OL Vj (v) oy, a0,
9 x4.
00
Hence the desired true dense (out) x (in) gradient matrix g (g) is equal to g(v)x", Our task reduces to collect the K entries with the largest absolute values while keeping away from the locations that have just been dropped in Line In fact, this procedure can efficiently implemented by scanning via no!ut)) pointers with the help of heap (a.k.a,, priority queue), described in the following pseudo-code Algorithm
can be seen that Algorithm consumes (n + (in) + |U] (1 s). heap operations and set operations. If all sets S and U are implemented using binary search trees hash tables, the costs for each heap operation and each set operation are all within Ollog N). Therefore the total running time of Algorithm 2 is ((11 s)N log N).
15

---

Published as a conference paper at ICLR 2023
Algorithm 2 Efficient Link Growing
Input: Input featrue vector output gradient vector g (1) base index set Sp, dense matrix size target sparsity 8, forbid index set U Output: An index set S such that |S| = (1 s)N, S 2 So and S S =
N,
1: Initialize S So.
2: Sort [xr] to get permutation di,.... (it),, such that [ T
Tgg s,,,
3: Create a max-heap whose elements are 3-tuples, comparing elements according their first components.
4: for j 1/=1.... (out) do (1)
5: Append (10) Foi 1. 1) into H.
6: end for
7: while |S| (1 s)N do
8: Pop the top triple (w, out of H.
9: if (.)))) $ S and (j, $ then
10:
U-SU(,,
11:
end if
12: if < (im) then
13:
Append (lo, (0 end if
so 1. 1 + into H.
14:
15:
end while
16: Return S.
A.2 DERIVATION OF EQ. (4) IN SECTION 4.2.
Denote Pr( a++1, St+n at+n st, a.) the distribution of the trajectory starting from the current state 8t and action at under policy x. For simplicity, we use E, denote ((ee41 "er11.."" (epp (-)a..a)) and use E, to denote ((e11100-11 Benn (ea (1).)))))))))))))))))))))))))))) where and denote the current policy and the behavior policy, respectively. Qr denotes the function associated with policy as defined in Eq. I1) in the manuscript, i.e., Q=(s,a)
E Ei t (8i, as) |s, s, 5. a). We also use e(s, a) to denote the network fitting error, i... E(s, a) Q(s, 0) (s, Subsequently, we have:
Eg Ta (st, az) (s(s),
E&
rre* + ((c44)::)) En
* Preck + (s (st4m)))
0=
k=0
Esl
Mr4* + "" (Q T ((s+n)):
E
A " ((s+n)::))
k=
k=
+ E,
= (EE.TT a:)l - (st, a)) + E" T (s)+n))]]
The first equality holds in the manuscript by definitions Eq. ①1 and of multi-step TD targets. The second equality holds by firstly adding and then subtracting the term E, (Q artni 0))] and the last equality holds by definitions of (s, a) and E(s, a). This decomposition shows that the expected error consists of parts, i.e., the network fitting error and the policy inconsistency error, which are well handled by our multi-step TD targets with a ddyaamic-capaco buffer.
A.3 ALGORITHM OF DYNAMIC-CAPACITY BUFFER IN SECTION 4 4.2.2
Algorithm presents our formal procedure for dynamically controlling the buffer capacity in Sec- tion 4.2.2 For each step, a new transition is inserted into the replay buffer. To avoid the problem of policy inconsistency, check the buffer per As steps and drop the oldest transitions if needed.
16

---

Published as a conference paper at ICLR 2023
Specifically, first set hard lower bound min, and hard upper bound Bmax of the buffer pacity. (i) If the buffer size is below Bmin- we store all newly collected data samples. (ii) the amount of buffered transitions has reached m the oldest transitions will be replaced by the latest transitions. (iii) When the buffer is in ( (Bmin: Bmax), ) for each time, we calculate the policy distance between the oldest behavior policy and the current policy, based the oldest transitions stored in the buffer. In addition, if the policy distance exceeds the threshold Do. the oldest transitions are discarded. The full algorithm is given in Algorithm 3
Algorithm Saynamic-Capacciy Buffer for off-policy DRL
x: current policy
2: B: Replay buffer at training step
3: As: Buffer checking interval
4: Do: Non-conse connconsist threshold
5: p: Shrinking ratio
Bmin: Bux: Range of capacity
7: for each training step do
8: Interact with environment and store new transitions in B // Add new transitions into the buffer
9: Sample mini-batch from B
10: Training with sampled data Depends
training algorithim
11: if t mod As = 0// Check the buffer periodically then
12:
if [B] E (Bain: Bmas) // The buffer capacity is limited in certain range then
13:
Batch of oldest transitions (st,
14:
if D(T) > Do Drop the oldest transitions if the policy distance exceeds the threshold then
end Drop ratio p of oldest transitions
16:
17: end if
18: end if
end for
A.4 DETAILED ANALYSIS OF DYNAMIC BUFFER
Using dynamic buffer can reduce the gap between target policy and behavioral policy, we have shown empirically in Section 5.2 In this section, we give a more detailed analysis of the influence of the dynamic buffer.
We first define the notations used in our analysis.
p(s [|. a): Environment transition probability
a.t State distribution in time / under policy π
(+).
7. State-action pair distribution in time under policy π
(a,0)
Ma (T): Distribution of trajectory T (St, Ot, St+1. ... 8ttn: Oc+n) under policy π
d, State-action visitation distribution under policy (s) - 7 'PPr(sr 7
s), ~ T T--1s))
Lemma A. below first shows that for two trajectory distributions generated by different policies,
their KI. divergence can be expressed by the Kl. divergence between these two policies.
Lemma A.1. DKL (us(sss (x)l|ux (-18),)))
Σi=1 k=
E (
(bc(b( (1l
Proof. The conditional trajectory distribution can be expressed as:
St, a.)= = II x
(c4b-11)
Thus,
Dxr(uu((-ss, (x)l az)) Σ po(T]Se, ar) log Ho(Tise, ai)
" 8t, az)
17

---

Published as a conference paper at ICLR 2023
pb(T) (t. ar) log I blatk 8(+k)
k= π (at+k ((+k)
(M)(T) log b(a1+bl51+k) π (((c4s(544))
Note that
E E flog |b(T]s(, St,
(e (p) flog [b(T st, ar) Pe (T| ac)
Mx (xx(T)s s2, ar)
then
(s.a) Portth log b(a++1)04)) x (ac+x18442)
DKI(Ho(-1se (+1s), at))
Σ
Σ
(sete cr44))
SetN Po.14
1=
Proposition A.2 below shows that the relation between the policy inconsistency eπror defined in equation 4 and policy distance. This proposition shows that multi-step TD learning can indeed be more robust with dynamic buffer.
Proposition A.2. The policy inconsistency error in equation can be upper bounded by
1 - 7 <
RR [T]] Er [T]]]
1 7 Pm + 7m))m)
Σ k=1
o 2200 (xa(6((1s]1=(((b))
where F. supr inf Qm sup ax, az
inf as),
Proof. Suppose the multi-step TD target is bounded, we have
[T E, [T.]] = |E st.a:) [Tn]] - Er (1s
S(sup Tn - inf Tv Drv ad) (1)s, a.)).
According to Pinsker's ' inequality,
DKL (2)
(u) (-1s,, at))
Drv(uu (1)0,)))))) S
Thus,
R-1
(xo( (e; (-18,, 0r))
Eg Tm
E TT
S
Tm + 7" Qm)m]
2
Finally, using Lemma A.I
we obtain
1 yn
E [T.] - E. [Ta]l <(
+ n 7mm)
2
E
p
22
(b((s)(
Our next result, Proposition controlled by reducing the KL divergence between the target policy and the behavior policy (com- plete proof in Appendix A.4 Therefore, one can improve the value estimation by eliminating data inconsistency.
A.3 below shows that the mismatch between L(0) and L()) can be
18

---

Published as a conference paper at ICLR 2023
Proposition A.3. For the target policy and behavioural policy b, have
27 A
V Daa (()))))))(((())))
[C(0) - [(0) - 1-
where A = ssup (Q(p - ((s,a))))))))))))) a))² and the loss functions are defined as
((0) = ((s.a)))-l((((: - ((s (. ((0) = (.) [[Q(s;, a,; T(;,, a;))
Proof. Denote
A = =Qppp(((((s - a(),
Then, we have
[C(0) [(0) =IE (QQ a;; -(s,)) ( (ss (Q 0;; ((s 0;3)
22Drv (dn, d
i.e., the gap between the two loss functions can bounded by the total variance distance between the two state-action visitation distributions.
According to Achiam et al. (011)) we obtain
Drv(da do) S I-
7 -_Ewnd (Drv(a((1s))
Thus, the loss function gap can be bounded by the total variance distance between the two policies, i.e.,
27 AR (1))(
[C(0) - -
With Pinsker's inequality, we can express the upper bound with the KL divergence between the two policies:
2% [C(0) 2(0) S AB ( b(-1s)
B DETAILS OF RLx2 WITH TD3 AND SAC
In this section, we provide the pseudo-codes of instantiations of RLx2 on TD3 and SAC in Al- gorithm g and Algorithm 5) respectively. We emphasize that RLx2 is general sparse training framework for off-policy DRL and can applied training other DRL algorithms apart from TD3 and SAC, with sparse networks from scratch. Below, we first illustrate the critical steps of RLx2 in Algorithm using TD3 as the base algorithm.
Topology evolution is performed in Lines 15-17 and Lines 20-22 in Algorithm [ Specifically, we first calculate the sparsity of each layer according to target sparsity the total model at initialization. The sparsity of each layer is fixed during the training. And the Padös-Réény) strategy, which is introduced in Mocanu et al. (2018), to allocate the sparsity to each layer. For sparse network with L. layers, this strategy utilizes the equations below:
(1 (1- 5) 10, (1-SS)
1=1
1+
1- = 1 O
where S is the target sparsity of the model, S is the sparsity of the 1-th layer, is the input di- mensionality of the 1-th layer, O is the output dimensionality of the I-th layer, and & a constant. The motivation of this strategy is that layer with more parameters contains more redundancy. As
19

---

Published as a conference paper at ICLR 2023
result, it can be compressed with higher sparsity. The topology evolution update is performed every Am time steps. Definitions of other hyperparameters related to topology evolution are listed in Algorithm[]in the manuscript.
Buffer capacity adjustment is performed in Lines 7-9. This adjustment conducted every Ab step, with detailed procedure shown in Algorithm 3
Multi-step TD target computed in Lines 10-13. We found that using a multi-step TD target the early stage of training may result in poor performance because the policy may evolve quickly, which results in severe policy inconsistency. Thus, we start the multi-step TD target only when the number of training steps succeeds pre-set threshold To- As mentioned in Section 4.2.1 target and multi-step TD target in TD3 are computed as:
the one-step TT)
T T 7
T, Σ rznh 7" ($t"""" (824n);;)
Note that the calculation of the multi-step TD target in SAC is slightly different from that TD3. Specifically, the one-step TD target for SAC is computed
T1 re + (1((= 0++1: 0) log ग ( (1+1)),
where 8e②1 (-| and the n-step TD target for SAC is computed by
n-1
Tn 7 Ar + 7 "Q ni ; 0) a
log n (0r+kt1/S
k=0
k=
where 0++++1 T- (++++1), 0,1. , 1. Due this difference, will see later in Section C.3 that the resulting FLOPs are slightly different for TD3 and SAC.
Besides, Algorithm also gives the detailed implementation of RLx2 with SAC, where topology evolution is performed in Lines 15-17 and Lines 20-22, buffer capacity adjustment performed in Lines 7-9, and the multi-step TD target is computed in Lines 10-13.
Algorithm 4 RLx2-TD3
I: Initialize sparse critic network Qe, Qa, and sparse actor network Te with random parameters 61, 02, and random masks Ma,, Mo,. Ms with determined sparsity "(")),(<e
2: 01 <- 0, @ Ma 02 <- 02 @ Maz, <- &o @ Mc. // Start with random sparse network
3: Initialize target networks 0g +- 01, 0g O2, Ø
Initialize replay buffer B.
for t= to T do
5: Select action with exploration noise a ~ Tp(s:) + e NNO, σr) and observe reward rz and new state
8141
6: Store transition tuple (st, Tt, St+1)
7:
if mod A. then
Buffer capacity adjustment
9:
end if // Check the buffer periodically
10: Set N = temporarily if To // Delay use multi-step TD larget
11:
Sample mini-batch of B multi-step transitions (s;, ae, Ts; 8441, 0:41, 814 from
<- no. (S++N) e, clip(N (0, ,): cc
13:
Calculate multi-step TD target y target
=k=0 T + o minj=1.2 Qo (8i+N, a) // multi-step TD
14:
Update critic networks 0g + XVo, 3 Evv Qo a,)) for 1,2 if mod Am then
15:
16:
Topology Evolution (Q.,) for j = 1,2 end // Update the mask of critic periodically
17:
18:
if l mod d = 0 then
19:
Update actor network t/d mod A. then Topology Evolution
AVo 20
20:
21:
22:
end if // Update the mask of actor periodically
23:
Update target networks:
7)0', T (1 T) 0, + 0% @ Mo. ø' o' @ M // Target networks are also sparsified with same mask
0. 70,
24:
end if
25:
end for
20

---

Published as a conference paper at ICLR 2023
Algorithm 5 RLx2-SAC
1: Initialize sparse critic network Qa, Qa, and sparse actor network ne with random parameters 01, 82. and random masks Ma, .Ma. MA with determined sparsity S(o), g
2: 01 <- 01 @ Mo,- 02 +- 02 @ Ma <- Φ Mp. // Start with random sparse network
3: Initialize target networks 8y +- 01, 02 02. Initialize replay buffer B.
to 7 do 5: Select action d, ~ (s1), and observe reward ra and new state $141
for
6: Store transition tuple (st, rt, $141) 1 in B
7: if mod A. 0 then
Buffer capacity adjustment
9:
end if Check the buffer periodically
10:
Set N = temporarily t < To // Delay to use step TD targer
11:
Sample mini-bateh of B multi-step transitions (s;, 04, Pi, 8.+1, 0i+1, Si+N) from B
12:
d.+++1 TT-1)(-441))
0.1,...,NN
13:
Calculate multi-step TD target multi-step target
E r + + min, Q0 BG=,,a.-NV x) Ex + (o
14:
Update critic networks 0 + 03 2V Ev - (s,, (o,(si, for = 1,2 if t mod Am = 0 then
15:
16:
Topology Evolution( Qo,) for 1,2 end if // Update the mask of critic periodically
17:
18:
Update actor network < AV L 1 Qo, (st,
19:
if /d mod Am then
20:
Topology Evolution(=.)) end // Update the mask of actor periodically
21:
22:
Automating Entropy Adjustment: a +- a XV El- T(a]ss)) - -aH)
23:
Update target networks:
B. + T0, (1 T) 0 + 0 Ma, // Target networks are also sparsified with the same mask
24: end for
C EXPERIMENTAL DETAILS
We provide more experimental details this section, including the detailed experimental setup, the calculations of model size and FLOPs, and supplementary experiment results.
C.I HARDWARE SETUP
Our experiments are implemented with PyTorch (Paszke et al. (017)) and on 8x P100 GPUs. Each run needs 12 hours for TD3 and 2 days for SAC for three million steps. The code will be open-sourced upon publication of the paper.
C.2 HYPERPARAMETER SETTINGS FOR REPRODUCTION
Table 4
presents detailed hyperparameters of RLx2-TD3 and RLx2-SAC in our experiments.
C.3 CALCULATION OF MODEL SIZE AND FLOPs
We present the details of calculating model sizes and FLOPs in this subsection, where focus on fully-connected layers since the networks used in our experiments are all Multilayer Perceptrons (MLPs). These calculations can be easily extended to convolutional layers or other architectures. Besides, we omit the offset term in fully-connected layers in our caleulations.
C.3.1 MODEL SIZE
We first illustrate the calculation of model sizes, i.e., the total number of parameters in the model. Initially, for sparse network with L fully-connected layers, we calculate the model size as:
M = St)L -=
21

---

Published as a conference paper at ICLR 2023
Table 4: Hyperparameters of RIx2-TD3 and RLx2-SAC.
Category
Hyperparameter
Value
Optimizer
Adam
Learning rate
3 x 10
Discount factor y
0.99
Number of hidden layers (all networks)
2
Number of hidden units per layer
256
Shared
Activation Function
ReLU
Hyperparameters
Batch size B
256
Warmup steps
25000
Target update rate т
0.005
Initial mask update fraction So
0.5
Mask update interval Am
10000
Buffer adjustment interval As
10000
Buffer policy distance threshold Do
0.2
Buffer max size Brax
106
Buffer min size Buin
x 105
Multi-step delay To
3 x 10
Target update interval
2
Hyperparameters for RLx2-TD3
Actor update interval Exploration policy Multi-step
2
(O 0,
3
Target update interval Actor update interval
1
Hyperparameters for RLx2-SAC
Entropy target H. Multi-step
dim (A)
2
where S is the sparsity, 1t the input dimensionality, and is the output dimensionality of the 1-th layer. Specifically, the "Total Size" column in Table in the manuscript refers the model size including both actor and critic networks during training.
For both TD3 and SAC, double Q-learning is adopted, i.e., train two value networks concurrently. Besides, we also use target networks in our implementations as target crities for both TD3 and SAC, a target actor only for TD3. Thus, if we denote MA and Merit as model sizes of actor and model sizes can obtained shown the second column of Table We denote B as the batch size used for training process.
critic, respectively, the detailed calculation
Table 5: FLOPs and model size for RLx2-TD3 and RLx2-SAC.
FLOPs
FLOPs (inference)
Algorithm
Model size
(average of each iteration during training)
RLx2-TD3
2M 4Meriti MAdor + 4.Mrrtic
B x (2.5FLOPSAcor + 8.5FLOPs
FLOPs
RLx2-SAC
B x (5FL.PS + 10FL.P (FF.OPScrrine
FL.OPs
C.3.2 FLOPS
Initially, for sparse network with L fully-connected layer, the required FLOPs for a forward pass is competed as follows (also adopted in Evei et al. (2020) and Molchanov et al. (2019a)):
FLOPs (1- -1)4
(10)
where S, is the sparsity, I the input dimensionality, and O is the output dimensionality of the 1-th layer. Denote FL.OPs and FI.O FL.OPscric the FL.OPs required in forward pass of a single
22

---

Published as a conference paper at ICLR 2023
actor and critic network, respectively. The inference FLOPs is exactly FLOPSActor shown in the last column of T As for the training FLOPs, the calculation consisted of multiple forward and backward passes in several networks, which will be detailed below.
In particular, we compute the FLOPs needed for each training itcration. Besides, we omit the FLOPs of the following processes since they have little influence the final result.
(i) Interaction with the environment. Each time the agent decides an action to interact with the environment takes FLOPs as ALOPs which is much smaller than the FLOPs need for updating networks as shown in Table 5 since B >> 1.
(ii) Updating target networks. Every parameter in the networks is updated as 0' 70 +)0' Thus, the number of FLOPs here is the same as the model size, which is also negligible.
(iii) Topology evolution and buffer capacity adjustment. These two components performed every 10000 steps. Formally speaking, the average FLOPs of topology evolution is give by Bx 2FI OPswar for the actor, and B critic, respectively. The FLOPs of buffer capacity adjustment is 8B FL.OPsaxx where 8B is because An that we use the oldest &B transitions compute the policy distance. Thus, they are both negligible.
(1-St)))Am
AFI AFLOOPs for the critic, where S(a) and ((c) are the sparsity of actor and (1 SS)))))m Am
Therefore, we focus on the FLOPs of updating actor and critic. The average FLOPs of updating actor and critic can be given as:
FLOPs FLOP
FLOPSsupatee Supdate
.ritic critic + FLOP Supdate_.ctorr
(11)
where d is the actor update interval (2 for TD3 and 1 for SAC in implementations). Next calculate the FLOPs of updating actor and critic, i.e. FLOPS critic and FLOPSupdatee actor- focus on TD3, while that for SAC is similar.
We first
Training FLOPs Calculation in TD3
(i) Critic FLOPs: Recall the way update critic (two critics, 01 and 02) in TD3 is given by
0 0, AV B ETT Q(s,,,,;;0))
(12)
for j 1,2, where B is the batch size, n-step TD target
N 1
Tn
r4tk + nM min Q(s++N,;; 1-1
(13)
k=0
and 69 refers to the target network.
Subsequently, we can compute the FLOPs of updating critic as:
FLOPs Supdtttttte.ttticcc critic = FLOPSTD.tame target FLOP
FLOOScompute Scompute_Jos + StL.OPPPekaaddppps..
(14)
where FLOOPTTD.aree Larget- FLOPScompute Scomutte. loss. and FL.OPSackkwand.pp refer to the numbers of FLOPSs in computing the TD targets in forward pass, loss function in forward pass, and gradients in backward pass beackward-propagatiin respectively. By Eq. (12) and Eq. (13) we have:
FLOPSTD.tom sarget sarget = B (FLOPSAcoo + 2FLOPSCriie)
(15)
Fl.OPScompate.. loss = B 2FL.OPScHte-
For the FLOPs of gradients backward propagation, ELOPShackward. pass we compute it two times the computational expense of the forward pass, which is adopted in existing literature (Evci et al.) 2020 i.e..
FLOP Spackward, pass B x 2 2FLOPs
(16)
where the extra factor 2 comes from the cost of double Q-leanning.
Combining Eq. 14 Eq. 15 and Eq. (16). the FLOPs of updating critic in TD3 is:
FLOPs FLOPs Supdate.critic
B x (FLOPS Actor + SFLOPSc
(17)
(ii) Actor FLOPs: Recall the way to update actor (parameterized by Φ) in TD3 is given by
レ x- AVo(
Σ (0:0));;81);
B
23

---

Published as a conference paper at ICLR 2023
where 01 refers to critic network. Subsequently, compute the FLOPs of updating actor
Pupdttpdetcc FLOP computee.Jss + FL.OP baakwaarrrdrsssppsss.....
(18)
and similar to the calculations of updating critic, we have:
FLOPSSoompte Scompute loss B x PActr FLOPscatte))
(19)
FLOPs
PhoPShackaand Sbackardd.pas
B x 2FLOPS Acctor
Combining Eq.
Eq. 18
and Eq.
19 the flops of updating actor in TD3 is:
FLOPS Supdate actor
B x (3FLOPS Actor + FLOPSCritie))
(20)
Training FLOPs Calculation in SAC
(i Critic FLOPs: Calculations of FLOPs for SAC are similar to that in TD3. The way to update the critic in SAC is:
8 AVo, B
E(T, Q0, a:))
for 1, where B is the batch size, n-step TD target
N-1 Tn > rr4k + N min Qo, (S++ a++N) a log π (
1=12
and 8 refers the target network. Note that the way compute multi-step TD target in SAC is slightly different from which in TD3, we have:
FLOPSTD.targe = x (2FLOPsActor + 2FLOPschtic)
(21)
Other terms for updating critic the same those Eq. 14 for TD3. Thus, the FLOPs of updating critic in SAC can be computed by
FLOPSupdaae..rtinnn = B (2FLOPsActor + 8FLOPs
(22)
(ii) Actor FLOPs: The way update the actor in SAC is:
Φ - - - -
Σ min Qo (8 (;),
B 1=1
where 8 refers to critic network. Subsequently, have:
FLOPScompute Scompute
B x (FLOPS + 2FLOPSCrtte.
(23)
The backward pass FL.OPs is the same as that in TD3, i.e.
Stackwawarrd pass B x 2FLOPS Actor-
(24)
Thus, the FLOPs of updating the actor in SAC is:
FLOPSupdate.acto = (3FLOPSActor + 2FFLOPccite).
(25)
Thows the relative average FLOPs of each iteration for different algorithms, where the FLOPPs of training a sparse network without any of these three methods is set to 1x. The sparsity is set to the average sparsity in different environments. The additional computations induced by topology evolution and dynamic buffer are negligible. Using multi-step TD learning also does not increase computations in TD3, and only introduces small extra computation SAC (< 5%), analyzed above.
Table 6: Relative FLOPS of different methods
Algorithm
Tiny
SS
SET
RigL
RLx2
TD3
1.00000x 1.00000x
1.0000xx 1.00000x
1.0000xx 1.00000x
1.00067x 1.00033x
1.00072x
SAC
1.04432x
24

---

Published as a conference paper at ICLR 2023
C.4 COMPARISONS BETWEEN DIFFERENT MASKS IN OTHER ENVIRONMENTS
Figure 8 shows the improvement of the learned sparse network topology by robust value learning. We see that in addition Ant-v3, significant improvements also achieved in Hopper-v3 and Walker2d-v3. The only exception HalfCheetah-v3. We hypothesize that this environment is casier for the agent to learn (as the performance improves much faster in the early stage than in the other three environments), and good mask can be found comparatively casier. It is an interesting direction for future work to systematically analyze this problem.
2500
Righ
3500
mark
3000
2000
8000
Avstage Ilftum
W
1500
Average
2000
m
4000
1,000
1000
2000
500
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
3.5
2.0
2.5
3.0
0.0
0.5
3.0
1.5
2.0
2.5
3,0
sleps
stapss
staps
(a) HalfCheetah
(b) Hopper
(c) Walker2d
Figure 8: Comparison of different sparse topologies learned in HaliChectah-v3, Hopper-v3, and
Walker2d-v3.
C.5
TRAINING CURVES COMPARATIVE EVALUATION SECTION 5.
Figure and Figure TO show the training curves of different algorithms in four MuloCo environ- ments. RLx2 outperforms baseline algorithms on all four environments with both TD3 and SAC.
Dense
Tiny Dense
Static Sparse
SET
RigL
RLx2
4000
3500
10000
3000
Average ratur
8000
2500
2000
1500
4000
1000
500
2000
0.0
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
steps
(a) HDlfhhalChe with actor sparsity 90% and critic sparsity 85%
(b) TD3-Hopper with actor sparsity 98% and critic sparsity 95%
5000
4000
5000
Tverager return
4000
3000
2000
2000
1000
1000
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
stops
106
steps
Lue
(c) TD3- TD3-Walker22 Walker2d with actor sparsity 97% and critic sparsity 95%
(d) TD3-Ant with actor sparsity 96% and critic sparsity 88%
Figure 9: Training processes of RLx2-TD3 on four MuJoCo environments. The performance is calculated as the average reward per episode over the last evaluations the training.
25

---

Published as a conference paper at ICLR 2023
Dense
Tiny Dense
Static Sparse
SET
RigL
RLx2
4000
3500
10000
Averigge return
6000
2500
2000
6000
4000
1000
500
0.0
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Time steps
steps
(a) SAC-HalfChoctah with actor sparsity
(b) SAC-Hopper with actor sparsity 98% and critic sparsity 95%
90% and critic sparsity 80%
5000
6000
4000
5000
Average return
3000
3000
2000
2000
1000
1000
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Time steps
:06
Time stops
106
(c) SAC- SAC-Walker2d with actor sparsity 90% and critic sparsity 90%
(d) SAC-Ant actor sparsity 90%
critic sparsity 75%
Figure 10: Training processes of RLx2-SAC on four MuJoCo environments. The performance is calculated the average reward per episode the last 30 evaluations of the training.
C.6 STANDARD DEVIATIONS OF RESULTS IN TABLE
Table shows the performance different algorithms on four MuJoCo environments with standard deviations. Each result is calculated on 8 random seeds. RLx2 does not lead to larger variance with topology evolution.
Table 7: Results in Table 2 with standard deviations SS(%)
Alg.
Env.
Tiny(%)
SET(%)
RigL(%)
RLx2(%)
Hal.
86.3+11.6
17.1
92.6:16.1
90.8:16.3
99.8-14.7
Hop.
64.5+31.1
67.7+:2.00
66.5+33.0
90.6=14.9
97.0=17.6
TD3
Wal.
60.8+13.1
42.9427.8
39.3126.1
35.7=24.3
98.1+15.2
Ant.
16.5+11.9
49.6+
62.5++1.8
68.5+15.0
103.9+11..9
Avg. 16.9
59.3420.0
65.2420.0
11.4
99.7+12.4
Hal. 95.0+12.7
75.418.7
94,8+1600
89.818.4
102.243.22
Hop.
89.1+22.0
81.61130.6
103.9+15.7
110.0+10.4
109,7+19.9
SAC
Wal.
73.8+11.4
83.4+14.8
95.8+13.1
81.914.3
103.2+13.3
Ant
49.6+
49.34-31.0
79.8+20.88
90.9+21.3
105.6++24.7
Avg.
76.9
72.4121.2
93.6113.9
93.2=11.1
105.2
Avg.
67.0+15.5
65.9++20.6
79.4+17.0
82.3+13.1
101.8+113.9
26

---

Published as a conference paper at ICLR 2023
C.7 SCPPLEMENTARY RESULTS FOR ABLATION STUDY
Sable&Sshww the performance of four environments with different buffer capacities. Consistent with the results in Section 5.2 buffer that either small large result in poor performance.
Our dynamic buffer outperforms buffers with fixed capacity.
Table 8: RLx2-TD3 with different buffer capacity.
Capacity 5 X 10*
Capacity
Capacity
Capacity
Environment
1 x 10"
2 10
1 x 106
Unlimited
Dynamic
HalfCheetah
84.5%
86.2%
93.7%
95.6%
88.2%
99.8%
Hopper
79.9%
94.1%
91.4%
94.5%
89. 1%
97.0%
Walker2d
77.9%
94.7%
90.9%
90.2%
79.2%
98.1%
Ant 75.6%
75.1%
83.4%
80.0%
72.6%
103.9 %
Average 79.5%
87.5%
89.9%
90.1%
82.3%
99.7%
C.8 SENSITIVITY ANALYSIS FOR HYPERPARAMETERS
In this section, provide the detailed sensitivity analysis for new hyperparameters used in RLx2, including initial mask update fraction ç, mask update interval Am. buffer adjustment interval As, buffer policy distance threshold Do, and multi-step delay To-
Initial mask update fraction Table shows the performance with different initial mask update fractions (denoted as () in different environments. We also include the special case of keeping the mask static, i.e., 0. From Table find that the sensitivities of the initial mask update fraction among different environments are similar. Besides, RLx2 achieves better performance with a large value of the initial mask update fraction. There is apparent performance degradation even if is set to 0.9, which may be due to the update fraction annealing scheme.
Table 9: Sensitivity analysis on initial mask update fraction.
Environment
Static Sparse
(= 0.1
Ç 0.3
( 0.5
0.7
= 0.9
HalfCheetah
86.1%
93.5%
96.2%
99.8%
100.5%
103.5%
Hopper
84.2%
98.2%
103.7%
97.0%
97.0%
92.4%
Walker2d
83.8%
98.2%
96.4%
98.1%
104.3%
101.4%
Ant
80.2%
79.1%
83.8%
103.9%
101.9%
102.8%
Average
83.6%
92.3%
95.0%
99.7%
100.9%
100.0%
Mask update interval Table 10 shows the performance with different mask update intervals (de- noted as Am) in different environments. We also include the special case of never updating the mask, i.e., Am = co. From Table I0] we find that the sensitivities the mask update interval among different environments reduces the performance since adjusting the mask too frequently may drop the critical connections before their weights are updated large values by the optimizer. On the contrary, large mask update interval reduces the impact caused by topology evolution but degrades to training a static sparse network. In general, a moderate value of 10a favoured.
similar. Table I0 also shows that small mask update interval
Table 10: Sensitivity analysis on mask update interval.
Am
Am
Am
Am
Am
Static Am
Environment
1 103
3 x 10
1 10
3 x 10a
1 x 105
HalfCheetah
96.4%
93.7%
99.8%
100.8%
94.6%
86.1%
Hopper
97.1% %
98.7%
97.0%
91.8%
97.8%
84.2%
Walker2d
98.2%
103.7%
98.1% 1%
97.7%
91.8%
83.8%
Ant
83.5%
89.6%
103.9%
95.3%
94.2%
80.2%
Average
93.8%
96.4%
99.7%
96.4%
94.6%
83.6%
27

---

Published as a conference paper at ICLR 2023
Buffer adjustment interval Table shows the performance with different buffer adjustment in- tervals (denoted as Ab) in different environments. We also include the special case of never adjusting the buffer capacity, i..., A, co. From Table II] find that the sensitivities of the buffer adjust- ment interval among different environments are similar. We find the performance is not sensitive to the buffer adjustment interval Ag. We only observe apparent performance degradation when the buffer adjustment interval is too large such that the policy distance cannot be reduced promptly.
Table 11: Sensitivity analysis on buffer adjustment interval.
A
A
A
A.
As
No adjustment As = 00
Environment
1 x 103
3 x 10a
1 x 10*
3 x 10*
1 x 106
HalfCheetah
99.6%
100.6%
99.8%
99.5%
100.5%
95.6%
Hopper
96.7%
94.8%
97.0%
93.8%
94.8%
94.5%
Walker2d
96.6%
103.1%
98.1%
101.2%
96.1%
90.2%
Ant 97.5%
92.4%
103.9%
103.3%
79.7%
80.0%
Average 97.6%
97.7%
99.7%
99.5%
92.8%
90.1%
Buffer policy distance threshold Table 12 shows the performance with different buffer policy distance thresholds (denoted as Do) in different environments. We also include the special case of never adjusting the buffer capacity, i.e., Do co. From Table [2] we find that the sensitivities of the buffer policy distance threshold among different environments are similar. It is shown that a very small threshold may reduce the performance, while the dynamic-capacity buffer helps improve the performance in wide range the threshold, especially when Do = 0.1 or 0.2.
Table 12: Sensitivity analysis buffer policy distance threshold.
No adjustment Do
Environment
Do = 0.05
Do = 0.1
Do = 0.2
Do 0.3
Do = 0.5
HalfCheetah
90.8%
102.9%
99.8%
98.8%
99.2%
95.6%
Hopper
99.7%
98.0%
97.0%
96. 1%
93.5%
94.5%
Walker2d
85.3%
94.8%
98.1%
90.0%
89.5%
90.2%
Ant
97.9%
100.2%
103.9%
85.9%
83.5%
80.0%
Average
93.1%
99.0%
99.7%
92.7%
91.4%
90.1%
Multi-step delay Table 13 shows the performance with different multi- delays (denoted as To) in different environments. We also include special cases of no delay, i.e., To O, and delay all the time, i.e., Tp = oc. From Table we find that the sensitivities the multi-step delay among dif- ferent environments can be improved by using delay mechanism in the early stage of training, also outperforming the one-step scheme. In addition, the performance is not very sensitive to the multi-step delay To. find that performance gains with different delays are similar.
similar. Compared the multi-step method without delay, the performance
Table 13: Sensitivity analysis on multi multi-step delay.
No delay
To
To
To
To
To
one-step Ti 0c
Environment
To 0
1 105
2 10
3 103
4 105
5 x 10
HalfCheetah
87.1%
95.7%
99.2%
99.8%
100.8%
95.5%
96.5%
Hopper
100. 1%
104.5%
101.0%
97.0%
96.8%
100.3%
77.9%
Walker2d
100.1% 1%
98.1%
97.5%
98.1%
102.1 1%
99.2%
73.9%
Ant
83.3%
81.6%
90.7%
103.9%
103.3%
98.2%
103.9%
Average
92.7%
95.0%
97.1%
99.7%
100.1%
98.3%
88.1%
28

---

Published as a conference paper at ICLR 2023
C.9 ADDITIONAL RESULTS IN HUMANOID-V3
In this subsection, we investigate the effect of RLx2 in Humanoid-v3, one of the control tasks from MuloCo. Humanoid-v3 is considered relatively complex due the high input dimensionality (376). Thus, apart from the standard 256 neurons in each hidden layer (same as other environments), we also train a dense model with 1024 neurons in each hidden layer. As shown Table 14, the model with more hidden parameters (1024 hidden dimensions) does not achieve a better performance than small model (256 hidden dimensions). This implies that the latter one seems to have sufficient capacity for the control task in Humanoid-v3.
Table 14: RLx2-TD3 in Humanoid-v3. We train two dense models with 256 and 1024 neurons in each hidden layer, respectively. We apply RLx2 and baseline algorithms in these two dense models. The table shows the average returns of each algorithm.
Model
256 hidden neurons
1024 hidden neurons
Dense
5721.8 + 173.0
5097.3 228.9
Tiny
3835.0 + 762.5
3835.0 + 762.5
SS
4248.4 + 1588.7
4916.2 1058.5
SET
5536.4 = 152.0
5829.9 + 295.5
RigL
5070.5 = 335.2
5147.0 + 493.3
RLx2
5482.0 = 603.0
5899.1 + 520.6
Actor Sparsity (%)
92
99
Critic Sparsity (%)
85
98
Number of Parameters
63000
72000
In addition, Table 14 shows the performance of the sparse models trained with RLx2 and other baseline algorithms, where RLx2 succeeds in training highly sparse model in Humanoid-vā with performance degradation less than 5%. In particular, RLx2 can achieve sparsity of around 90% in small model with only 256 hidden neurons, showing its robustness for complex control tasks. Also, RLx2 outperforms most of the baseline algorithms. Although SET shows comparable performance with RLx2, RLx2 shows much higher sample efficiency than SET according to Figure 11
6000
6000
5000
5000
return
Average return
4000
4000
3000
3000
Tiny
2000
2000
SS
SET
1000
1000
RigL
RLx2
0.5
1.0
1.5
2.5
3.0
0.0
0.5
1.5
2.0
2.5
3.0
Time steps
Time steps
(a) 256 hidden neurons
(b) 1024 hidden neurons
Training processes of RLx2-TD -D3 in Humanoid-v3, The performance of each method is calculated as the average reward per episode over the last evaluations the training.
Figure 11:
When applying RLx2 to larger models with 1024 neurons in each hidden layer, we find that it still performs well with extremely high sparsity (around 99%). We calculate the number of parameters of the two sparse models and find they are very close, as shown Table It suggests that RLx2 is an effective way train a sparse model with the least parameters, and is robust the hidden width of the dense model counterpart.
29

---

Published as a conference paper at ICLR 2023
C.10 SUPPLEMENTARY RESULTS FOR INVESTIGATION THE LOTTERY TICKET HYPOTHESIS
We provide additional experiments with TD3 for investigation of the lottery ticket hypothesis (LTH) in the other three environments, including HalfCheetah--33, Hopper-v3, and Walker2d-v3, As shown Figure winning tickets fail achieve the same performance the dynamic topology in the reinforcement learning setting, while they all perform well under behavior cloning. This shows the necessity for reinforcement learning to adjust the network structure during the training process.
12000
10000
10000
Avrerage return
Average return
8000
8000
6000
6000
4000
Deree
Desss RuK
RL2
4000
2000
Ransom Tico:: Wiringg Tickt
Random Tidke Winnig Ticet
1.0
2.5
3.0
1.5 Time steps
2.5
3.0
Time steps
(a) RL: HalfChectah-v3 with actor sparsity 92% and critic sparsity 85%.
(b) BC: HalfChectah-v3 with actor sparsity 92%.
4000
3500
3000
3000
Average return
2500
2500
Dense
Average
2000
1500
1500
1000
1000
500
0
0.5
1.0
1.5
2.0
2.5
3.0
1.0
1.5
2.0
2.5
3.0
Time steps
Time steps
(c) RL: Hopper-v3 with actor sparsity 99% and critic sparsity 95%.
(d) BC: Hopper-v3 with actor sparsity 99%.
5000
4000
Average return
3000
3000
2000
2000
Donse
Dense
RLx2
RUX2
Random Ticket
Randum Tibee
Winning ckot
Waninn Tickst
0.0
0.5
1.0
0.5
1.0
2.0
Time steps
Time
RL: Walker2d-v3 with actor sparsity 98% and critic sparsity 96%.
BC: Walker2d-v3 with actor sparsity 98%.
Comparisons of different methods for training sparse DRL agent, where RLx2 is in- stantiated with TD3. Here, "RL" stands for the reinforcement learning setting, and "BC" stands for behavior cloning setting.
Figure 12:
30

---

Published as a conference paper at ICLR 2023
C.11 VISUALIZATION OF SPARSE MODELS
In this section, we show visualizations of the sparse networks obtained under RLx2 in our experi- ments. Note that each layer in the sparse network in our implementation is bound to a binary mask Mg, i.e., 02 = 0g @ Mj. for the !-th layer. the rest of this section, we investigate the property of the binary masks with visualization.
By visualizing the final mask after training, we find RLx2 drops redundant dimensions in raw inputs adaptively and efficiently. Specifically, Figure 13 shows the distribution of connections each state dimension in Ant-v3. The figure shows that the resulting topology almost has no connections for input dimensions 28 111, and the connections heavily concentrate on the first 27 dimensions. This observation indicates that the topology evolution method drops redundant dimensions in raw inputs adaptively and efficiently. A similar observation also been made Vischer al. (2022), where it is shown that by iterative magnitude training, input dimensions irrelevant to the task pruned entirely to help yield a minimal task-relevant representation.
Redundent
20
40
Figure 13: Distribution of connections to each state dimension Ant-v3. Connections mainly concentrate on the first 27 dimensions.
In Figure 14 we further show visualizations of the raw binary masks (i.e., matrices with items representing neuron connections) of the actor in Ant-v3, where black dot denotes an existing connection between the corresponding neurons, and a white point means there is no connection. As mentioned above, only very few connections are kept for the redundant input dimensions. We also find that the connections in the hidden layers tend to concentrate on a subset of neurons, showing that different neurons can play different roles representations.
31

---

Published as a conference paper at ICLR 2023
(a) Original mask the first layer
(b) Final mask (256 111)
first layer
(256 x 111)
(c) Original mask of the second layer (256 x 256)
(d) Final mask of the second layer (256 256)
(e) Original mask of the third layer (1 x 256)
:
(I) Final mask the third layer (1 256)
Figure 14: Visualization of the binary masks of the actor
32


Reviews:
[{"review_id": "uzAeGXkam4", "reviewer": "Reviewer_Qrdc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Sparse neural networks that have an order of magnitude fewer parameters than a dense network without a significant drop in performance can greatly reduce computational requirements of reinforcement learning (RL) agents, allowing their deployment on resource-limited devices. Different sparse network topologies can have drastic differences in performance. A method for the training of sparse networks from scratch, known as RigL, learns the topology throughout training by dropping low-magnitude weights and growing connections that have large gradients. This paper extends RigL to the deep RL domain by adding two techniques for making the value function estimation more robust. One technique is using multi-step temporal difference targets instead of one-step ones. The other is dynamically changing the size of the replay buffer if its stored actions get too far from the current policy. The paper demonstrates with examples that robust value learning is essential for improving the performance of RigL since, in part, the value function affects the gradient-based topology learning of RigL. Experiments with RLx2 using SAC and TD3 algorithms on four mujoco environments show significant model compression with minimal loss in performance.", "summary_of_the_review": "The paper establishes that having a robust value estimator is important when learning the sparse network topology using RigL in Deep RL. The method trains a sparse network from scratch and achieves significant model compression with minimal loss in performance. Claims are backed up with experiments. The two introduced techniques are simple yet effective for applying RigL to Deep RL. The paper can be improved by adding sensitivity analyses for the new hyper-parameters as discussed previously.", "strength_and_weaknesses": "The RigL method learns a sparse topology from scratch. Unlike other model compression methods, there is no need for pre-training a dense network, which markedly reduces the computational requirements of both training and inference. RLx2 could therefore greatly benefit online learning problems on resource-constrained devices. I appreciate that the author has validated the importance of robust value estimation and its effect on the topology learning of RigL.\n\nThere are two main weaknesses in the paper. The first is the lack of proper sensitivity analysis for some of the new algorithm hyper-parameters. Although the hyper-parameters are the same for all four environments, it is unclear how sensitive the performance is to each hyper-parameter and how the sensitivity differs between environments. These hyper-parameters include the initial mask update fraction, mask update interval, buffer adjustment interval, buffer policy distance threshold, and multi-step delay.\n\nThe second weakness relates to the computational complexity of the algorithm. The topology learning of RigL consumes O(N logN) time with N as the number of weights in the corresponding dense network. Although the topology is updated every 10000 steps and the cost may be negligible compared to the main RL algorithm in these experiments, 10000 is still a constant. With the exponentially increasing number of parameters, in the future the computational requirement of the topology learning step might begin to dominate and grow faster than the main algorithm, making it less scalable. It would be great if the dropping and growing of connections could be done locally for each connection instead of requiring a global sort. This issue concerns the RigL method adopted by this paper and not the techniques introduced in this paper for making the value estimate robust.\n\nHere are some more questions and comments:\n* What do the shaded areas in the plots represent?\n* Did you experiment with using multi-step TD targets from the beginning? What did you observe?\n* What happens if you do not anneal the update fraction, but keep it constant at a smaller value?\n* Should the two robustification techniques be applied to the original algorithms even when not using RigL?\n* “We also show in Appendix C.2 that the performance of RLx2 is insensitive to the policy threshold.” I do not see any sensitivity plots in C.2.\n* “Upon any update to \\phi and appending new transitions to…” Is this sentence correct? Was the buffer adjustment not done every \\Delta_b steps?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UisXoZFdAJC", "reviewer": "Reviewer_i62R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the first work to train a sparse DRL agent from scratch. The major contributions are that the robust value estimation and effecient topology evolution are important to train a sparse DRL agent, and this paper proposes to leverage the gradient for network topology evolution and multi-step TD-target with dynamic buffer for robust value estimation. Experimental results are presented using two off-policy TD learning DRL models, and show the best performance over existing approaches.", "summary_of_the_review": "Beyond the technical side, this paper is of good readability (well organized, empirically grounded motivations, etc). However, in terms of the technical novelty, the proposed solutions are a little bit trivial in sense. ", "strength_and_weaknesses": "Strengths:\n- Training ultra-sparse DRL model from scratch is important while underexplored topic in practice.\n- The contributions of this paper are clearly summarized, and the motivations are grounded in empirical observations.\n- Experimental results show significantly better performance than the listed baselines.\n \nWeaknesses:\n- Since the one-step and multi-step TD-targets are used in a hybrid way in training, it's not a proper claim that a multi-step TD target could overcome the issue of inaccurate value estimation from a sparse network. \n- Also, from the Eq. (4), it's unclear why the multi-step TD-targets could overcome the unreliable and inaccurate value estimations.\n- The dynamic size of replay buffer is controlled by a pre-defined threshold on the policy distance, which is kind of heuristic in practice. \n- Moreover, transitions are actually **randomly** selected from the replay buffer (to keep samples iid) rather than using the oldest top-K transitions. Therefore, the policy distance measure (Eq. (6)) may not effectively evaluate the inconsistency between behavior policy and current target policy.\n- From Table 3, it's curious to me that the performance degrades when more than 3 steps are used. On the Hal. and Ant. benchmarks, the best number of steps is 2. Since it's a consistent observation that the performance would degrade with too many steps, this paper should present more in-depth discussion to explain this phenomenon.\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1QpvdM95Ab", "reviewer": "Reviewer_B2aD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies sparse training with dynamic sparsity in the context of extremely sparse neural network models, trained from scratch, as function approximators for deep reinforcement learning. Consequently, the paper proposes a new sparse training method for deep reinforcement learning, named the Rigged Reinforcement Learning Lottery (RLx2). According with the experimental evaluation on four continuous control task environments, RLx2 seems to perform considerably better than the sparse baseline methods considered. Moreover, it can also outperform in a handful number of cases the dense equivalent.", "summary_of_the_review": "Overall, I believe that the paper has a good level of novelty, with good results, and a good presentation. I am curious to see the authors’ opinion about the comments raised above.", "strength_and_weaknesses": "Strength:\n* Timely, relevant, and under-studied topic, particularly for deep reinforcement learning. If enough efforts are put into it, and if it will become more mature, it has the potential of seriously decreasing the costs associated with deep reinforcement learning. Counterintuitively, as shown in the paper, it also has the potential of improving the state-of-the-art performance in deep reinforcement learning.\n* The proposed method has a good level of novelty.\n* The empirical validation is well-designed. The results show that the proposed method can outperform the baselines.\n* Well written paper\n\nWeaknesses – in my opinion, the paper doesn’t seem to have major flaws. Further, I would raise some points for discussion which are intriguing me:\n* Why the Humanoid environment hasn’t been considered in the experiments? I assume that at this very high sparsity levels, given its large state space, the performance may be far from optimal. Can you consider adding an extra small experiment on Humanoid where you can study two network architectures (one like in the paper and one with more hidden neurons to compensate for the high sparsity regime) for RLx2 and the typical baselines used in the paper?\n* From Table 2, I see that the proposed method (RLx2) consistently improves the dense SAC baseline in terms of performance even when over 90% sparsity is considered. At the same time, on TD3 the gain induced by RLx2 is considerably smaller. Do you have any idea why RLx2 impacts SAC in a different manner than TD3?\n\nOther comments:\n* In table 1, I believe that it is worth also adding the work of Graesser et al., 2022 for a more complete overview.\n* page 4, I believe that “…links with the smallest weights.” is not accurate for negative weights. According with Alg 1, line 9, it is rather about … links with the smallest absolute value of the weights… \n* page 5, “…Here we adopt the methods in (Frankle & Carbin, 2019) for…”. The following procedure is not exactly the one from the mentioned paper. I believe that it would be informative to clarify exactly what was taken from that paper and what was added by you.\n* page 6, Dynamic capacity buffer – According with the description, the buffer size (capacity) is always increasing? Is my understanding correct?\n* page 7, “…SET (Bellec et al., 2017),…” is not correct. SET was proposed by Mocanu et al., July 2017, if referring to the first arxiv version, or 2018 if referring to the year of formal publication.\n* page 8, “…which replace the topology evolution scheme in RLx2 with Tiny, SS and SET, while keeping…” may be ok, but it is not sufficiently clear to me what are the changes as some of those methods are static and some are dynamic (evolves). Perhaps you can clarify it better in the paper.\n* page 9 – Conclusion – a fair paper, with a good level of self-criticism, discusses also a bit of the limitations of the proposed methodology, while possibly indicating future work directions. These are missing in this paper.\n * page 25 – Appendix C.5 – Figures 9 ad 10 – it may be useful for the reader to see also somehow the sparsity levels in those plots without the need of scrolling back to Section 5.1\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-Yg3J9WeY5R", "reviewer": "Reviewer_7aAy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn an efficient sparse network for reinforcement learning (RL). It learns the sparse network from scratch instead of relying on knowledge distillation or pruning. The proposed method is designed based on the gradient-based topology evolution criteria. Experiments show that the proposed method can learn very sparse networks with satisfying performance. ", "summary_of_the_review": "The major contribution is about how to integrate the RigL based sparse network learning approach into the RL framework, which limits the novelty. But the work conducts very detailed design and studies and analyses of the sparse network learning for RL. Many learning strategies are investigated.  \n\nThe rebuttal addressed my concerns. I am happy for this paper to be accepted. ", "strength_and_weaknesses": "Strength\n- The proposed method is carefully designed and well implemented. It can maintain good performance with very high-level sparsity on different datasets. The proposed method integrates the topology evolution method into the RL task very well. \n- The details of the algorithms and experiments are discussed and analyzed carefully, making the work easy to reproduce. \n- The paper is easy to follow. \n\nWeakness\n- The proposed method is an integration of the RigL method into the RL task and framework. The major contribution is on how to integrate it into the RL framework. This limits the significance of the proposed method, especially the lottery ticket based method for sparse network learning has been studied wildly.\n- Apart from RigL, the authors may consider studying how different sparse network learning methods can help RL in what kind of way. This may generalize the proposed techniques to a more general sparse network based RL framework. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%