Parallel $Q$-Learning: Scaling Off-policy Reinforcement Learning
Authors: Zechu Li
Tao Chen
Zhang-Wei Hong
Anurag Ajay
Pulkit Agrawal
Venue: Submitted to ICLR 2023 (2023)
Keywords: GPU-based simulation
off-policy learning
distributed training
reinforcement learning

Abstract: Reinforcement learning algorithms typically require tons of training data, resulting in long training time, especially on challenging tasks. With the recent advance in GPU-based simulation, such as Isaac Gym, data collection speed has been improved thousands of times on a commodity GPU. Most prior works have been using on-policy methods such as PPO to train policies in Isaac Gym due to its simpleness and effectiveness in scaling up. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, resulting in a much longer wall-clock training time in practice. In this work, we presented a novel parallel $Q$-learning framework that not only gains better sample efficiency but also reduces the training wall-clock time compared to PPO. Different from prior works on distributed off-policy learning, such as Apex, our framework is designed specifically for massively parallel GPU-based simulation and optimized to work on a single workstation. We demonstrate the capability of scaling up $Q$ learning methods to tens of thousands of parallel environments. We also investigate various factors that can affect the policy learning training speed, including the number of parallel environments, exploration schemes, batch size, GPU models, etc.

Under review as a conference paper at ICLR 2023
PARALLEL Q-LEARNING: SCALING OFF-POLICY RE- INFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning algorithms require a long time to learn policies on com- plex tasks due to the need for a large amount of training data. With the recent advances in GPU-based simulation, such Isaac Gym, data collection has been sped up thousands of times on a commodity GPU. Most prior works have used on-policy methods such as PPO to train policies in Isaac Gym due to its sim- pleness and effectiveness in scaling up. Off-policy methods are usually more sample-efficient but more challenging to be scaled up, resulting in a much longer wall-clock training time in practice. In this work, we presented a novel Parallel Q-Learning (PQL) framework that is substantially faster in wall-clock time and achieves better sample efficiency than PPO. Our key insight is to parallelize the data collection, policy function learning, and value function learning as much possible. Different from prior works on distributed off-ppolicy learning, such as Apex. our framework is designed specifically for massively parallel GPU-based simulation and optimized to work on single workstation. We demonstrate the capability of scaling up Q learning methods to tens of thousands of parallel envi- ronments. We also investigate various factors that can affect policy learning speed, including the number of parallel environments, exploration schemes, batch size, GPU models, etc.
INTRODUCTION
Reinforcement learning (RL) has achieved impressive results on many problems: video games (Berner et al.] 2019 Mnih et al. 2015), robotics (Kober et al. 2013, Miki al. 2022). drug discovery (Popova et al. 2018)) and others. A primary challenge in using RL is the need for large amounts of training data. One way to tackle this problem is by improving off-policy RL algorithms (Mnih et al. 2015; Lillicrap et al. 2015) that make better use of data than on-policy algorithms (Schulman et al. 2017) Mnih et al. 2016). Another strategy is to substantially reduce the need for real-world data collection by training in simulation. Recent works have achieved remark- able success in deploying policies trained in simulation to the real-world (Hwangbo et al.] 2019)] OpenAl et al.) 2020 Margolis et al. 2022 Miki et al., (022).. In sim-to-real training paradigm, it is not the amount of training data but the wall-clock time that is a major constraint. Faster train- ing, measured wall-clock time, speeds up the experimentation cycle and unlocks the potential for addressing a broader range of more complex problems that currently take a long time to train.
The need for faster training has been recognized in the literature resulting in several distributed leamingg frameworks (Horgan et al. [2088] Espeholt et al., 2018). Typically these frameworks operate the server scale requiring hundreds or thousands of computers, making them infeasible for most researchers. Many of these computers are used to run multiple copies of "slow" simulator in parallel to feed the learning process. Recent advances in GPU-based simulation, such as Isaac Gym (Makoviychuk et al., | (021I), have mitigated the need for large number of machines for parallel simulation by enabling the parallel simulation of thousands environments on a commodity GPU on single workstation. One natural question ask is, what RL algorithm is apt in such setting? Many prior works (Allshire et al., 2021 Rudin et al., 2022 Chen et al., 2022) use PPO (Schulman et al. 2017) for training agents in Isaac Gym due to its simplicity and easy-to-scull nature.
Intuitively, by virtue of requiring lesser data than on-policy algorithms, off-policy algorithms should reduce the wall-clock time of training. However, naive implementation of off-policy algorithms

---

Under review as a conference paper at ICLR 2023
such DDPG (Lillicrap et al. environments usually requires a much longer training time than PPO, even though these algorithms more sample efficient. We build upon distributed frameworks for Q-learnigg developed and deployed in server-seale settings (Horgan et al., 2018 Nair et al.) 2015) to leverage GPU-based sim- ulation single workstation. We present approach scale up Q-learning Parallel Q-Learning (PQL), that can be deployed on a workstation to leverage thousands of environment simulations run- ning in parallel efficiently. The key factor that boosts the learning speed in PQL is that we parallelize the data collection, policy function update, and value function update as much as possible on a sin- gle workstation. Such parallelization would be non-trivial for on- on-policy algorithms such as PPO as the policy update require on-policy interaction data, which means the data collection and the policy update have to happen in sequence. PQL. outperforms state-of-the algorithms such as PPO in terms of both wall-clock and data efficiency.
2015) SAC (Haarnoja et al. 2018) without making of parallel
We empirically investigate the effectiveness of our method on six Isaac Gym tasks (Makoviychuk eet al. 202I). demonstrating the superior performance of PQL against commonly used state-of- the-art (SOTA) RL algorithms. We also analyze several important factors that can affect learning speed, such as the number of parallel environments, batch size, balancing between parallel processes performing simulation, data-collection and learning, exploration scheme, replay buffer size, number of GPUs, different GPU hardware, etc. Other noteworthy findings are: (i) we empirically found that DDPG performs better than SAC when using a large number of parallel environments. (ii) We can mitigate the need for tuning the nyper-paramete controlling exploration. Overall, while previous distributed learning frameworks were accessible only to researchers with access to server- scale compute, we hope our framework leveraging recent advances in GPU-based simulation will be a useful tool for the broader research community training RL. agents on single workstation.
2 RELATED WORK
Massively Parallel Simulation Simulation has been an important tool for various research fields, including robotics, drug discovery, physics, etc. In the past, researchers have used simulators like MuJoCo (Toorovv al./ 2012), PyBullet (Coumans & Bail 2016) for rigid body simulation. Recently, there has been a new wave of development in GaU-b simulation, such as Isaac Gym (Makoviychuk et al. 2021 GPU-based simulation has substantially improved the simula- tion speed, allowing a massive amount of parallel simulation on single commodity GPU. It has been used in various challenging robotics control problems such as quadruped locomotion (Rudin) eet al. 2022 Margolis et al. 2022), dexterous manipulation (Chen et al. [022] Allshire et al. 2021) With the advent of fast simulation, one can get much more environment interaction data in the same training time before. This poses challenge to the reinforcement learning algorithm in making the best use of the massive amount of data. A straightforward way is to use on-policy algorithms such as PPO, which can be scaled up easily and is also the default algorithm that researchers use in Isaac Gym. However, on-policy algorithms are less data efficient. In our work, we investigate how to scale up off-policy algorithms to achieve both higher sample efficiency as well as shorter wall-clock training time for massively parallel simulation. Our parallel training framework works on a commodity workstation without requiring a big compute cluster.
Distributed Reinforcement Learning Model-free reinforcement learning typically requires large number of environment interactions (low sample efficiency). One way to speed up policy learning is by distributed training. There have been numerous works developing different distributed reinforcement learning frameworks. One line of work focuses on Q-learning methods. Gorila (Nair al. 2015) distributes DQN agents to many machines where each machine has its local envi- ronment, replay buffer, value learning, and uses asynchronous SGD for a centralized Q function learning. Similarly, [Popov et al.] (017)) applies asynchronous SGD to the DDPG algorithm (Lilli) crap et al. 2015). Combining with prioritized replay (Schaul et al. 2015), n-step returns (Sutton 1988). and double-Q learning (Hasseli 2010)) HHrgan et al. (2018) (Ape-X) parallelizes the actor thread (environment interactions) for data collection and uses centralized learner thread for policy and value function learning. Built upon Ape-X. Kapturowski et al. (2018) adapts the distributed prioritized experience replay for RNN-based DQN agents.
Another line of work improves the training speed on policy gradient methods. A3C (Mnih et al. 2016) uses asynchronous SGD across many CPU cores, with each one of them running actor
2

---

Under review as a conference paper at ICLR 2023
Value ? learning
(te, des Beads Tes dess)}
Replay Buffer
V-learner
Enviromments
π
[(+)]
((a)))
((+))
Repley Butfer
Palicy
π
leamning
Actor
P-learner
Figure Overview of Parallel Q Learning (PQL). We have three concurrent processes running: Actor, P-learner, V-learner. Actor collects interaction data. P-learner updates the policy network. V-learner updates the Q functions.
learner single machine. Babacizzddh al. (016)) develops hybrid CPU/GPU implementation of A3C, but it can have poor convergence due to the stale off-policy data being used for the on-policy off-policy correction scheme (V-trace) to mitigate the lagging issue between the actors and learners in distributed on-policy settings. [Epechol| al. (2019) further improves the IMPALA training speed by moving the policy inference from the
update. [Eseholt et al. (2018) (IMPALA) introduces
actor to the learner. Clemente et al. (2017) parallelizes the environments for synchronous advantage actor-critic. Heess al. [201 proposes distributed version of PPO (Schulman et al., 2017) for
training various locomotion skills in diverse set of environments, Wijmaans et al. (2019) develops
decentralized version of distributed PPO to mitigate synchronization overhead between different
actor processes and applies it to point-goal navigation task.
Our framework is closest related Ape-X (Horgan et al. at, 2018), but has a number of key differences. First, our framework is specifically designed for massively parallel GPU-based simulation. Our framework optimized for a single-machine setup, which can further democratize large-scale RL research. Second, we put a replay buffer the same process the learners (policy function learning and Q function learning). This can reduce the communication cost between the replay buffer and the learners. Third, working with a single machine presents new challenges in balancing the compute resource between different parallel processes. Our framework offers a mechanism to balance the compute resource among different processes.
3 METHOD
We developed parallel off-policy training framework, Parallel Q-Learning (PQL), for massively- parallel GPU-based simulation. Our intuition is simple: parallelize the computation as much as possible workstation. A typical actor-critie Q-learning method involves three components: policy function, a Q value function, and an environment. In many popular reinforcement learning libraries (Raffin al., 2021 (Stooke & Abbeel 2019 Weng al. 2021), these three components run sequentially. A typical workflow is as follows: agents roll out the policy in the environments and collect interaction data which will be added a replay buffer, then the value function will be updated to minimize the Bellman error, after which the policy function will be updated to maximize the Q values. terms of the training wall-clock time, such a sequential scheme slows down the training as each component needs to wait for the other two to finish to proceed. To maximize the learning speed and reduce the waiting time, our idea to parallelize the computation of all three components. On the other hand, parallizing the data collection and policy/value function update would be non-trivial for on-policy methods like PPO as the policy update requires on-policy interaction data.
Our framework is optimized for training speed in terms of wall-clock time and can be readily ap- plied on a workstation. Our framework is built upon DDPG (Lillicrap et al. 2015), but can be easily extended to other off-policy algorithms such as SAC (Haarnoja et al. 2018). To reduce the overestimation issue in Q values, we also add double Q learning (Hasselt) 2010). To improve the learning efficiency, we also use n-step returns (Sutton 1988) for function learning. We will use the following notations in the following paper: 8, for observation data at time step t, a, for action command at time step rt for the reward at time step 1, d, for whether the environment terminates
3

---

Under review as a conference paper at ICLR 2023
at time step t, π (s²) for the policy network, Q(s a:) for the network, ('(si, ar) for the target Q network, N for the number of parallel environments.
3.1
FRAMEWORK OVERVIEW
Our framework consists of three parallel processes:
Actor: collecting interaction data in different environments. We use Isaac Gym MMakoviychuk (et al., (02T)) as the simulation engine which supports massively parallel environment simulation. In one actor process, we can parallelize tens of thousands of environments on commodity GPU. The actor process has local policy network x (8t), which periodically updated with the policy network (s,) in P-learner.
V-learner:
We create dedicated process for training the critic functions in DDPG. V-learner
has local replay buffer consisting of the transition data {(st, "t 8 re: de+1)). We put the replay buffer on GPU unless otherwise specified. It also contains one local actor network x (s(s which gets updated to (*(s)) in P-learner, two critic networks Q (s,, at), (5t,,, (5((s,,,) and two target critic networks (st, at), Q2' (st, (st,at). The Q functions are optimized to minimize the mean squared Bellman error. Having dedicated process for updating the Q functions allows the critic functions to be updated continuously without being interrupted by data collection actor network updates.
We another dedicated process for updating the actor network =*(s))- It has replay buffer of {(xr), and a local value function (?(s,, 4x) which is periodically updated with Qi(se, (x) in V-learner. The actor network (*t) optimized maximize the (st,,, (s:)))
P-learner:
Parallelizing each part (data collection, actor, and critic update) of the policy learning allows for more data being collected and more network updates, which can improve the training speed, as we will show in the experiments. We Ray (Moritz al. 2017) to achieve parallelization. The pseudo-code for the framework is shown in Algorithm
and
Data Transfer Suppose there N parallel environments in the Actor process. At each rollout step, the Actor rollouts the policy Ta (5t) and generates N pairs of (st, at, St41, Ft, de+1). Then the Actor sends the entire batch of interaction data {(*,, at, 8, Fe,1,,e,d44111 de+1)} to the V-learner. Since policy update in P-learner only needs state information, Actor only needs to send {(x)] to the P-Jearner.
Network Transfer The V-learner periodically sends the parameters of the Qi ar) to P- learner, which updates the local (st, a;) in P-learner. The P-learner sends the actor network (s:) to both the Actor and V-learner.
Both the data transfer and network transfer happen concurrently.
3.2 BALANCE BETWEEN ACTOR, P-LEARNER, AND V-LEARNER
Our framework allows the Actor, P-learner, V-learner to run concurrently. However, the data collection frequency, policy network, and Q network update need to constrained properly. In other words, each process should not be running on its own as fast as possible. For example, prior works (Fujimoto et al. 2018) have shown that updating the policy network less frequently than the Q functions Icads to better learning. On the other hand, leaving each process rumning freell creates more variance in the training speed and learning performance as the simulation speed and network training speed are heavily dependent For example, simulation for some tasks might be slower than others. Some tasks might require a deeper policy network or Q networks. Even the GPUs on machine might have different running conditions at different times, leading to different speeds across processes and further leading to different learning performance. To reduce the variance of our framework's performance, we add explicit control on the ratio among the data collection frequency, policy update frequency, and the Q function update frequency. We define two ratios as follows:
the task complexity, network size, hardware, etc.
fa
Sp Bp:
Bav
fo
fo
4

---

Under review as a conference paper at ICLR 2023
Figure 2: We experiment on six Isaac Gym tasks: (from left to right) Ant, Humanoid, ANYmal, Shadow Hand, Allegro Hand, Franka Cube Stacking.
where fo is the number of rollout steps per environment in Actor per unit time, fo is the number of Q function updates in V-learner per unit time, fp is the number of policy updates in P-learner per unit time. Pa: determines how many Q function updates are performed in the V-learner when Actor rollouts the policy for one step with environments. Sp: decides how many Q function updates are performed in V-learner when P-Jearner updates the policy once. With the ratios being set, we keep monitoring the progress of each process and dynamically adjust Actor and P-learner speed by letting the process wait if necessary. Controlling the three processes via Ba:s 9pen brings another advantage on resource allocation. When working with limited compute resources on single workstation, having the ability let some of the processes wait allows other processes to use the GPU resource more. For example, if there is only one GPU, and if we put all three processes on the GPU, simulation with a large number of environments can cause very high GPU utilization, slowing down the P-learner and V-learner. By controlling Sa:eo Ppev the processes can be balanced in terms of how much computing can used.
3.3 MIXED EXPLORATION
Balancing the exploration and exploitation requires extensive hyper-parameter tuning in reinforce- ment learning. In DDPG, since the policy network outputs the actions deterministically, one needs to add exploration noise explicitly to the action commands during training. The amount of noise being added needs tuning for each task. One common practice to add exploration noise in DDPG is to add uncorrelated and zero-mean Gaussian noise: dz max(min((T(s) + (N(0, o), a,), ai) (Achiam) 2018 Fujita et al. 2021) where 0, E [a, aul, and a is the standard deviation of the Gaussian distr- bution and controls how much noise is being added. One typically needs tune to get the best per- formance for each task. Since different a leads to varied exploration and exploitation balance, given that we can massively parallelize the environments, one natural idea is to use different exploration strategies in different environments, i.e., set different a values for different environments, which call mixed exploration. Similar ideas have been used in some prior works (Horgan et al. 2018 Mnih et al. 2016). In our work, we uniformly generate the noise levels in the range of Iiin O max In other words, for zth environment out of N environments, a; mmin N-1 (a miin) where e 21, N.. In our experiments, use min 0.05, maax 0.8 for all the tasks.
EXPERIMENTS
We evaluate our method on six Isaac Gym benchmark tasks (Makoviychuk experiments are carried out on single workstation with few GPUs. We consider the following baselines: (1) PPO (Schulman et al. 2017), which is the defacto algorithm used by many prior works (Makoviychuk et al. | Chen et al. 2022 Allshire et al.) (02T)) that use Isaac Gym for simulation, (2) DDPG the sequential (no paralle leliztion)) version of DDPG (Lillierap et al. 20155) implementation with double Q learning and n-step returns, (3) SAC(n): the sequential version of SAC (Haarnoja et al. 2018) implementation with double Q learning and n-step returns. We run each experiment with five random seeds and plot their mean and standard deviation.
2021). All our
4.1 SETUP
Tasks We experiment with six Isaac Gym benchmark tasks (Makoviychuk et al., 2021): Aut, Hu- manoid, ANYmal, Shadow Hand, Allegro Hand, and Franka Cube Stacking (Figure 2). For more details of the tasks, we refer the readers to (Makoviychuk et al. 2021
Compute We use NVIDIA GeForee RTX 3090 GPUs as our default GPUs for the experiments unless otherwise specified. More details are shown in Table B.3
5

---

Under review as a conference paper at ICLR 2023
Ant
Humanoid
Anymal
14k1
8000
70
12%
7000
Retum
10k
06
5500
0000
50
400
PRD
20000
6230
PRD
-04229
2000
-50030
-DD "Cot
1000
10
-po"Go1
(QJ00a)
Wall clock time (min)
Wall clock time (min)
Wall clock time (min)
(a)
(b)
(C)
Franka Cube Stack
Allegro Hand
Shadow Hand
800
7000
8000
0000
7030
600
55000
6000
5500
56000
E5000
4000
2300
200
2000
2000
100
-00K000
1000
PQUDOun)
0
10
60
50
10
20
40
50
60
10
20
50 60
Wall clock time (min)
Wall clock time (min)
Wall clock time (min)
(d)
(e)
(l)
Figure 3:
We compare our method to the SOTA RL algorithms (PPO, SAC with n-step returns,
DDPG with n-step returns). We use 4096 environments for training in all tasks except the PPO baseline on Shadow Hand and Allegro Hand tasks, where we use 16384 as it gives the best perfor- mance for PPO on these two tasks as shown in Figure 4c Our method achieves the fastest learning
speed in almost all the tasks.
(Ours)
Shadnw Hand (PPO)
Shadoow
Penint 1
Wedl dodk iir (1811)
Vad dack km (ran)
Well dock 11m (181))
Vid tlack km (111)
(a)
(b)
(c)
(d)
Figure 4: We sweep over different numbers of environments (N) on both PPO and PQL (our method). Overall, PQL is less sensitive to the number of environments than PPO both tasks.
4.2 DOES PQL REDUCE THE WALL-CLOCK TIME FOR POLICY LEARNING?
As shown in Figure 3 our method (PQL) achieves the fastest policy learning in five out of six tasks compared to all the baselines. We can see that PQL learns faster than DDPG(n) on all tasks, demonstrating that our parallel scheme can speed up policy learning. In addition, we also found that (DPG(n) outperforms SAC(n) in all tasks. This might be because the exploration scheme in DDPG be scaled up better than the in SAC. In DDPG, to a deterministic policy output, while the exploration solely comes from sampling in the stochastic policy distribution, which can be heavily affected by the quality of the policy distribution.
add intepenndeenddnt Gaussian noise
4.3 How DOES THE NUMBER OF ENVIRONMENTS N AFFECT POLICY LEARNING?
GPU simulation allows for running thousands of environments in parallel on a single workstation. We anticipate that GPU simulation is only going to improve with time. However, more parallel environments will be only be useful if RL algorithms are able to exploit such data - in other words only if performance scales with more data. We therefore investigated how different algorithms scale with the number of environments. As shown in Figure on the simple task (Anf), the learning performance is less sensitive to N. However, on the hard task (Shadow Hand), PPO's learning per- formance substantially drops as we decrease the number of environments. In contrast, on both tasks, our method demonstrates stable and similar learning with all the different number of environments except when N is very small (N 256) on Shadow Hand, suggesting PQL is more robust to N.
4.4 EFFECTS OF 3p.u AND Bare
As we discussed in Section 3.2 explicitly controlling the Pa.v and 9p:e can help reduce the vari- ance in the learning performance under different training conditions such as fluctuated hardware utilization. If 8pe. bigger, the policy updates more frequently than the value functions, potentially

---

Under review as a conference paper at ICLR 2023
Ant
Shadow Hand
16k
14k-
8000
7000
12k
6000
Return
5000
-21
4000
-21
6k
11:
3000
4k
112
11
2k
-14
-14
-188
-18
2048
4096
26384
2048
4096
8192
26384
Number of environments
Number of environments
Figure 5: We show the averaged returns in evaluation after a fixed amount of training time AT. experimented (2048, 4096, 8192, 16384), we
Across the set of different number of environments found that setting 3pv = 1: 2 generally works well. AT 60mins for Ant, and AT 80mins for Shadow Hand. The complete learning curves are in Figure C,5 Ant
Shadow Hand
16k
9000
8000
14k
7000
12k-
6000
Retum
10k-
-1:4
3000
1:4
-18
2000
.1:12
1:12
1000
-1:26
11:16
2048
4096
8192
16384
2048
4086
8192
16384
Number of environments
Number of environments
Figure 6: Given different values of N, we show the effect of different Ba:-- An overall trend we observe that as N gets bigger, it's more beneficial to update the critic more frequently. We also found that Bax = 1 8 generally works well given different N values. So one can set 8a.. = 1 1 8 as good initial value, and tune it if necessary on new tasks.
leading to policy overfitting to the stale value function, leading to bad exploration. If the policy updates much slower than the value function, then the policy might lag behind the value function lot, which hurts the learning speed. Similarly, if Barv is bigger, the V-learner might need to wait for Actor to collect enough data as the simulation speed cannot be changed, leading to slower learning. If Sa:v is smaller, the value function updates more given the generated rollout data. To further see the effects of different Bar and Spre. we sweep over different values on these two
and compare them in Figure Figure 6 Figure shows that PQL is relatively robust to wide range of 8p.. values. We use Sp:. = : as the default value in our experiments shown in the paper. @ that Ba.v has a greater impact on the learning performance. An overall trend is that we increase the number of environments, then we need to have V-learner update the Q functions more times. For example, on Shadow Hand, Ba.. 1 4 performs the best when N 2048 and N= #040, 3 = 1 : 12 performs the best when N = 8192 and N = 16384. We use 3a. = 1 : 8 by default as it achieves good performance across different N values. In summary, Figure 5 and Figure
6 show that 8pm and Bav do affect the performance with a varied number of environments. We suggest setting 3pe = 1 2, Ba:v = 1 8 as a good starting point for new tasks and tune them if necessary, as these are the values we found work well on six different tasks with a different number of environments.
4.5 How WELL DOES MIXED EXPLORATION PERFORM?
With massively parallel simulation, one can deploy different exploration strategies in different en- vironments to generate more diverse exploration trajectories. We use a simple mixed exploration strategy as described in Section 3.3 To see its effectiveness, we compare it to the cases where all the environments use the same exploration capacity (the same a values). We experimented with E {0.2, 0.4, 0.6, 0.8). As shown in Figure 7 the learning performance is significantly affected by the choice of a value. If we use the same o value for all the environments, then we need to tune o for every task. In contrast, if we use the mixed exploration strategy where every environment use different a value, the agent outperforms (learn faster or at least as fast) all the other fixed a values. It implies that using the mixed exploration strategy can reduce the tuning effort a values per task.
7

---

Under review as a conference paper at ICLR 2023
Ant
Humenood
Franki Cube Stacking
Shadow Hand
2000
7000
Reom
10
20
25
40
50
20 20
40
10
30
43
55
20
20
20
00
70
dock (ni)))
(a)
(b)
(c)
(d)
Figure 7: We compared our proposed mixed exploration scheme with applying different constant maximum noise values. We can see that the mixed exploration scheme either outperforms or is on par with other schemes, which can save the tuning effort the noise level.
Ant
Shadow Hand
Return
14cc
12k
8000
6300
7000
55000
023000
2000
1000
-512 11122-2004-2000 4000 H0182 22384-22705
222
2042
4006 0102
10004-22768
100
20
30
40
50
10
20
30
40
50
60
70
Wall clock time (min)
Wall clock time (min)
(a)
(b)
Figure& 8: Effect of different batch size. Small batch size usually leads to slower learning. If the batch size is too big, the policy learning can get slowed down because GPUs have a limited amount of cores and it takes more time to process very big batch of data.
4.6 EFFECT OF BATCH SIZE
With many parallel environments (N), lot of data is quickly generated. While one can easily increase N from 100s to 10,000s in Isaac Gym on a single GPU, it is infeasible to increase the replay buffer size by 100 times due to the limited GPU memory or CPU RAM (if the data is stored on CPU). Consequently, the replay buffer is overwritten frequently - mcaning each collected sample may not be used efficiently. One way to efficiently utilize large amounts of changing data is increase the batch size. To test how much increase in batch size is necessary for Q-learning with a limited capacity replay buffer to take advantage of the large amounts of incoming data, we investigated the relationship between performance and batch size. Hence, we also investigate the effects of different batch size B. There have been many prior works showing that using large batch size can improve the network performance such as in contrastive learning settings (Grill et al., [020 Chen jet al. 2020). In our work, we also found that large-batch training can notably improve the learning speed in off-policy reinforcement learning for massively parallel simulation However, if the batch size is too big, then the learning speed can be slowed down. It is because GPUs have limit number of CUDA cores, and it takes more time to process very big batch data once the batch size is above some threshold value, which is another underlying trade-off.
shown in Figure B
4.7 EFFECTS OF OTHER FACTORS
GPU hardware The simulation speed and network training speed vary across different GPU mod- els. In Table B.3, we list how much time it takes for the simulator to generate 1M environment interaction data with 4096 parallel environments on four machines with different GPU models. In our test, the simulation speed on different GPU models is as follows: GeForce 3090 > Tesla A100 Tesla V100 > GeForee 2080Ti. We test PQL. performance configurations (Table [B.). In Figure / we can see that different GPU models affect the policy learning speed, especially on complex tasks like Shadow Hand which takes more simulation time.
all these four different machine
Number GPUs Our PQL. framework is flexible regarding the number of GPUs available on a workstation. In other words, Actor, P-learner, and V-learner can be placed on any GPU on workstation. To see how much performance variation it creates when we place the three components on different number of GPUs, we experimented with three scenarios on Tesla A00 GPUs: (1) place all three processes on the same GPU, (2) place the Actor on one GPU, P-learner and V-learner on another GPU, (3) place the Actor, P-learner, V-learner GPU case, the reason why we allocate Actor with a dedicated GPU is because the simulation for many tasks with a large number of environments can cause very high (almost full) GPU utilization.
different GPU respectively. In the two-
8

---

Under review as a conference paper at ICLR 2023
Am
Shadow Hand
Art
Shadow Hand
Ri0m
ROU
-2
20 10
20
20
10
20
30
42
70
Viel choce
27
30
30
0 32
Wedl (ock (181))
40 50
70
(a)
(h)
(d)
Figure 9: (a) and (b): effect of GPU models used for running PQL. Overall, we see that PQL works robustly across different GPU models, and running on newer GPUs tend to give a faster learning.
(c) and (d): effect of number of GPUs used for running PQL.. PQL can be deployed on a flexible number of GPUs. In complex tasks such as Shadow Hand, it is beneficial to have at least 2 GPUs where the Actor runs on separate GPU as the simulation itself consumes more GPU compute as the task complexity increases.
As shown in Figure our PQL framework works well in all three scenarios with one, two, three GPUs. When the task becomes more complex like Shadow Hand, the simulation takes much more computation and time. Putting all three processes will slow down each one of them as the GPU utilization will be full all the time, which is why we see a bigger gap between the 2-GPU or 3-GPU training and I-GPU training on Shadow Hand. this case, it beneficial to place the Actor on one GPU, and put P-learner and V-learner on other GPUs.
4.8 VISION-BASED TASK
Simulating vision-based tasks is much slower and more demanding on the GPU as each simulation step involves both the physics simulation and image rendering. To demonstrate our framework's generality in operating in this different but practical setting of vision-based training. we consider simple vision-based Ball Balancing task. Since directly learning rision-based policy with RL is time-consuming, we use the idea of asymmetric actor critic learning (Pinto et al. 2017) to speed up vision policy learning. To reduce the bandwidth requirement, image data is compressed using the l24 library for fast communication between processes. More setup details are in Appendix [B.3] As shown in Figure B.I] PQL achieves better sample efficiency and higher final performance than PPO with N 1024 parallel environments.
DISCUSSION
We have presented a framework for scaling up off-policy methods, especially when using GPU- based simulation. Our framework achieves state-of-the-art results on the Isaac Gym benchmark tasks, both in terms of the training wall-clock time and the final performance.
Our PQL framework provides a mechanism to balance and control the speed in different processes (data collection, policy network update, and Q functions update), which leads to more stable per- formance across different hardware conditions or when the GPU resource is limited. While PPO requires large number of environments work complex tasks such as Shadow Hand, PQL is more lenient on the number of environments and works well on a wide range of different numbers of environments. With large number of parallel environments, it's beneficial to use big batch size for training agents with a caveat that if the batch size is too big, it might take the GPU more time process the batch data and lead to a slowdown in policy learning. We also found using dif- ferent exploration scales in different environments achieves better or similar performance compared to carefully-tur -tuneed same exploration scale in all environments, which means we need less hyper- parameter tuning. Even though the number of environments is 1000 more compared to many Q-learning based agents, we did not find it necessary to use a replay buffer that 1000x bigger. In fact, replay buffer with capacity of 5M transitions is sufficient for experiments even with 16843 parallel environments. Our framework's 's requirement for hardware is also flexible and works well with different numbers of GPUs and various GPU models.
We do not use techniques such as prioritized experience replay, dueling networks, distributional RL, noisy nets for exploration in work. Nonetheless, our PQL framework already gets better per- formance and faster learning than PPO. Whether adding these extra techniques can further improve the learning speed or final policy performance remains to be explored in future work.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We describe our framework in detail in Section 3.1 Our mixed exploration strategy also reduces the need for iypee--arameeter turning the exploration noise level (Section 4.5). Details of all the hyper-paramelers are in Appendix [B.1] We have used the same set of hyper-parametert for all the tasks. Each curve shown in the paper is obtained after running the experiment with five different random seeds. We have also investigated the performance variation on different GPU models and different numbers of GPUs in Section 4.7 Lastly, we plan release the code in the future.
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
Arthur Allshire, Mayank Mittal, Varun Lodaya, Viktor Makoviychuk, Denys Makoviichuk, Felix Widmaier, Manuel Wüthrich, Stefan Bauer, Ankur Handa, and Animesh Garg. Transferring dexterous manipulation from gpu simulation remote real-world trifinger. arXiv preprint arXiv:2108.09779, 2021.
Mohammad Babacizadeh, Iuri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on a gpu. arXiv preprint 2016.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint /9Xiv:/9/2.0000 19/2.06680, 2019.
Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. In Conference on Robot Learning. pp. 297-307. PMLR, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Alfredo Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint
2017.
Erwin Coumans and Yunfei Bai. Pybullet. python module for physics simulation for games, robotics and machine learning. 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scal- able and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:1910.06591. 2019.
Scott Fujimoto. Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Yasuhiro Fujita, Prabhat Nagarajan, Toshiki Kataoka, and Takahiro Ishikawa. Chainerrl: A deep reinforcement learning library. Journal of Machine Learning Research, 22(77):1-14, 2021. URL 576.htm..
httt:://jmix wttp:o/pppppppp///////...
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bermardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised lcarning. Advances in neural information processing systems, 33:21271-22 2212222444 2020.
Tuomas Haamojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Hado Hasselt. Double q-learning. Advances in neural information processing systems, 23, 2010.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environ- ments. arXiv preprint arXiv:1707.022200 2017.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018.
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26))eaau5872, 2019.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent perience replay in distributed reinforcement learning. International conference on learning representations, 2018.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238-1274 2013.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 02977199. 2015.
Denys Makoviichuk and Viktor Makoviychuk. rl-games: A high-performance framework for rein- forcement learning.
games
May 2022.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 108. 10470, 2021.
Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. arXiv preprint arXiv:2205. 02824, 2022.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hut- ter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eabk2822, 2022.
2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5187540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1938-1937. PMLR, 2016.
Philipp Moritz, Robert Nishihara, Stephanic Wang, Alexey Tumanov, Richard Liaw, Eric Liang, William Paul, Michael I Jordan, and Ion Stoica. Ray: distributed framework for emerging ai applications. corr abs/1712.0588 (2017). arXiv preprint arXiv: 2017.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforeement learning. arXiv preprint arXiv: 150..042900004999 2015.
OpenAl, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojciech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot leaming. arXiv preprint arXiv: 2017.

---

Under review as a conference paper at ICLR 2023
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth- Narth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXh::1704.0300 2017.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeaap7885, 2018.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. Conference on Robot Learning, pp. 91-100. PMLR, 2022.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511.0599 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Adam Stooke and Pieter Abbeel. ripyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint 19X.v:1909.014.. 2019.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Jayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Dubureq, Minghao Zhang, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. arXiv preprint 1111::2107.14171, 2021.
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. 2019.
arXiv preprint
12

---

Under review as a conference paper at ICLR 2023
APPENDIX A
PSEUDO CODE
Algorithm Actor Process (main process)
for W. do
+- policy network from P-learner process Initialize an empty buffer Φ for f 1 H do
a; < (8,) with mixed exploration noise 8t-1) envs.step(a.)
= {st, az Tz, $1+1)
end for
Q1, Q2 + Q functions from V-learner process
send B, π V-learner, send {st} B, Q1, Q2 P-Jearner sleep for to seconds to satisfy Bav
end for
Algorithm P-learner Process
Initialize an empty buffer Bp Φ for 1: Wp do
new data received then
{ from Actor process
Q1, Q2 t from Actor process B {5r}
end if
sample a batch of {st)
update ı by maximizing the min;=1.2 (i(s;: (s;)) sleep for lp seconds to satisfy 3pm end for
Algorithm V-learner Process
Initialize an empty buffer B, for m 1: W, do
new data received then
{st, Tt, $t-1) } + from Actor process
< from Actor process
Q1, Q2 <- from Actor process
B (s)
end if
sample a batch of {st, (s, Jt. St.11
update Q1, Q2 by minimizing the mean-squared Bellman error (with Double Q-learinn) sleep for t, seconds to satisfy Spes Ba:o end for
APPENDIX B TRAINING SETUPS
B. HYPERRPPRRMMTTRS
We use the hyper- prper-parameter values shown in Table B.I and the reward scaling shown in Table B.2 for all the experiments unless otherwise specified. As for PPO, we use the same hyperparameter setup in Makoviychuk et al. (2021).
B.2 HARDWARE CONFIGURATIONS
Table B.3 lists the hardware configurations the workstations we used for the experiments. We use the machines with GeForce RTX 3090 for experiments by default. We also measure how much time
13

---

Under review as a conference paper at ICLR 2023
Table B.l: Hyper-parameter setup for six Isaac Gym benchmark tasks
Hyper-parameter
PQL(ours)
DDPG
SAC
Num. Environments
4,096
4,096
4,096
Critic Learning Rate
5 x 10-4
5 X 10
5 x 10
Actor Learning Rate
5 10
5 x 10
5 x 10
Learnable Entropy Coefficient
True
Optimizer
Adam
Adam
Adam
Target Update Rate (T)
10 10-2 2
x 10
10-2
Batch Size
8,192
8,192
8,192
Num. Epochs (Base)
Discount Factor(y)
0.99
0.99
0.99
Normalized Observations
True
True
True
Gradient Clipping
0.5
0.5
0.5
Exploration Policy
Mix
Mix
step target
Warm-up up Steps
32
32
32
Replay Buffer Size
10
x 10
x 106
Table B.2: Reward scale
Reward scale
Ant
0.01
Humanoid
0.01
ANYmal
1.0
Franka Cube Stacking
0.1
Allegro Hand
0.01
Shadow Hand
0.01
Ball Balance
0.1
it takes for the simulator to generate 1M interaction data with 4096 parallel environments on Ant and Shadow Hand. We generate 1M data via the following command.
for i in range (244):
action torch. randn ( (4096,
envs antiiaaatiossppeeee shape [0]), device=' cuda')
out envs step (act ion)
Table B.3: Hareware configurations on different workstations
Workstation
Workstation
CPU
AMD Threudripper 3990X
Intel Xeon Gold 6248
AMD Rome 7742
Intel Xeon W-2195
GPU
GeForce RTX 3090
Tesla V180
Tesla A100
GeForce RTX 2080
GPU CUDA Cores
10496
5120
6912
4352
GPU FP32 TFLOPs
35.58
16.4
19.5
13.45
Time geacrating
Aat
1.678 0.006
2.117 0.038
1,999 0.001
3.397 0.011
IM data (N
4096)
Shudow Hand
6.706 0.028
9.051 0.035
8.653 4. 0.101
10.885 0.025
B.3 VISION EXPERIMENT SETUP
We render the RGB camera image in a resolution of 48 x 48. The CNN part of our vision network g(o.) is follows:
Conv (3, 32, 3, 2) -BN (32) -ReL.U3x 3x (Conv (32, 32, 3, -BN (32) -ReTUU
14

---

Under review as a conference paper at ICLR 2023
where Conv (a, b, k, s) is Convolutional layer with input channels a, output channels b, kernel size k stride s.
Since policy imput contains history of observations (or-2, 0t-1, 0t). use the same CNN to extract the feature of each observation and then concatenate all the embeddings. Then, the concate- nated embedding goes through MLP network h:
FC (256) Reee-e FC (63) RRLUU-FC
In summary, at each time step f, the policy output is a replay buffer can take up a lot of memory. Therefore, we experiment with different placements of the replay buffer: (1) put the replay buffer
g(o))) Storing images
GPU with big memory, (2) put the replay buffer on CPU RAM. We use same A A100 GPUs for these image-based experiments. Figure [.. shows that our method (PQL) works with either the replay buffer the GPU or CPU, and it achieves much faster learning and better performance than PPO.
Ball Balance
Ball Balance
225
5000
2224)
2130
3150
100
Heplay sules aind
120
PQ- Replly tulle canu
30
50
05
1.5
Number (M)
clock (min)
(a)
(b)
(c)
(d)
Figure B.1: (a): the Ball Balancing task in Isaac Gym. (b): the rendered RGB image from the simulated camera. (c) shows the learning curves regarding the number of environment steps. (d) shows the training wall-clock time. We can see that PQL achieves both better sample efficiency and higher final performance than PPO.
Table B.4:
setup for the Ball Balancing task.
PQL(surs)
PPO
Num. Environments
1,024
1,024
Critic Learning Rate
10
5 x 10 10-4
Actor Learning Rate
5 10
5 x 10-4
Optimizer
Adam
Adam
Target Update Rate (r)
5 10
Batch Size
4,096
4,096
Horizon length
16
Num. Epochs
12
5
Discount Factor(y)
0.99
0.99
Normalized Observations
True
True
Gradient Clipping
True
True
Exploration Policy
Mix
N-step target
Warm-up up Steps
32
Replay Buffer Size
106
Clip Ratio
0.2
GAE
True
À
0.95
APPENDIX C ADDITIONAL EXPERIMENTS
n-step returns We investigate how much does n-step returns help for PQL. As shown in Fig- ure C.2, adding n-step return leads to faster learning than not using n-step return (n 1). However, using big value hurt the learning. Empirically found that gives us the best performance.
15

---

Under review as a conference paper at ICLR 2023
Replay buffer size We investigate how the replay buffer size [B] affect the policy learning perfor- mance. We experimented with replay buffers with a size of (1, 10, 20 20 The replay buffers stay on GPUs. As shown in Figure C.2 in all cases, policies to faster policy learning in the beginning of the training than [| = (10, 20). We hypothesize this is because with smaller replay buffer, the old and less informative samples get replaced much faster which is more important in the early stage of the training. On the other hand, as shown in Table B.3 generating 1M interaction data with Isaac Gym takes a very small amount of time. For example, on GeForce 3090 GPU, it only takes 1.7s to generate 1M data on Ant, and 6.7s on Shadow Hand. This implies that with massively parallel simulation, the data in the replay buffer becomes more on-policy than in most Q-learning agents. We hypothesize that PQL. DDPG still works well in this case because the large amount of parallel environments can generate diverse enough data short period of time.
learn well. In addition, [B] {1,5} lead
Ant
Shadow Hand
Shaddow Hand
201
(mir)
13 oine (mi)))
20 (c)
(min))
Wet (mm (mii))
30 45
70
(a)
(b)
(d)
Figure C.2: (a) and (b): effect of n-step return. performs the best. (c) and (d): effect of different replay buffer size.
PQL for SAC As discussed above, PQL framework is flexible and can be combined with different
Q-learning methods. Here, we show that PQL. can combined with SAC as well. Figure shows that adding the PQL framework to SAC substantially speeds up the learning speed of SAC.
Ant
Humanoid
Anymal
12k
5000
70
10k
5000
Return
8. 9
14000
50
3000
40
1000
20
10
-PgulOoas, 5912
100
40
50
10
20
30
40
50
10
15
20
Wall clock time (min)
Wall clock time (min)
Wall clock time (min)
(a)
(b)
(c) Shadow Hand
Franka Cube Stack
Allegro Hand
900
800
700
8000
5600
Ee000
83000
84000
200
100
2000
1000
2000
P0I
0
10
40
50
10
20
30
40
50
80
10
20
30
50
60
70
Wall clock time (min)
Wall clock time (min)
Wall clock time (min)
Figure C.3: We apply our parallel Q-learning to SAC. PQL SAC achieves faster learning than SAC itself.
Sample efficiency compared to baselines Figure C.4 shows the sample efficiency of each algo- rithm on different environments. Overall, that PQL achieves the best sample efficiency. In addition, DDPG(n) also outperforms SAC(n) in terms of the sample efficiency on these tasks.
Sweep over different Ba. and 3pev Figure C.5 shows the complete learning curves with different Spee values and different number of environments. Similarly, Figure C.6 shows the learning curves for different Ba.e.
Comparison of our implementation with RL-games In this work, we implemented all the al- gorithms (PQL. and all the baselines) from scratch, as it gives us the most flexibility in exploring
16

---

Under review as a conference paper at ICLR 2023
Humanoid
Anymal
8000
7000
6000
E5000
Retur
40000
22000
24010
2000
1000
-FQU0000)
100
300
400
500
600
50
100
150
200
250
300
350
400
10
30
50
Number Steps (M)
Number steps (M)
Number steps (M)
(a)
(b)
(c)
Franka Cube Stack
Allegro Hand
Shadow Hand
900
8000
5000
800
E5000
6000
4000
200
100
1000
2000
54010
50
100
150
200
50
300
150
200
250
100
200
300
400
500
Number stops (M)
Number of stops (M)
Number steps (M)
(d)
(e)
(f)
Figure C.4: Similar to Figure we show that our method (PQL) also achieves better sample effi- ciency than baselines.
different design choices that can affect learning performance. To show that our codebase provides good performance, we compare against the most commonly used RL codebase used for Isaac Gym, which is RL-games (Makoviichuk & Makoviychuk 2022). However, RL-games only sup- port PPO and SAC. Hence, we compare our implementations PPO and SAC against the ones in RL-games.
17

---

Under review as a conference paper at ICLR 2023
Ant 2048
Ant 4096
14k
14k
Return
Retur
884
-21
-21
-1:
68
-11
-14
24
-14
0K
0k
10
20
30
20
30
40
50
Wall clock time (min)
Wall clock time (min)
(a)
(b)
Ant 8192
Ant 16384
140
14k
12k
126
Return
Return
-1:1"
66
-122
4k
-18
0k
Wall clock time (min)
Wall clock time (min)
(c)
(d)
Shadow Hand 2048
Shadow Hand 4096
84
84
0k
6k
Return
Return
4k
-12
10
60
20
30
70
(min)
Welll
(min)
(c)
(f)
Shadow Hand 8192
Shadow Hand 16384
86
Return
Retur
-2.1
-21
-1.2
-14
-14
0k
0k
10
20
30
40
50
60
70
80
10
20
30
40
50
60
70
80
Wall clock time (min)
Wall clock time (min)
(g)
(h)
Figure C.5: Learning curves for different Bp:e-
18

---

Under review as a conference paper at ICLR 2023
Ant 2048
Ant 4096
14k
14k
Return
Return
880
-11
-11
-14
68
-14
1:12.
24
-1112
0K
0k
20
20
30
40
50
60
70
80
20
30
40
50
Wall clock time (min)
Wall clock time (min)
(a)
(b)
Ant 8192
Ant 16384
144
14k
12k
126
Return
Return
-1:4
66
114
4k
1:16
11:6
0k
Wall clock time (min)
Wall clock time (min)
(c)
(d)
Shadow Hand 2048
Shadow Hand 4096
84
84
0k
6k
Return
Return
10
4k
-18
-18
1:16
11:6
10
60
20
60
(min)
Walll (min)
(c)
(f)
Shadow Hand 8192
Shadow Hand 16384
86
Return
Return
1:1
-1
1.8
-1:12
11:22
0k
0k
10
20
30
40
50
60
70
10
20
30
40
50
60
70
00
Wall clock time (min)
Wall clock time (min)
(g)
(h)
Figure C.6: Learning curves for different Base-
19

---

Under review as a conference paper at ICLR 2023
Ant
Shadow Hand
12k
7000
10k
6000
Return
8k
5000
6k
4000
PPO (Ours)
3000
PPO (Ours)
(RL Games)
2000
Gamas)
SAC (Ours)
1000
SAC (Durs)
(RL Gamas]
(RI Games)
20 30
60
70
Wall clock time (min)
Wall clock time (min)
(a)
(b)
Figure C.7: Comparison between our implementations of PPO and SAC against the ones provided in RL-games. We can see that both codebases provide similar performance, showing that our im- plementation is good and reliable. On Shadow Hand, our PPO learns even faster and better than the PPO in RL-games.
20


Reviews:
[{"review_id": "bFxkeDvfl0b", "reviewer": "Reviewer_xfGC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate the interplay between the training speed/performance of an offline RL method and the total number of environments used during training. A large number of RL papers default to online RL methods, specifically PPO, because of the relative ease of these methods, however, offline methods should be more sample efficient by virtue of reusing samples from the replay buffer. The authors implement an offline RL training scheme, PQL, to leverage recent advances in GPU simulation environments to accelerate the training process while achieving comparable results with online methods. PQL breaks up the data collection and training pipeline into 3 processes, an Actor, P-learner, and V-learner. Since the simulation environment is located on the GPU all 3 processes may be co-resident on a single GPU or distributed across multiple GPUs on a single node. The placement of processes on different GPUs and the ratio of data collection vs training speeds are studied empirically and optimal settings tend to vary, as expected, across environments. The authors also study empirically how the batch-size and replay-buffer sizes influence the training process as the number of processes changes.", "summary_of_the_review": "For researchers that do not have access to a large pool of computational resources the ability to take maximum advantage of the resources they do have on hand is imperative. As such, on-policy methods typically underutilize GPU resources because they cannot generate a sufficient amount of data to keep the GPU saturated. Though many works have tried to address this issue by increasing the number of environments this typically requires an increase in the number of CPUs as well. I think it is quite interesting to try taking full advantage of the GPU-centric training environments to better utilize the hardware and investigate the interplay between the number of environments and the training dynamics. This type of investigation seems to be inline with the spirit and direction of the Issac Gym GPU simulator project.\n\nThe improved training time to accuracy is visible in a number of the plots. In Section 2 the authors reference an improvement in regard to both the sample efficiency and training times of off-policy algorithms but when I think of sample efficiency I would like to see plots of the return vs the number of environment interactions. The authors noted in section 4.6 that the volume of data generated by the simulators is so high that data in the replay buffers are being replaced rapidly causing the method to become more on-policy. I assume this means the sample efficiency also resembles that of an online policy in some cases, is that correct? Does this training scenario detract from one of the selling points of offline algorithms to reduce the volume of required training data?\n\nIn the discussion section the authors mention not using many modifications that are typical of modern offline algorithms. Was it too difficult to add these features on top of the current implementation? I feel that omitting these features could be a mistake since top-performing offline algorithms use them so without those reference points it's more challenging to compare PQL performance against prior work.\n\nA concern is that the primary contribution is the framework that implements PQL, however, this artifact mostly builds on existing work (Issac Gym, PyTorch, etc). This diminishes the novelty of the work and makes the results appear more informative rather than a novel contribution that rises to the level required for publication at the current venue.", "strength_and_weaknesses": "Strengths\n--------------------\n- The fundamental question being addressed by the paper is interesting. The training dynamics of RL algorithms using GPU-simulated environments may be dramatically different from those of CPU-simulated environments. The abundance of environments producing data at a rate much higher than previously studied coupled with lower communication overheads, because data is generated directly on the device, provide ample motivation to rethink training algorithms and hyperparameters typically used by members of the community. In short, is PPO the best off-the-shelf algorithm in light of the differences between training environments?\n- The authors demonstrate faster convergence for their offline training method compared to PPO on several GPU-simulated environments.\n- The influence of several hyperparameters, such as batch-size, replay-buffer size, the number of training environments, and the number of GPUs were studied as part of the empirical results.\n\nWeaknesses\n-------------------\n- The paper is empirically heavy but theoretically lite. There's no motivation underlying the choices for exploration of the design space of the parallel training environment hyperparameters.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PcE5Fr6iZp", "reviewer": "Reviewer_iU6x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the data inefficiency problem of off-policy algorithms, like PPO, this paper proposed a parallel version of the Q-learning algorithm. Specifically, this approach uses three parallel processes, namely actor, V-learner and P-learner, to perform a DDPG-like learning process. Empirical evaluations on six Isaac Gym tasks has demonstrated that the proposed approach has better learning efficiency than baselines like PPO, SAC, etc.", "summary_of_the_review": "Despite an extensive amount experiments have been conducted, the writing of this paper requires a considerable revisions to convince the contributions of this paper and illustrate how each design consideration leads to performance improvement over existing methods.", "strength_and_weaknesses": "Strength:\n1. The evaluation of the proposed method is extensive and involves many different tasks.\n2. The parallel simulation environments considered here are interesting and they might be important to greatly reduce the training costs for RL. \n\nWeaknesses:\n1. The introduction of the three parallel processes is quite abrupt without adequate discussions about the motivation of such considerations. It's unclear why these design choices are inspired from existing methods like DDPG or other variants. Furthermore, what's the challenges to design a Q-learning algorithm for parallel environments? Why a direct extension of DDPG can not work?\n2. It's not clear what's the key difference between parallel environments on a single workstation and parallel environments on a computer cluster. Does such difference really matters in the design of the algorithm? Why?\n3. The discussion of the main techniques is too brief and can not convince readers the design rationales behind the proposed algorithm.\n4. Another suggestion is to write the proposed method as an algorithm so that it would be more clear the algorithmic improvements made over existing approaches. Stressing the design considerations that make the proposed approach more efficient than others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HDr3-3aAY5", "reviewer": "Reviewer_BMNj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes paralleled Q-learning, a three-component framework to accelerate Q-learning. The design of the components enables concurrent execution, and experiments on Isaac Gym and vision-based benchmarks against other state-of-the-art algorithms such as PPO, DDPG and SAC show significant improvement as measured by cumulative return. ", "summary_of_the_review": "I would rate this paper a 8, given its scope of contributions and good quality. ", "strength_and_weaknesses": "Strength:\n\n1. The paper is written very clearly, with all the sections accurately describing what the algorithm is about. The relationship between the different components in the framework has been clearly demonstrated by graphs, and the intuition behind the components as well as choices of returns/RL structures is easy to follow.\n\n2. The experiments are comprehensive, and provide sufficient details for reproducibility as well as ablation studies. In particular, the discussion on the ratio of beta values(indicating update frequency comparisons) is very informative.\n\nWeaknesses:\n\n1. The algorithm is named Parallel Q-learning. However, the key of the algorithm lies in decomposing DDPGs/SACs and making use of techniques of improving Q-learning to better estimate Q-functions in the learning process. It would be helpful to emphasize in the abstract and intro where the novelty lies since the title may be a bit misleading.\n\n2. While the results of the experiments are largely self-contained and informative, it would be helpful to include the performances of PQL with SAC as the underlying algorithm. Since it is indicated in the paper that the PQL framework can be naturally extended to SACs, the inclusion of such ablation studies would answer the question how much the effectiveness of the PQL framework would depend on the choice of off-policy algorithms.    \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FCMU7k2rGV", "reviewer": "Reviewer_bLp3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to run thousands of simulations of a RL environment to speedup Q learning. Unlike previous efforts, that leverage distributed computing to run agents on large numbers of machine, they propose to leverage a single workstation running numerous tasks in parallel.\nThe paper contributes a novel framework, called PQL, that can train RL policies faster than previous efforts without requiring a large amount of compute resources.", "summary_of_the_review": "PQL is an interesting paper that demonstrates that a better implementation of the Q learning strategy can make it competitive against PPO from the point of view of training time. However, the author need to provide an intuition as to why their approach is inherently more performant. Furthermore, the evaluation needs more work to clearly demonstrate how the authors achieve superior performance.", "strength_and_weaknesses": "Scale is a major force behind the success of reinforcement learning. However, unlike numerous previous approaches that achieve scale by distributing the computation over large amounts of servers, PQL proposes to leverage a single workstation by multiplexing the simulation of thousands of environments along with a value function and a policy on a single GPU.\n\nThe approach appears to be similar to that of DDPG, and the implementation is based on an existing framework targeted for distributed training that was repurposed to run multiple simulations of the Isaac Gym environment on a single workstation, which is the main contribution of the paper. However, several points are unclear:\n * Where do the wall-time gains come from ? Is Isaac Gym more efficient when running multiple simulations in parallel, or is there something else driving the performance improvement ? Ideally, I'd like to be able to answer the question: what aspect(s) of PQL approach is inherently better and drives most of the performance improvements.\n * How applicable is this beyond Isaac Gym ? Would it help with other simulation based gyms ?\n * How would the PQL approach compare against other distributed RL implementations such as IMPALA/seed-RL if these implementations were also repurposed to target the Isaac Gym ?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject