QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learning
Authors: Srivatsan Krishnan
Max Lam
Sharad Chitlangia
Zishen Wan
Gabriel Barth-maron
Aleksandra Faust
Vijay Janapa Reddi
Venue: Accepted by TMLR ()

Abstract: Deep reinforcement learning continues to show tremendous potential in achieving task-level autonomy, however, its computational and energy demands remain prohibitively high. In this paper, we tackle this problem by applying quantization to reinforcement learning. To that end, we introduce a novel Reinforcement Learning (RL) training paradigm, \textit{ActorQ}, to speed up actor-learner distributed RL training. \textit{ActorQ} leverages 8-bit quantized actors to speed up data collection without affecting learning convergence. Our quantized distributed RL training system, \textit{ActorQ}, demonstrates end-to-end speedups of $>$ 1.5 $\times$ - 2.5 $\times$, and faster convergence over full precision training on a range of tasks (Deepmind Control Suite) and different RL algorithms (D4PG, DQN). Furthermore, we compare the carbon emissions (Kgs of CO2) of \textit{ActorQ} versus standard reinforcement learning on various tasks. Across various settings, we show that \textit{ActorQ} enables more environmentally friendly reinforcement learning by achieving 2.8$\times$ less carbon emission and energy compared to training RL-agents in full-precision. Finally, we demonstrate empirically that aggressively quantized RL-policies (up to 4/5 bits) enable significant speedups on quantization-friendly (supports native quantization) resource-constrained edge devices, without degrading accuracy. We believe that this is the first of many future works on enabling computationally energy-efficient and sustainable reinforcement learning. The source code for QuaRL is available here for the public to use: \url{https://bit.ly/quarl-tmlr}.

Published in Transactions Machine Learning Research (07 (022)
QuaRL: Quantization for Fast and Environmentally Sustainable Reinforcement Learning
Srivatsan Krishnan" Horvard University
sriuatsan@ssea.haruuat edu
Maximilian Lam"
mazlam@g.harrarr
Harvard University
Sharad Chitlangia"t
chitshrr@amazon.coom
Amazon Advertising
Zishen Wan Georgia Institute of Technology
zishenwon&&yallcch.com
Gaberial Barth-Maron DeepMind
gabriellm.&googleecoom
Aleksandra Faust Google Research, Brain
sondrafouss@@googl..com
Vijay Janapa Reddi Horvard University
vjGeecs.harvare edu
Reviewed OpenReview:
Abstract
Deep reinforcement learning continues to show tremendous potential in achieving task- level autonomy, however, its computational and energy demands remain prohibitively high. In this paper, we tackle this problem by applying quantization to reinforcement learning. To that end, we introduce a novel Reinforcement Learning (RL) training paradigm, ActorQ, to speed up actor-learner distributed RL training. ActorQ leverages 8-bit quantized actors to speed up data collection without affecting learning convergence. Our quantized distributed RL training system, ActorQ, demonstrates end-to-end speedups between 1.5 X and 5.41x, and faster convergence over full precision training on a range of tasks (Deepmind Control Suite) and different RL algorithms (D4PG, DQN). Furthermore, we compare the carbon emissions (Kgs of CO2) of ActorQ versus standard reinforcement learning algorithms various tasks. Across various settings, we show that ActorQ enables more environmentally friendly reinforcement learning by achieving carbon emission improvements between 1.9x and 3.76x compared to training RI-agents in full-precision. We believe that this is the first of many future works on enabling computationally energy-efficient and
'Equal Contribution
Work done when Sharad was a visiting undergraduate student at Harvard. Work done when Zishen was a graduate student at Harvard.

---

Published in Transactions on Machine Learning Research (07/2022
sustainable reinforcement learning. The source code is available here for the public to use:
1 Introduction
Deep reinforcement learning has attained significant achievements in various fields (Bellemare et al., 2013; Kempka et al., 2016; Kalashnikov et al., 2018; Silver et al., 2016; 2017; OpenAI, 2018; Chiang et al., 2019; OpenAI et al., 2019). Despite its promise, one of its limiting factors is long training times, and the current approach to speed up RL training on complex and difficuul tasks involves distributed training (Espcholt et al., 2019; Nair et al., 2015; Babacizadeh et al., 2016). Although distributed RL training has demonstrated significant potential in reducing training times (Hoffman et al., 2020; Espeholt et al., 2018), this approach also leads to increased energy consumption and greater carbon emissions. Recently, work by (Wu ct al.; 2021) indicates that improving hardware utilization using quantization and other performance optimizations can reduce the carbon footprint of training and inference of large recommeudation models by 20% every six months. In this same vein, we believe reinforcement learning can also benefit from these optimization techniques, particularly quantization, to reduce training time, improve hardware utilization and and reduce carbon emissions.
In this paper, we tackle the following research question How can we speed up RL training withoul significantly increasing its carbon emissions? To systematically tackle this problem, we introduce ActorQ. In ActorQ, we use quantization to reduce the computation and communication costs of various components of distributed RL training system. Basod on the charactcrization of core components of distributed RL training, we find that majority of the time is spent on actor policy inference, followed by the learner's gradient calculation, model update, and finally, the communication) cost between actors and learners (Figure 1). Thus, to obtain significant speedups and reduce carbon emissions, we first need to lower the overhead of performing actor inference. To achieve this in ActorQ, we employ quantized policy (neural network) inference, simple yet effective optimization technique to lower neural network inference's compute and memory costs.
Applying quantization to RL is non-trivial and different from traditional neural network quantization. Due to the inherent feedback loop in RL between the agent and the environment, the errors made at one state might propagate to subsequent states, suggesting that actor's policies might be more challenging to quantize than traditional neural network applications. Despite significant research on quantization for neural networks in supervised learning, there is little prior work studying the effects of quantizing the actor's policy in reinforcement learning. To bridge this gap in literature, we benchmark the effects of quantized policy during rollouts in popular RL algorithms namely DQN (Mnih et al., 2013), PPO (Schulman et al., 2017), DDPG (Lillicrap et al., 2015), A2C (Mnih et al., 2016). Our benchmarking study shows that RL policies are resilient to quantization error, and the error docs not propagate due to the feodback loop between the actor and the environment.
Although our benchmarking study shows the potential benefits of applying quantization (i.c., no error propagation), we still need to apply quantization during RL training to realize real-world spcedups. Applying quantization during RL training may seem difficult duc to the myriad of different learning algorithms (Lillicrap et al., 2015; Mnih et al., 2016; Barth-Maron et al., 2018) and the complexity of these optimization procedures. Luckily, most popular RL training algorithms can be formulated using the actor-learner training paradigm such as Ape-X (Horgan et al., 2018) and ACME (Hoffman et al., 2020). ActorQ leverages the fact that many RL training procedures can be formulated in a general actor-learning paradigm in order to apply quantization specifically during actor's policy inference and policy broadcast between learners to actors during the RL training. To demonstrate the benefits of our approach, we choose two RL algorithms, namely DQN (Mnih
2

---

Published in Transactions Machine Learning Research (07/ 20222
17.5
0.20
Actors
15.0
Actors
Carbon (Kgs)
0.15
Learner
12.5
Leamer
Broadcast
Broadcast
Runtime
10.0
0.10
7.5
0.05
5.0
2.5
0.00
0.0
int8
fp32
int8
fp32
(a) Carbon breakdown
(b) Time breakdown
Figure 1: (a) Breakdown of carbon emissions between using a quantized and non-quantized policy in RL training for MountainCar. (b) Wall clock time for 'Actors', 'Learner', and 'Broadcast' for over a window of 1000 steps in ACME (Hoffman et al., 2020)
et al., 2013) and D4PG (Barth-Maron et al., 2018) from Deepmind's ACME framework (Hoffman al., 2020) and show speedups in training time and reduced carbon emissions while maintaining convergence.
In summary, our fundamental contributions are as follows:
We introduce ActorQ, speed up distributed reinforcement learning training. ActorQ operates by quantizing the actor's policy, thereby speeding up experience collection. ActorQ achieves between 1.5 x and 5.41 speedup variety of tasks from the Deepmind control suite (Tassa et al., 2018) and OpenAl gym (Brockman et al., 2016) compared to its full-precision counterparts.
Using our ActorQ framework, we further explore opportunities to identify various bottlenecks in distributed RL training. We show that quantization can also minimize communication overheads between actors and learners and reduce reinforcement learning time.
Finally, by quantizing the policy weights and communication between actors and learners, show reduction in carbon emissions between 1.9 x and 3.76x versus full precision policies, thus paving the way towards sustainable reinforcement learning.
To address lack of benchmarks in applying quantization for RL in the literature, we extensively benchmark quantized policies on standard tasks (Atari, Gym), algorithms (A2C, DQN, D4PG, PPO), and policy architecture (MLPs, CNNs). We demonstrate little to no loss in mean return, especially in the context of using quantized policy for rollouts.
2
Related Work
Both quantization and reinforcement learning in isolation have been the subject of much research in recent years. However, to the best of our knowledge, they have seldom been applied together (as surveyed in broad range of quantization and RL papers) to improve RL elficiency. Below we provide an overview of related works in both quantization and reinforcement learning and discuss their contributions and significance in relation to our paper.
2.1 Quantization
Quantizing a neural network reduces the precision of neural network weights, reducing memory transfer times and enabling the use of fast low-precision compute operations. Innovations in both post-training quantization (Krishnamoorthi, 2018; Banner et al., 2018; Zhao et al., 2019; Tambe
3

---

Published in Transactions Machine Learning Research (0/2222
Actor-Q
Metries
A3C
CULE
Ray
Gorila
Seed-RL
ACME
(Our Work)
Method
Distributed
Distributed
Distributed
Distributed
Distributed
Distributed
Distributel/Stanndaae
Increase in Speed-up
V
<
V
V
Decrease in
X
X
X
X
X
X
Carbon Emission
Decrease in Energy
X
X
X
X
Decrease in
X X X X X
Communication Cost
Framework Agnostic
X
X
Table 1:
Comparison of prior works on speeding-up RL training wrt to speed-up (lower training times), energy, and carbon emissions. Previous works compared include Nvidia's CULE (Dalton et al., 2019), Ray (Moritz et al., 2018), Gorila (Nair et al., 2015), Seed-RL (Espeholt et al., 2019) and ACME (Hoffman et al., 2020)
et al., 2020) and quantization aware training (Dong et al., 2019; Hubara et al., 2018; Choi et al., 2018) demonstrate that neural networks may be quantized to very low procision without accuracy loss, suggesting that quantization has immense potential for producing efficient deployable models. In the context of speeding training, rescarch has also shown that quantization can yicld significant performance boosts. For example, prior work half or mixed precision training (Sun et al., 2019; Das et al., 2018) demonstrates that using half-precision operators may significantly reduce compute and memory requirements while achieving adequate convergence.
Although much research has been conducted on quantization and machine learning, the primary largets of quantization are applications in the image classification and nat ural language processing domains. Quantization as applied to reinforcement learning has been absent in the literature.
2.2 Reinforcement Learning & Distributed Reinforcement Learning Training
Significant work on reinforcement learning range from training algorithms (Mnih et al., 2013; Levine et al., 2015) to environments (Brockman et al., 2016; Bellemare et al., 2013; Tassa et al., 2018) systems improvements (Petrenko et al., 2020; Hoffman et al., 2020). From a system optimization perspective, reinforcement learning poses unique opportunity compared to traditional supervised machine learning algorithms as training policy is markedly different from standard neural network training involving pure backpropagation (as employed learning image classification language models). Notably, reinforcement learning training involves repeatedly executing policies on environments (experience generation), communicating across various components (in this case between devices performing rollouts and the main device updating the policy), and finally, learning policy based the training samples obtained from experience generation. Experience generation is trivially parallclizable by employing multiple devices that perform rollouts simultaneously, and various recent research in distributed and parallel reinforcement learning training (Kapturowski et al., 2018; Moritz et al., 2018; Nair al., 2015) leverages this accelerate training. One significant work is the Deepmind ACME reinforcement learning framework (Hoffman et al., 2020), which enables scalable training to many processors nodes on a single machine.
3 ActorQ: Quantization for Reinforcement Learning
In this section, we introduce ActorQ a quantization method for improving the run time efficiency of actor-learner training. We first provide high-level overview of the ActorQ system. Then, we
4

---

Published in Transactions Machine Learning Research (07/ 2022)
Legend
Learner
Computation
Optinien
Opanisa
Learner
Wiights
Aaie Action, Rescrrtoo
drillop
Actor
Actor
Quantiaur
(a) Tradition RL training setup,
(b) ActorQ RL training setup.
Figure 2: (a) Traditional RT. training setup. In non-distributed RT, training scenario, the number of actors is 1, and all the components are run on the same machine. In distributed RL training setup (e.g,, ACME), the actors are distributed across multiple CPUs. (b) ActorQ system setup. In ActorQ, we add a quantizer block to the RL-training loop. The learner performs full- full-precision policy training in GPU. Before the policy broadcasted to the actors, the quantizer block quantizes the policy. The quantized policy reduces the communccttin of the updated policy between the GPU learner and CPU actors. The actors rollouts the quantized policy for the experience generation. The learner and actors are instrumented with carbon monitoring APIs (Henderson et al., 2020) to quantify the impact of carbon emission with and without quantization.
characterize the effects of quantization on different reinforcement learning algorithms. Lastly, we apply quantization to a distributed RL training framework to show speed-ups Our results demonstrate that apart from reducing training time, ActorQ also leads to lower carbon emissions, thus paving the way towards sustainable reinforcement learning research.
real system.
3.1 ActorQ System Architecture
Traditional RL training can be formulated within context of an actor- lcarner training paradigm (Hor- gan et al., 2018) as shown in Fig. 2a. In this framework, one or multiple actors collect new training examples based on its current policy and relays them to the learner (or replay buffer), which utilizes the generated experience to update the weights of its own policy. Periodically, the learner broadcasts its policy to the actors which update their internal policy with the learner's, 's synchronizing the models. In the non-distributed formulation, the actor, learner, and replay buffer reside in single machine. However, in the case of the distributed RL, the actor, learner, and the replay buffer run on different compute nodes, constantly communicating with each other. We refer to this communication between the actor/ /lcarner components as broadcast.
ActorQ introduces quantization in the actor-learner reinforcement learning framework to speed up training. There three main components in ActorQ system namely, 'Actors', 'Learner', and 'Quantizer' as shown in Figure? 2. During training, each actor instance performs rollouts and initially uses randomly initialized policy for decision making. At each step, the actors broadcast the environment state, action, and the reward for given state to the learner. The learner uses this information to optimize the policy. Periodically, the learner broadcasts the updated policy to the actors, who then use the updated policy to perform future rollouts.
There are two main performance bottlenecks in reinforcement learning training. First, each actor uses a neural network policy to generate an action. Thus, how fast it can perform rollouts depends the policy's inference latency. Second, the learner broadcasts the policy periodically to all the
5

---

Published in Transactions Machine Learning Research (07/202)
actors. Broadcasting of the entire policy network to all actors can cause communication overheads and slow down training.
In ActorQ, we use quantization to reduce these bottlenecks and achieve end-to-end speed-up. To systematically integrate quantization into the training algorithm, we first perform a study to characterize the effects of applying post-training quantization and quantization awarc training to various RL algorithms and environments.
In ActorQ, all actors use quantized policy to perform rollouts. Additionally, the broadcasted policy is also quantized. Note that ActorQ maintains all learner computation in full precision to maintain learning convergence; further note that the learner is significantly faster than the actors due to the utilization of hardware acceleration (e.g., GPU) to perform batched updates during policy optimization (Espeholt et al., 2019). Mcanwhile, actors are stuck performing rollouts that require repeated executions of single-example (non-batched) inference, and the difference in hardware utilization between the actors' rollouts and the Icarner's policy optimizations, based on characterization, is one of the key areas leading to inefficiency. This motivates to first apply quantization to the actors' inference and then to the policy broadcast (e.g: communication) between learners to actors.
While simple, ActorQ distinguishes from traditional quantized neural network training the inference-ouly role of actors enables the use of low precision (S bit) operators to speed up training. This is unlike traditional quantized neural network training, which must utilize more complex algorithms like loss scaling Das et al. (2018), specialized numerical representations Sun al. (2019); Wang et al. (2018), stochastic rounding Wang et al. (2018) to attain convergence. This adds extra complexity and may also limit speedup and, in many cases, this is still limited to half-precision operations due to convergence issues.
3.2 Effects of Quantization on Reinforcement Learning
As first step towards developing ActorQ; we first perform experiments to benchmark the effects of quantization on RL. Insights gained from these experiments will help verify that quantization can be applied to learning without significantly degrading quality. To this end, we apply post-training quantization (PTQ) to just the policies of various reinforcement learning agents: we take an RL policy fully trained in fp32 (flotiing point precision) and apply post-training quantization to it, evaluating the impact of quantization on the resulting model. Performing this experiment allows to understand whether the drift caused by quantization error at each policy step affects RL reward quality, and to what extent it impacts rewards with different degrees (i.e., number of bits) of quantization.
Post Tostnina Quantization (PTQ)
The post-training quantization is performed using standard uniform affine quantization (Krish- namoorthi, 2018) defined as follows:
W
Qn round
8
where
= minn (W, 0]] +
|ma W, W.O)
2n
Dequantization is defined as
D(W 8) 5(Wg)

---

Published in Transactions Machine Learning Research (07/2022
Algorithm
A2C
DQN
PPO
DDPG
Datatype
Ep32
fp16
int8
KL.div
Fp32
fp16
int8
KL.div
fp32
fp16
intä
KL.div
fp32
fp16
intS
KL-div
Euvironment
Rwd
Rwd
Rwd
Rwd
Rwd
Rod
Rwd
Rwd
Rwd
Rod
Rwd
Rwd
Breakout
379
371
350
0.00260
214
217
78
0.06015
400
400
368
0.08592
SpaceInvaders
717
667
634
04000066
586
625
609
0.01363
698
6662
694
D.08115
BeamRiderss
3087
3000
2793
0.00993
925
823
721
0.09787
1655
1820
1697
0.0186
McPacman
1915
1916
2046
0.17536
1433
14229
2024
0.01531
1735
17355
1846
0.11978
Qbort
5002
5002
0611
0.01957
641
641
616
0.01886
15010
10010
14426
0.02573
Seaquest
782
756
7:3
0.03358
1709
1885
1582
0.02536
1782
1784
1796
0.02991
Cartpole
500
500
500
0.00113
500
500
500
0.1019
500
5500
500
0.00586
Pong
19
0.01528
21
0.1257
20
20
0.01
Walker2D
339
422
442
0.05371
2274
2273
2208
0.03
1890
1924
1866
0.0376
HallChootah
2199
2215
2208
0.13427
30235
3032
3080
0.02
2553
2551
2473
0.0763
Bipeda/Walker
0.0252
83
MountainCar
0.05705
1/2
92
0.07
92
922
0.0651
Table 2: Post-training quantization error for DQN, DDPG, PPO, and A2C algorithm on Atari and Gym. Quantization down to int8 yields similar episodic rewards to full precision baseline. To measure the policy distribution shift, we use KL-divergence. We mcasure KL-divergence between the fp32 policy and the int8 policy. The action distribution for the fp32 policy and int8 is shown in the appendix.
In our study, policies are trained in standard full precision. Once trained, we quantize them to fp16 and int8 first and dequantization them to simulate quantization error. For convolutional neural networks, we use per-channel quantization, which applies Qn to each channel of convolutions individually. Also, all layers of the policy are quantized to the same precision level.
We apply the PTQ to Atari arcade learning (Bellemare et al., 2013), OpenAI gym environ- ments (Brockman et al., 2016) and different RL algorithms namely A2C (Mnih et al., 2016), DQN (Mnih et al., 2013), PPO (Schulman et al., 2017), and DDPG (Lillicrap et al., 2015). We train three-layer convolutional neural network for all Atari arcade learning. For openAI Gym environments, we train neural networks with two hidden layers of size 64. In PTQ, unless otherwise noted, both weights and activations quantized to the same precision.
Table shows the rewards attained by policies quantized via post-training quantization in. The mean of int8 and fp16 relative errors ranges between 2% and 5% of the full precision model, which indicates that policies can be represented in 8/16 bits precision without much quality loss.
a few cases (c.g., MsPacman for PPO), post-training quantization yiclds better scores than the full precision policy. We believe that quantization injected an amount of noise that was small enough to maintain good policy and large enough to regularize model behavior; this supports some of the results seen by Louizos et al. (2018); Bishop (1995); Hirose et al. (2018).
To quantify the shift in policy distribution, we use KI--divergence (Kullback & Leibler, 1951). Table shows the KL-divergence for the fp32 policy and int88 (quantized) policy. Across all the evaluated algorithms (both on-policy and off-policy), we observe that the KL-divergence is very small, suggesting that the quantization of policy does not change the inherent distribution significantly. The effect of small KL-divergence is also reflected in the minimal degradation in mean return.
We also visualize the action distribution of both the fp32 policy and int8. Fig. 3 shows the action distribution for two on-policy algorithms, namely A2C and PPO, for the Walker2D environment.

---

Published in Transactions Machine Learning Research (07/ 2022)
1500
1p32 policy
1500
fp32 policy
int8 policy
Int8 policy
Frequency
1000
Frequency
1000
500
500
-1.0
-0.5
0.0
0.5
1.0
1..0
-0.5
0.0
0.5
1.0
(a) A2C.
(b) PPO.
Figure 3: Small variation in action distribution for A2C and PPO in WalkerStand environment for fp32 and int8 policies for the same observation. We run each policy for 5000 steps. The small change in action distribution for quantized policy suggests safe exploration without significantly degrading the rewards.
We observe a small variation in the action distribution, suggesting that quantizing the policy allows the agent to perform a small safe exploration compared the fp32 policy. This small variation is consistent with other environments and RL algorithms. Appendix C shows the action distributions for other environments.
Based on this study, we observe that the quantization of RL policy does not cause a significant loss in reward compared to an fp32 policy. Appendix shows the results of applying quantization aware training (QAT) with RL. Since QAT uses fake quantization nodes to estimate the statistics of the parameter distributions, it allows for more aggressive quantizations. Our study on QAT suggests (See Appendix A) that we can safely quantize the policy up to 5-bits without significantly affecting the agent's rewards. However, to fully utilize the gains from aggressive quantization, we need native hardware support to run computations lower than 8-bits. Commonly used hardware accelerators used in RL training support native int8 computations. To that end, in ActorQ, we use PTQ for the quantizer block. Also, it important to note that ActorQ uses simple uniform affine quantization to demonstrate how to apply quantization in RL policies; however, we can easily swap the quantizer function to include other quantization techniques (Krishnamoorthi, 2018).
In summary, at int8, we see minimal degradation in rewards. Hence, when designing quantizer block in ActorQ, we quantize the policy to int8. By using int8 quantization and leveraging the native int8 computation support in hardware, we achieve end-to-end speed-up in reinforcement learning training.
4 Experimental Setup
We evaluate the AclorQ system for speeding up distributed quantized reinforcement learning across various tasks in Deepmind Control Suite (Tassa et al., 2018). Overall, we show that: (1) we see significant speedup (between 1.5 and 5.41 x) in training reinforcement learning policies using ActorQ; (2) convergence maintained even when actors perform int8 quantized inference; (3) Using ActorQ, we lower the carbon emissions between 1.9x and 3.76x compared to training without quantization.
4.1 ActorQ Experimental Setup
We evaluate ActorQ on a range of environments from the Deepmind Control Suite (Tassa et al., 2018). We choose the environments to cover wide range of difficulties to determine the effects of
8

---

Published in Transactions Machine Learning Research (07/ 2022)
Task
Algorithm
Steps Trained
Model Pull Frequency (Steps)
Cartpole Balance
D4PG
40000
1000
Walker Stand
D4PG
40000
1000
Hopper Stand
D4PG
100000
1000
Reacher Hard
D4PG
70000
1000
Cheetah Run
D4PG
200000
1000
Finger Spin
D4PG
200000
1000
Humanoid Stand
D4PG
500000
100
Humanoid Walk
D4PG
700000
100
Cartpole
DQN
60000
1000
Acrobot
DQN
100000
1000
MountainCar
DQN
200000
100
Table 3: Tasks evaluated using ActorQ range from casy to difficult, along with the steps trained for corresponding tasks, with how frequently the model is pulled on the actor side.
quantization on both easy and difficult tasks. The difficulty the Deepmind Control Suite tasks is determined by (Hoffman et al., 2020). Table 3 lists the environments tested on with their corresponding difficulty and number of steps trained. Each episode has a maximum length of 1000 steps, so the maximum reward for each task is 1000 (though this may not always be attainable).
Policy architectures are fully connected networks with three hidden layers of size 2048. We apply a Gaussian noise layer to the output of the policy network on the actor to encourage exploration; sigma is uniformly assigned between 0 and 0.2 according to the actor being executed. On the learner side, the critic network is three-layer hidden network with hidden size of 512. We train policies using D4PG (Barth-Maron et al., 2018) on continuous control environments and DQN (Mnih et al., 2013) on discrete control environments. We chose D4PG as it was the best learning algorithm in (Tassa et al., 2018; Hoffman et al., 2020), and DQN is widely used and standard reinforcement learning algorithm. An example submitted by an actor is sampled 16 times before being removed from the replay buffer (spi=16) (lower spi is typically better as it minimizes model staleness (Fedus et al., 2020)).
All the experiments are run in distributed fashion to leverage multiple CPU cores and GPU. A V100 GPU is used the learner, while the actors are mapped to the CPU (1 core for cach actor). We run each experiment and average over at least three runs to compute the running mean (window=10) of the aggregated runs.
4.2 Measuring Carbon Emissions
For measuring the carbon emission for the run, we use the experimenttimmaac--raaddd proposed in prior JMLR work (Henderson et al., 2020).² We instrument the ActorQ system with carbon monitor APIs measure the energy and carbon emissions for each training experiment in ActorQ.
To measure the improvement (i.c., lowering of carbon emissions) when using the int8 policy, we take the ratio of carbon emission when using the fp32 policy and int8 policy as:

---

Published in Transactions Machine Learning Research (07222
Carbon Emission
Time to Convergence (s)
Carbon Emissions (Kgs)
Improvement
Mean
Task
Speedup
fp32
int8
Return
fp32
int8
foan/
(paz)
(Co2mas)
Cartpole Balance
941.22
870.91
279
3.12x
0.359
0.15
2.39
Walker Stand
947.74
871.32
534.37
1.63x
0.67
0.178
3.76
Hopper Stand
836.41
2660.41
1699.17
1.57x
0.34
0.17
2.00
Reacher Hard
948.12
1597
875.34
1.82x
0.35
0.18
1.94
Choetah Run
732.31
2517.3
891.84
2.82x
0.263
0.12
2.19
Finger Spin
810.32
3256.56
1065.52
3.06x
0.361
0.19
1.90
Humanoid Stand
884.80
13964.92
9302.82
1.51x
0.55
0.27
2.04
Humanoid Walk
649.91
17990.66
6223.35
2.89x
0.56
0.278
2.01
Cartpole (Gym)
198.22
963.67
260.1
3.70x
0.188
0.089
2.11
Mountain Car (Gym)
-120.62
2861.8
1284.32
2.22x
0.21
0.098
2.14
Acrobot (Gym
-107.45
912.24
168.44
5.41x
0.198
0.097
2.04
Table 4: ActorQ time, speedups, carbon emission improvements to 95% reward on select tasks from DecpMind Control Suite and Gym. 8 bit inference yiclds between 1.5 x and 5.41 x specdup over full precision training. Likewise, int8 policy inference yields between 1.9x and 3.76x less carbon emission compared to fp32 policy. We use D4PG on DeepMind Control Suite environments (non-gym), DQN on gym environments.
CO2
Copmmproement
CO2int8
where, Co2 and Co2ints are the carbon emissions when using fp32 and int8 policy respectively.
5 Results
5.1 Speedup, Convergence, and Carbon Emissions
We Iocus our results on three main areas: end-to-end speed-ups for training, model convergence during training and environmental sustainability from a carbon cmissions perspective.
End-to-End Speedups. We show end-to-end training speedups with ActorQ in Figure(s) 4 and 6. Across nearly all tasks, see significant speedups with both 8-bit inference. Additionally, to improve rcadability, we estimate the 95% pereentile of the maximum attained score by fp32 and measure time to this reward level for fp32, int8, and compute corresponding speedups. This is shown in Table Note that Table 4 does not take into account cases where fp16 or int8 achieve a higher score than fp32.
Convergence.
We show the episode reward versus total actor steps convergence plots using ActorQ in Figure(s) 5 and 6. Data shows that broadly, convergence is maintained even with 8-bit actors across both casy and difficult tasks. On Cheetah, Run and Reacher, Hard, 8-bit ActorQ achieve even slightly faster convergence, and we believe this may have happened quantization introduces noise which could be scon as exploration.
Carbon Emissions. Table 4 also shows the carbon emissions for various task in openAl gym and Deepmind Control Suite. We compare the carbon emissions of policy running in fp32 and
10

---

Published in Transactions Machine Learning Research (07/ 2022)
1000
800
900
800
Retur
800
Return
Return
600
Return
600
700
400
400
400
500
9-8
200
0m32
0=8
200
q-8
200
9
A
6
A
55
100
3.5
7
Time (s)
1e6
Time (s)
Time (s)
Time (s)
1e6
(a) Cartpole Balance
(b) Cheetah Run
(c) Walker Stand
(d) Hopper Stand
900
600
Retur
600
800
600
Retur
600
500
400
500
400
300
200
q- 32
9-8
200
0-8
200
0
9-32
Q-8
100
32 q=
25 50
0.73,00
1.15
2
3
0.
00
6.
0.
9.9
1
2.
25 le7
Time (5)
1e6
Time (s)
1e0
Time
le7
Time
(e) Reacher Hard
(I) Finger Spin
(g) Humanoid Stand
(h) Humanoid Walk
Figure 4: End-to-end speedups of ActorQ across various Deepmind Control Suite tasks using 8 bit and 32 bit inference. int8 training yields significant end-to-end training speedups over the fp32 basclinc. The x-axis denotes the wall-clock time and y-axis denotes the reward. Training uses the D4PG algorithm.
1000
800
Q-32
800
Q-32
900
800
Retum
800
Retum
600
Retum
Return
600
600
700
400
600
400
500
q=332
200
0=32
8-8
200
400
0.0
0.5
1.0
1.5
200
Steps
Steps
Steps
Steps
(a) Cartpole Balance
(b) Cheetah Run
(c) Walker Stand
(d) Hopper Stand
700
800
800
Return
800
700
Return
600
500
600
400
400
480
300
200
8-32
300
Q-
100
Q-32
8
am8
8=
8=
0
6,6
2
0.2
6.6
8.8
0
6,,66
Steps 167
Steps
l07
Steps
1e8
Steps
1es
(e) Reacher Hard
(f) Finger Spin
(g) Humanoid Stand
(h) Humanoid Walk
Figure 5: Convergence of ActorQ across various Deepmind Control Suite tasks using int8 and fp32 inference. int8 quantized training attains the same or better convergence than full precision training. Training uses the D1PG algorithm.
int8. We observe that quantization of policies reduces the carbon emissions anywhere from 1.9x to 3.76x depending upon the task. As RL systems are scaled to run on 1000's distributed CPU cores and accelerators (GPU/TPU), the absolute carbon reduction (measured in Kgs of CO2) can be significant.
11

---

Published in Transactions Machine Learning Research (07/ 2022)
q-32
120
q-8
160
Return
140
Return
-00
140
160
300
120
100
4232
-180
-400
92
80
8=8
200
550
0.0
1.0
2.5
-200
3
0.5 1
1.5
2 Steps
30
3.5
Steps
Steps
1eb
1e5
(a)
(b) MntnCar(Conyy
(c) Acrobot (Conv)
200
100
-100
Return
120
150
Return
140
-200
140
120
8-
-180
-30
-350
200
q=8
-400
9-8
50
0.1
09
1.13,0
à
6
2.0
2
30 3.5
Time (s) le6
Time
166
166
(d) Cartpole(Spd)
(e) MntnCar(Spdd)
(f) Acrobot(Spd)
Figure 6: Convergence (Conv.) and end-to-end speedups (Spd) of ActorQ across various Gym tasks using int8 and fp32 inference. int8 training yields significant end-to-end training speedups over the full precision baseline. For the speed-up plots (d-f), the x-axis denotes wall-clock time. Training uses the DQN algorithm.
5.2 Communication vs Computation The frequency of model pulls on actors is a hyperparameter and may have impacts on convergence it affects the staleness of policies being used to populate the replay buffer; this has been witnessed in both prior research (Fedus et al., 2020) and our experiment with the hyperparameter. Figure 7a shows that a higher update frequency of 100 can help in faster convergence compared to an update frequency of 1000 for the Humanoid stand task. This hyperparameter has system-level implications since a higher update frequency can increase the commuicatiion cost (policy broadcast from learner to actors). In contrast, a lower update frequency can increase the computation cost since it will take more steps to converge, incrcasing the computation cost. Thus, to understand the tradcoff of quantization concerning this hyperparameter, we explore the effects of quantization of communication versus computation in both communication and computation-heavy setups.
To quantize communication, we quantize policy weights to int8 and compress them by packing them into a matrix, thus, reducing the memory of model broadcasts by 4x. Naturally, quantizing communication would be more beneficial in the communication heavy scenario, and quantizing compute would yield relatively more gains the computation-heavy scenario.
Figure 8 shows an ablation plot of the gains of quantization on both communication and compu- lation in communiction heavy scenario (Irequency=30) versus a computation-heavy scenario (frequency=300).
The figures show that in communication heavy scenario (Figure 8a), quantizing communication may yield up 30% speedup; conversely, in a compatation-heavy scenario (Figure 8b) quantizing communication has little impact as the overhead is dominated by computation. Therefore, we believe that communication would incur higher costs networked cluster as actors scale.
12

---

Published in Transactions Machine Learning Research (07/ /022)
800
1000
Return
600
800
update freq=100
Return
400
update freq=1000
600
update freq = 100
update _req 1000
200
400
0
200
100000
200000
300000
400000
%
20000
40000
60000
80000
100000
Steps
Steps
(a) Humanoid Stand.
(b) Walker Stand.
Figure 7: Studying the effect of training with more frequent actor pulls in Humanoid stand and Walker Stand. We observe that the more frequent actor pulls learns faster than with less frequent actor pulls and demonstrates model pull frequency affects staleness of actor policies and may have an effect on training.
1000
1000
900
900
800
800
Return
700
Return
700
600
500
q=32.q_c=32
400
q=32.a_c=32
300
9-32.0_c-8
300
200
q=8
200
q=8
250
500
750
1000
1250
1500
1750
2000
200
400
600
800
1000
1200
Time (s)
Time (s)
(a) Communication Heavy (Update Freq=30)
(b) Computation Heavy (Update Freq=300)
Figure 8: Effects of quantizing commuiccttin versus computation in compute heavy and communi- cation heavy training scenarios. is the precision of inference; c_c is the precision of communication. Note q= implicitly quantizes communication to 8 bits. Experiment run on the walker stand task, using the D4PG algorithm.
5.3 Rationale for Why Quantization of Actors Speed-Up RL Training
Wc further brcak down the various components contributing runtime on a single actor understand how quantization of actor's policy inference speeds up training. Runtime components are broken down into: step time, pull time, deserialize time, and _iet time. Step time is the time spent performing neural network policy inference. Pull time is the time between querying the Reverb queue (DeepMind, 2020) for model and receiving the serialized models' weights; deserialize time is the time spent deserialize the serialized model dictionary; load_state time to call Py Torch load_state___ dict (used for loading and storing the policy).2
state state_dict dict time is the
Figure 9a shows the relative breakdown of the component runtimes with fp32 (denoted as q=32), fp16 (dcnoted as q=16), and int8 (denoted as q=8) quantized inforence in the communication heavy scenario. It is communication heavy because every 30 steps (Update Freq=30), the learner's policy is updated to each actors. This causes increased communication between the learner and actor
hhttpe:/bit..l//pytorr api
13

---

Published in Transactions Machine Learning Research (07/ 2022)
10
Step Time
10
Step Time
Pull Time
Pull Time
8
Time (s)
Deserialize Time
Time (s)
Deserialize Time
load state_ _iicttedictteeee time
6
load state dict time
4
2
q=32
q=16
q=8
q=32
q=16
q=8
(a) Communication Heavy (Update Freq=30)
(b) Computation Heavy (Update Freq=300)
Figure 9: Breakdown of components for quantized and non-quantized training over 1000 steps.
components. As shown, step time is the main bottleneck, and quantization of the actor's policy significantly speed-up each roll-out, speeding up the overall training. Figure 9b shows the cost breakdown in the computation heavy scenario. While quantization speeds up the step time (Actor's policy inference), the pull time is also significantly reduced since there is less communication (Update Freq=300) happening between learner and actor. Quantization can still reduce the deserialize time and state dict time since quantization lowers the memory footprint compared to the fp32 scenario.
In int8 and fp16 quantized training, the cost of PyTorch load load_state state dict is significantly higher. An investigation shows that the cost of loading a quantized PyTorrh model spent repacking the weights from Python object into C data. Int8 weight repacking is noticeably faster than fp16 weight repacking due to fewer memory accesses. The cost of model loading suggests that additional speed gains can be achieved by serializing the packed C data structure and reducing the cost of weight packing.
6 Discussion & Future Work
To the best of our knowledge, our work is one of the first to experimentally and quantitatively demonstrated that quantization may be effectively applied to many facets of reinforcement learning, from obtaining high quality and efficient quantized policies, to reducing training times and eliminating carbon emissions. More specifically, we have shown that reinforcement learning policies may be quantized down to 4-5 bits (See Appendix A) without significantly affecting their performance; based on this result, we have developed a simple but effective method for speeding reinforcement learning training, ActorQ, which achieves between 1.5x and 5.41 x speedup over non quantized training, with a reduction in carbon emissions between 1.9x and 3.76x. In the future, alternative and more competitive methods are likely to emerge.
The computational requirements for RI. training growing (Espeholt et al., 2019) (Espeholt et al., 2018). Training OpenAl Five to play Dota required a scaled-up version of Proximal Policy Optimization running on 512 GPUs and 51200 CPU cores (Berner et al., 2019). As we scale RL training to more thousands of cores and GPUs, even 50% improvement we have experimentally demonstrated (Table 4) will result enormous savings in absolute dollar cost, energy, and carbon emissions. We believe that quantization, which is already a standard technique applied to non-reinforcement learning neural network models, will likewise be a critical technique in optimizing the performance of reinforcement learning policies. Our paper demonstrates that, like for standard neural networks, quantization yields significant benefits for reinforcement learning
14

---

Published in Transactions Machine Learning Research (07/2022
while maintaining accuracy. We believe our work is first step towards quantization being applied to reinforcement learning to achieve efficient, and environmentally sustainable training.
Several design decisions in our system warrant further discussion and research. In our design of the quantizer in ActorQ, we relied on simple uniform quantization, however, we believe other forms of aggressive quantization / compression (Park et al., 2016; Polino et al., 2018; Tambe et al., 2020; Lam et al., 2021) can also be applied (e.g,, distillation, sparsification, etc.). Applying more aggressive quantization / compression methods may yield additional benefits to the performance accuracy tradeoff obtained by the trained policies. Additionally, in order to achieve tangible speedups from quantization, the underlying hardware system must support quantized operations at the machine level. However, with incrcased hardware support for neural network exccution, we believe that devices in the future will exhibit an increasing amount of hardware support for quantized operations (Jouppi et al., 2017). Finally, in ActorQ. we primarily focused on quantizing actor neural network execution and the communication between actors and learners (as these were the biggest computational bottlenecks), however, believe that the learner's policy can similarly be quantized to achieve further performance benefits.
7 Conclusion
To the best of our knowledge, are the first to apply quantization to not only speed up RL training and inference, but also reduce RI. carbon emissions. We experimentally demonstrate that standard quantization methods can quantize policies down to 8 bits with little quality loss. We present ActorQ to attain significant speedups and reductions in carbon emissions over full precision training. Our results demonstrate that quantization has considerable potential in speeding up both reinforeement learning inference and training while reducing carbon emissions. Future work includes extending the results to networked clusters to evaluate further the impacts of communication and applying quantization to reinforcement learning to different application scenarios such as the edge.
Acknowledgements
The authors will like to thank Google for providing GCP research credits. We would also like to thank anonymous reviewers for their valuable feedback on the manuscript.
References
Mohammad Babaeizadeh, luri Frosio, Stephen Tyree, Jason Clemons, and Jan Kautz. Reinforcement learning through asynchronous advantage actor-critic on a gpu. ar preprint ar Xrv: Xix:::66111..0 1611:.011.060 2016.
Ron Banner, Yury Nahshan, Flad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for repld-deploymentt, 2018.
Gabricl Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabncy, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. 2018.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253 279, jun 2013.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz,
15

---

Published in Transactions Machine Learning Research (0/22)
Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schncider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. CoRR, abs/1912.066833 2019. URL http://arxiv. http:///wrr/bbaa.......0
C. M. Bishop. Training with noise is equivalent to tikhonov regularization. Newral Computation, 7 (1):108 116, Jan 1995. doi: 20.1162/neco.199..
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schncider, John Schulman, Jic Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01540, 2016. URL
http://arxiv.org orgg hhbs/1606.0150
Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robolics and Aulomation Letters, 4(2):2007-2014 April 2019. ISSN 2377-3766. doi: M.1109/LRA.2014 28999889
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. 2018.
Steven Dalton, luri Frosio, and Michael Garland. Gpn-accelerated atari emulation for reinforcement learning, 2019.
Dipankar Das, Naveen Mellempudi, Dheevatsa Mudigere, Dhiraj Kalamkar, Sasikanth Avancha, Kunal Banerjee, Srinivas Sridharan, Karthik Vaidyanathan, Bharat Kaul, Evangelos Georganas, Alexander Heinecke, Pradcep Dubcy, Jesus Corbal, Nikita Shustrov, Roma Dubtsov, Evarist Fomenko, and Vadim Pirogov. Mixed precision training of convolutional neural networks using integer operations. International Conference on Learning Representations (ICLR), 2018.
DeepMind. Reverb. https: http:/ggttttbbccc http:://github.ccomdeeeee wwt..///meeemmnn///eeeet.t 2020.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hes- sian aware quantization of neural networks with mixed-preeision. In Proceedings of the IEEE International Conference on Compuler Vision, pp. 293 30-3002241 2019.
Lasse Espcholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning (ICML), 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. 2019.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International Conference on Machine Learning (ICML), 2020.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pinean. Towards the systematic reporting of the energy and carbon footprints of ma- chine learning. Journal of Machine Learning Research, 21(218):1113, 2020. URL
http://jmlr.oor//eeee wttpppapers/vv2//2-311111 htm
16

---

Published in Transactions Machine Learning Research (07/2022
Kazutoshi Hirose, Ryota Uematsu, Kota Ando, Kodai Ueyoshi, Masayuki Ikebe, Tetsuya Asai, Masato Motomura, and Shinya Takamacda- Yamazaki. Quantization crror-based regularization for hardware-aware neural network training. Nonlinear Theory and Its Applications, IEICE, 9(4): 453-465, 2018. doi: 10.1587/nolta.9.
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabricl Barth-Maron, Foryal Bchbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang. Kate Baumli, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint ar arXiv:2006.00979, 2020.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In International Conference on Learning Representations (ICLR), 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-Yaniv, and Yoshua Ben- gio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1-30, 2018. URL http://jmlr.orgge h/tpa//rr//pappe////////1--- html.
Norman Jouppi, Al Borchers, Rick Boyle, Pierre-lue Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Cliff Young, Tara Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Ho, Doug Hogberg, John Hu, and Nan Boden. In-datacenter performance analysis of tensor processing unit. pp. 1-12, 06 2017. doi: 10.1145/37995563000226
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. CoRR, abs/1806. 10293, 10293.
2018. URL http:///xxxxv..rrabbbe
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning, In International Conference on Learning Representations (ICLR), 2018.
Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojeiech Jaškowski. Vizdoom: poom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.0820 08342, 2018.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-88)) 86, 1951.
Maximilian Lam, Zachary Yedidia, Colby R Banbury, and Vijay Janapa Reddi. Precision batching: Bitserial decomposition for efficient neural network inference on gpus. In 2021 30th International Conference on Parallel Architectures and Compilation Techniques (PACT), pp. 129-141, 2021. doi: 20.109//AACT522952222.1
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies, 2015.
17

---

Published in Transactions Machine Learning Research (07/2022)
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement lcarning, arXiv preprint arxiv: 509.009999999 2015.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling, Relaxed quantization for discretized neural networks. International Conference on Learning Represenlations (ICLR), 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01788 2016. URL http://arxiv.oo http:/bbs/602..1170
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I Jordan, et al. Ray: A distributed framework for emerging {AI} applications. In 13th { {USENIX} Symposium on Operating Systems Design and Implementation ((ODDI) 18), pp. 561-577, 2018.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, et al. Massively parallel methods for deep reinforcement learning. arXiv preprint orXiv:1507.04225 2015.
OpenAI. Openai five.
com/openai-five// 2018
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Matcusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.
Jongsoo Park, Sheng R. Li, Wei Wen, Hai Li, Yiran Chen, and Pradeep Dubey. Holistic spar- secnn: Forging the trident of accuracy, speed, and size. International Conference on Leorning Representations (ICLR), 2016.
Alcksci Petrenko, Zhchui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. International Conference on Learning Representations (ICLR), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint XrXii:1707.033// 2017.
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:181-503, 2016. URL http:///wttttttttoooocmoooootttn html.
http:// http://wwwr nature. htt.tuuur.co///atee//eeeeeee http:///aaeeeeooooo////////////a
18

---

Published in Transactions Machine Learning Research (07/202)
David Silver, Julian Schrittwieser, Karen Simonyan, loamnis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017.
Xiao Sun, Jungwook Choi, Chia-Yu Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi (Viji) Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishuan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems 32. 2019.
Thierry Tambe, En- Yu Yang, Zishen Wan, Yuntian Deng, Vijay Janapa Reddi, Alexander Rush, David Brooks, and Yeon Wei. Algorithm-hardware co-design of adaptive floating-point encodings for resilient deep learning inference. 2020 57th ACM/IEEE Design Aulomation Conference (DAC), pp. 1-6. IEEE, 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint or 1801. 2018.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, 2018.
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, et al. Sustainable ai: Environmental implications, challenges and opportunities. arXiv preprint ar arXiv:211100330 2021.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving Neural Network Quantization without Retraining using Outlier Channel Splitting. International Conference on Machine Learning (ICML), pp. 7543- 75525,52 June 2019.
19

---

Published in Transactions Machine Learning Research (07/ 2022)
Appendix
A Quantization Aware Training (QAT)
To understand how aggressively (i.c., number of bits) we can quantify the RL policics, we use quantization aware training (QAT). In QAT, the RL policy weights and activations are passed through the quantization function Qn during inference; during backpropagation the straight-through estimator is used as the gradient of Qn (Krishnamoorthi, 2018)
V Qn) =
Note that quantization aware training does not speed up training as all operations are still performed in floating-point. Quantization aware training is used primarily to train a model with simulated quantized weights and activations to evaluate the reward loss (if any) for given RL task. Native hardware and library support for sub int8 quantization can speed policy inference time with quantized execution.
Cartpole
BreakOut
SeaQuest
Reward
550
Reward
400
Reward
2000
PPO
500
200
1000
A2C
A2C
450
PPO
Fp
8
8
6
5
3
2
Fp
8
8
7
6
5
4 3
2
Fp
8
8
5
3
2
bit
bit
bit
Pong
MsPacman
QBert
Reward
20
Reward
2000
Reward
15000
0
1000
10000
PPO
5000
-20
PPO
0
Fp
8
8
7
6
5
4
3
2
Fp 8
7
4
Fp
8
8
6
5
4
bit
bit
bit
BeamRider
Spacelnvader
MountainCar
Reward
3000
Reward
1000
Reward
100
2000
500
0
1000
A2C
A2C
100
Fp
8
8
6
5
3
2
Fp
8
8
7
6
5
4
3
2
Fp
8*
8
5
4
bit
bit
bit
Walker2D
HalfCheetah
BiPedalWalker
Reward
2000
Reward
2000
Reward
100
0
1000
100
DDPG
DOPG
DDPG
Fp
8"
8
7
6
5 4
2
Fp 8"
8
7
6 bit
5
4
3
2
Fp
8"
8
7
6
5
4
3
2
bit
bit
Figure 10: Quantization aware training of PPO, A2C, and DDPG algorithms on OpenAl gym, Atari, and PyBullet. FP denotes fp32 and 8* is achieved by int8 post-training quantization.
We present rewards for policies quantized via quantization aware training on multiple environments and training algorithms in Figure 10. We train three-layer convolutional neural network for all Atari arcade learning. For Ap Gym environments, we train neural networks with two hidden layers of size 64. We train all agents for 10 Million steps. Since in QAT, the trainer inserts fake quantization node is the neural network graph (policy in the context of RL), and the statistics
20

---

Published in Transactions Machine Learning Research (07/ (022)
of weights distribution during training are collected before the weights can be quantized. The quantization delay is a hyperparameter (Krishnamoorthi, 2018) that controls when the quantization is applied to the weights. One can consider this the "warm-up" period to collect the policy weight distribution statistics before applying quantization. We tried different values of quantization delay and used 5M since that gave us the best results.
Our results (see Fig. 10) suggest that generally, the performance relative to the full precision baseline is maintained until 5/6-bit quantization, after which there is a drop in reward, suggesting that for several of these environments, we can quantize the policy to 5-bits. However, to achieve measurable speed-up during training, there is need for native hardware and library support (e.g., Nvidia Cuda/Intel MKL/ ARM Ncon support in TensorFlow, Pytorch, and JAX frameworks).
Broadly, at 8-bits, see no degradation in rewards. Hence, when applying quantization in ActorQ, we quantize the policy to 8-bits. By quantizing the policy at 8-bits and leveraging the native 8-bit computation support in hardware, we achieve end-to-end speed-up in reinforcement lcarning training.
25000
Wall clock time (s)
fp32
20000
int8
15000
10000
4
6
8
Number of actors
Wall clock time for training DQN with varying number of actors. The actor policies are trained in full precision (fp32) and int8. Benefits of quantization is agnostic RL training setup and is effective for both single actor RL (non-parallel version) as well as distributed RL training (number of actors greater than one).
Figure 11:
B Parallelization and Scaling
In this paper, we show that applying quantization to reinforcement learning training improves training speed and lowers carbon emissions. Though in Table explicitly compares with distributed RL training, we believe quantization can also improve single actor training.
The cost of RL training can be formulated as
C' = Cb.Na + Ca Co.Rp
Where, ng: number agents
np: number of processors
Cb cost for broadcasting (computed as difference between quantized and non-quantized Ca: cost for actor inference (computed as a difference between quantized and non-quantized
C,: cost of overhead when more than one processor is used.
Our proposed method offers (nn) saving where n is the number of actors. Of course, this formula works only in the ideal cases, and will downgrade in real computer systems that other processes
21

---

Published in Transactions Machine Learning Research (07/ 2022)
600
400
action action
action
Freqeency
400
Frequency
action
Frequency
300
action, 9
200
200
200
100
0.0
0.2
0.4
0.6
0.8
1.0
Action
Action
Action
(a) BiamRR
(b) Breakout.
(c) CartPole.
action action_q
150
action action g
Frequency
150
Frequency
400
action
Frequency
100
200
50
Action
Action
(d) MsPacman.
(e) Pong.
(f) Qbert.
action
action action a
action_
400
400
Tesuency
300
200
200
100
10
3
Action
Action
(g) Seaquest.
(h) SpaceInvaders.
Figure 12: Variation in the action distribution for DQN between fp32 policy and quantized policy (int8) several environments evalauted in Table 2. In the legends, 'action' and 'action q" corresponds to the actions taken by the fp32 and int8 policies.
and are subject to network traffic. Still these slowdowns are scaling with the number of processors, leaving overall linear improvement with different scalers depending the number of processors used. Fig. 11 demonstrates the linear relationship between training time (savings) as number of actors scale. It is also important to note that in the case of single actor (non-distributed scenario), quantization of actor's policy still improves the training speed since Ca is minimized due to quantized inference of actor's policy.
C Action Distribution Visualization In this section, we include the variation in the action distribution between fp32 policy and quantized policy (int8) for PPO and DQN for several environments evaluated in Table 2.
Methodology for visualizing action distributions. To visualize the action distribution variation between int8 and fp32 policy, we load both the policies for rollouts. First, for each algorithm and task (environment) combination, we run a rollout of 5000 steps. The same observation passed to the int8 and [p32 policies. Finally, the actions for both policies are logged and visualized.
Across different RL algorithms and tasks, we observe that the variation of the action between the fp32 policy and int8 policy is very small. Also, since the expected mean return between fp32 and int8 policies is very similar, it suggests that quantization of the policies facilitates safe exploration.
22

---

Published in Transactions Machine Learning Research (07 /022)
1000
action action _
Frequency
750
Frequency
Frequency
400
action_
500
100
200
250
action
0
3
0.0
0.2
0.4
0.6
0.8
1.0
2
Action
Action
Action
aa] Breakout.
(b) CartPole.
(c) MsPacman.
300
action
action
rreguency
_
150
g
Requeinc
200
300
Frequency
100
100
action
100
50
Action
Action
(d) Pong.
(e) Qbert.
(f) Seaquest.
300
action action_a .q
100
action action_
Frequency
-0.5
0.0 Action
0.5
1.0
Action
(g) SpaceInvaders.
(h) SpaceInvaders.
Figure 13: Variation in the action distribution for PPO between fp32 policy and quantized policy (int8) several environments evalauted in Table 2. In the legends, 'action' and 'action q" corresponds to the actions taken by the fp32 and int8 policies.
Also, from a hardware utilization perspective, quantized computations (int8) are faster and more energy-efficient than fp32 computation. Hence, quantization is simple yet effective strategy to improve the training speed of RL sustainably (i.e., lower carbon emissions).
23


Reviews:
[{"review_id": "50hD4aGOuP", "reviewer": "Reviewer_3T9V", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nOverall, it is a good empirical paper which provides new insight into low-cost and high-efficiency experience collection. The experimental results have demonstrated good speedup and convergence performance in common RL tasks.\n\nWeaknesses\n\n1. Effects of quantization are unclear. This paper adopts two types of quantization: post-training quantization (PTQ) and quantization aware training (QAT). Essentially, the quantized policy is an actor policy close to the full-precision policy. Although the authors argue that the difference is small enough to maintain a good policy and large enough to regularize model behaviour. Therefore, it is worth using some quantifiable metrics (e.g., KL divergence) to measure the distribution shift between quantized and full-precision policies.\n\n2. When using the Quantizer for collection experience, the policy shift mentioned above would break the on-policy assumption in some RL algorithms(e.g., PPO). However, it is unclear how to deal with this problem (correct me if I have missed something).\n\n3. In some distributed RL frameworks, like SeedRL, the policy training and inference are both on GPU. In this case, the policy inference can also be sped up, and the communication can be reduced. So it would be good to clarify if the proposed quantized can also be used in the frameworks like SeedRL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an actor-network quantization method, ActorQ, to speed up data collection in distributed RL training. Specifically, this work adds a Quantizer between Actors and Learners to perform rollouts. The Quantizer is an inference-only actor and enjoys the usage of low precision operators to speed up training. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Adding a clarification of the consequent consequence after using the Quantizer in on-policy RL algorithms.\n\n2. Adding a metric to measure the policy shift caused by quantized policy and discuss how it would affect the RL algorithm performance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6tPCkm2b24", "reviewer": "Reviewer_nimW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the idea of using quantization in RL is novel\n- using quantization is shown to reduce carbon emission, whereas RL carbon emission  should be a growing concern. Additionally, it is shown to speed up learning, which is more a nice side effect than the primary concern\n\nWeaknesses:\n- parts of the methods and experimental setup are unclear. \n- the empirical study has several flaws (see below)\n- the writing needs significant improvement, with several unclarities and many typos that could easily be avoided. Most of this is easy to fix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to use quantization in distributed reinforcement learning. They show that significant speed up and reduced carbon emission can be obtained by quantizing the policy used for collecting data, without quantizing the learner part.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nFigure 1 shows ‘Actors’, a ‘Learner’, and a ‘Quantizer’, but Fig.2 studies the impact of the Actor, Learner and Broadcast parts. Is there a way to be more consistent? E.g. can the quantizer be considered as responsible for broadcast?\n\nUp to section 3.2, the intent of the paper is rather clear, but then the distinction between post-training quantization (PTQ) and quantiza-\ntion aware training (QAT) blurs the message by introducing several unclarities.\n\nFirst, PTQ and QAT are presented as \"two types of quantization\", but it is unclear whether they are alternative quantization methods or if they can be combined. Second, what is described as QAT seems to only correspond to a study of the impact of the hardness of the quantization, thus I don't unsderstand the name \"quantization-aware training\". From Fig. 1, I rather understood that training is not affected by quantization, hence I don't get \"quantization-aware training\". \n\nThe confusion is aggravated by \"We train a three-layer convolutional neural network for all Atari games for 10 million steps, with a\nquant delay of 5M (quantization aware training starts at 5M steps).\" Why mention QAT in a section devoted to PTQ? I think I'm missing something about the methods that the authors did not properly explain. Based on the many typos in these sections (see below), I suspect they have been reorganized at the last minute and confusions have been introduced.\n\nIn PTQ, it is not clearly stated where dequantization is applied and whether it is applied in the real RL method or just for the sake of temporarily evaluating the method (\"to simulate quantization error\"). An explanation of your intent is missing here.\n\nThe quantization delay of 5M is mentioned but not explained nor justified. Why don't you start quantizing immediately? This must be explained.\n\nThe text about Table 3 mentions 8-bit and 16-bit precision, but in the table we can only see int8 and fp16. Something is inconsistent here.\n\nIn Table 3, you may add the relative impact (e.g. -X% or + X%) and put the most important ones in bold or color to highlight them.\n\nSection 4 describes the experimental setup, but Section 3.2 already contains experimental studies and results. I suggest starting the experimental section at 3.2 and asking a list of addressed questions: the first corresponding to Section 3.2 and the others to the rest of experimental questions.\n\np3:\n\"reinforcement learning poses a unique opportunity\" an opportunity for what? The claim is unclear.\n\nIn Table 4, the \"difficulty\" statement is quite subjective (though it comes from another author). By the way, why use \"N/A\" for the last three rather than \"trivial\" or \"easy\". I suggest removing this column or replacing it with something more objective (dimensionality, number of steps for SOTA convergence...?).\n\nWhy not use the same environments and algorithms in Section 3.2 and Section 4? \n\nThe caption of Table 5 says: \"8 bit inference yields > 1.5 × −2.5× speedup over full precision training\" but the column shows speedup between x1.51 and x5.41. Which is right?\n\nAs RL systems are scaled to run on 1000’s distributed CPU cores and accelerators (GPU/TPU), the carbon reduction can be significant.\n=> well, the rate is still the same, and there is no reduction on the GPU/TPU part, right?\n\nWhy not run the experiments of Fig. 9 on Humanoid, as in Fig. 8, where the phenomenon was assessed?\n\nThe difference in impact between Figs 9a and 9b is far from obvious, so if there is such a difference, it should be better outlined with a more appropriate graph.\n\nThe relationship between Fig. 2 and Fig. 10 needs to be better outlined. Is it just the \"broadcast\" part that is scrutinized?\n\n------------------------------\n\nMinor and/or local issues, typos:\n\nAbstract: > 1.5x - 2.5x is hard to parse. I could not make sense out of this before reading \"between 1.5x and 2.5x\" later on. This second formulation is clearer.\n\n\"versus reinforcement learning in various tasks. Across various settings, we show...\"\n=> versus reinforcement learning *algorithms* in various tasks and settings, We show...\n\nBy the way, you may introduce \"RL\" early in the text and use it consistently.\n\nFigure 2, which is referred to in page 2, should be moved more forward (page 1 or 2).\n\nthe the overhead\n\nthus paving way for => thus paving the way towards\n\n\"Quantization as applied to reinforcement learning has been relatively absent in the literature.\"\n=> remove \"relatively\" to be consistent with a previous claim\n\nDeepmind Acme => consistently use upper or lowercase. I suggest using \\sc (small capitals) for all algorithms/frameworks.\n\nthus paving the path for => thus paving the way towards (there are a few repetition which should be avoided)\n\nactors, who then uses => use\n\n\"This is unlike traditional quantized neural network training, which must utilize more complex algorithms like loss scaling Das et al. (2018), specialized numerical representations Sun et al. (2019); Wang et al. (2018), stochastic rounding Wang et al. (2018) to attain convergence.\"\n=> all refs should use \\citep instead of \\cite here.\n\n\"This adds extra complexity and may also limit speedup and, in many cases, are still limited\"\n=> and, in many cases, this is still limited...\n\nthe labels \"fp16, fp32, int8...\" should be explained at least once: floating point precision with 16 or 32 bits, or binary integers on 8 bits, etc.\n\nWe apply the PTQ to Atari arcade learning (Bellemare et al., 2012), OpenAI gym environments (Brockman et al., 2016b) and different RL algorithms namely A2C (Mnih et al., 2016), DQN (Mnih et al., 2013a), PPO (Schulman et al., 2017), and DDPG (Lillicrap et al., 2015) and.\n=> remove final \"and\" (or complete the sentence)\n\nTable 3 shows the rewards attained by policies quantized via post-training quantization in.\n=> in what???\n\nThe plot shows that the 8-bit quantization can sufficiently capture the entire weight distribution policy trained by PPO and A2C. Whereas for DQN, ...\n=> Whereas should not start a sentence\n\na gaussian noise layer => Gaussian\n\nend to end => end-to-end (several times)\n\nwith a 2.8× reduction in carbon emissions.\n=> with an average 2.8× reduction in carbon emissions.\n\n(e.g., distillation, sparsification, etc). => etc.). (missing dot)", "broader_impact_concerns": "There is no broader impact statement, but it would be easy to write a positive one, putting forward the concern for carbon emission due to growing research in ever more complicated environments in RL.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z0QxNrIF0u", "reviewer": "Reviewer_axpH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an interesting problem. It considers many different base algorithms that are modified with quantization and a few different environments.\n\nWeaknesses:\n- The technical details are not always clearly explained. For instance, specific implementation details required for each base algorithm are not given.\n- The writing and the presentation of results should be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies how quantization of the weights of neural networks can speed up deep reinforcement learning. The quantized distributed RL training system, ActorQ is shown to have faster convergence on a range of tasks (Deepmind Control Suite). The paper also discussed the carbon emissions (Kgs of CO2) of ActorQ versus standard reinforcement learning on various tasks.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The reason why there is an emphasis on parallelization in the paper (Table1, Section 2.2, etc.) is not perfectly clear. A lot of research is done to speed up RL training without requiring parallelization while still reducing training time/costs and this does not seem well explained. In Table 1, it is unclear why only parallelized algorithms have been chosen to compare with your contribution, while you specifically claim a reduction in energy/cost (which is not improved by parallelization).\n- In the current form, it is difficult to understand clearly what the different elements of the algorithm are. What is the broadcasted policy (from Section 3.1) for instance in the context of DQN? The following is not trivial but is given without much explanation: \"the learner is significantly faster than the actors due to utilizing the GPU (with actor policy inference achieving poor utilization on GPU due to operating over low batch sizes).\" How is the following sentence supported by the results \"Based on our characterization, we observe that time spent by actors to perform rollouts is significantly greater than that of learners\"?\n\n\nOther comments related to the writing that should be improved. \n- typos/sentences not completed such as \"and DDPG (Lillicrap et al., 2015) and.\"\n- in Figure 2, the run time is shown to be about 15 seconds for \"fp32\", but what does the 15 seconds corresponds to? Figure 2 is only mentioned beginning of Page 2 but it is place beginning of page 5. Should it be brought closer to the relevant discussion?\n- Table 3 is explained before Table 2 in the text and it makes things a bit difficult to follow.\n- Figure 3 should be improved to increase clarity in what the figure is about. The y-axis is denoted as \"frequency\" but the results seem to provide a count on the number of times a given weight is encountered. The title \"weight\" looks as being at the same level than \"DQN\", \"PPO\", \"A2C\".\n- Table 2, the term \"reward\" is not accurate I believe. Do you mean the experimental mean return (sum of rewards) over a trajectory?\n- Referring to \"the former\" and \"the latter\" in Section 3.2 to refer from elements from the previous paragraph make things difficult to follow. Writing \"Based on our analysis (see Figure 3)\" is not ideal because Figure 3 (or even its caption) does not provide much of an \"analysis\".\n- Things are not always carefully defined (e.g. \"fp32\")", "broader_impact_concerns": "Some claims about \"reducing training times and eliminating carbon emissions\", \"paving the path for sustainable reinforcement learning research\" should be put in perspective with the broader context. Improving RL algorithms might not directly lead to these objectives and more careful claims and discussions would be useful (e.g. it can be used to just take up all the budget for more hyper-parameters tuning, which can be useful but would not lead to any reduction in carbon emission).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper describes a new framework to speed up RL techniques by quantization.  This work makes an important contribution to the literature of quantization for RL as well as sustainable RL.  It includes a good empirical analysis that clearly shows the benefits of the framework.  While the paper was initially unclear, the authors made many revisions that addressed the concerns of the reviewers.  The paper is now ready for publication.
Decision: Accept (poster)