Quantized Reinforcement Learning (QuaRL)
Authors: Srivatsan Krishnan
Sharad Chitlangia
Maximilian Lam
Zishen Wan
Aleksandra Faust
Vijay Janapa Reddi
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Quantization

Abstract: Recent work has shown that quantization can help reduce the memory, compute, and energy demands of deep neural networks without significantly harming their quality. However, whether these prior techniques, applied traditionally to image-based models, work with the same efficacy to the sequential decision making process in reinforcement learning remains an unanswered question. To address this void, we conduct the first comprehensive empirical study that quantifies the effects of quantization on various deep reinforcement learning policies with the intent to reduce their computational resource demands. We apply techniques such as post-training quantization and quantization aware training to a spectrum of reinforcement learning tasks (such as Pong, Breakout, BeamRider and more) and training algorithms (such as PPO, A2C, DDPG, and DQN). Across this spectrum of tasks and learning algorithms, we show that policies can be quantized to 6-8 bits of precision without loss of accuracy. Additionally, we show that certain tasks and reinforcement learning algorithms yield policies that are more difficult to quantize due to their effect of widening the models' distribution of weights and that quantization aware training consistently improves results over post-training quantization and oftentimes even over the full precision baseline. Finally, we demonstrate the real-world applications of quantization for reinforcement learning. We use half-precision training to train a Pong model 50 % faster, and we deploy a quantized reinforcement learning based navigation policy to an embedded system, achieving an 18x speedup and a 4x reduction in memory usage over an unquantized policy.

Under review as a conference paper at ICLR 2020
QUANTIZED REINFORCEMENT LEARNING (QUARL)
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has shown that quantization can help reduce the memory, compute, and energy demands of deep neural networks without significantly harming their quality. However, whether these prior techniques, applied traditionally to image- based models, work with the same efficacy to the sequential decision making process in reinforcement learning remains an unanswered question. To address this void, we conduct the first comprehensive empirical study that quantifies the effects of quantization on various deep reinforcement learning policies with the intent to reduce their computational resource demands. We apply techniques such as post-training quantization and quantization aware training to spectrum of re- inforcement learning tasks (such as Pong. Breakout, BeamRider and more) and training algorithms (such as PPO, A2C, DDPG, and DQN). Across this spectrum of tasks and learning al algorithms, we show that policies can be quantized to 6-8 bits of precision without loss of accuracy. We also show that certain tasks and re- inforcement learning algorithms yield policies that are more difficult to quantize due to their effect of widening the models' distribution of weights and that quanti- zation aware training consistently improves results over post post-training quantization and oftentimes even over the full precision baseline. Additionally, we show that quantization aware training, like traditional regularizers, regularize models by in- creasing exploration during the training process. Finally, we demonstrate useful- ness of quantization for reinforcement learning. We use sulf-precision training to train Pong model 50% faster, and we deploy quantized reinforcement learning based navigation policy to an embedded system, achieving an 18 x speedup and a 4x reduction in memory usage over an unquantized policy.
INTRODUCTION
Deep reinforcement learning has promise in many applications, ranging from game playing Sil-
ver al. 2016 2017 Kempka et al., 2016) to robotics (Liliicrap et al. [015]] Zhang et al. 2015) to locomotion and transportation (Arulkumaran et al. [017]] Kendall et al.) 2018). However, the training and deployment of reinforcement learning models remain challenging. Training is expen- sive because of their computationally expensive demands for repeatedly performing the forward and backward propagation in neural network training. Deploying deep reinforcement learning (DRL) models is prohibitively expensive, if not even impossible, due to the resource constraints embed- ded computing systems typically used for applications, such as robotics and drone navigation.
Quantization can be helpful in substantially reducing the memory, compute, and energy usage of deep learning models without significantly harming their quality (Han al., 2015 Zhou al. 2016 Han et al.) 2016). However, it is unknown whether the same techniques carry over to reinforcement learning. Unlike models in supervised learning, the quality of a reinforcement learning policy de- pends on how effective it is in sequential decision making. Specifically, an agent's current input and decision heavily affect its future state and future actions; is unclear how quantization affects the long-term decision making capability of reinforcement learning policies. Also, there are many different algorithms train reinforcement learning policy. Algorithms like actor-critic methods (A2C), deep-q networks (DQN), proximal policy optimization (PPO) and deep deterministic policy gradients (DDPG) are significantly different in their optimization goals and implementation details, and it unclear whether quantization would be similarly effective across these algorithms. Finally, reinforcement learning policies are trained and applied to wide range of environments, and it is unclear how quantization affects performance in tasks of differing complexity.

---

Under review as a conference paper at ICLR 2020
Here, we aim to understand quantization effects on deep reinforcement learning policies. We com- prehensively benchmark the effects of quantization on policies trained by various reinforcement learning algorithms on different tasks, conducting in excess of 350 experiments to present represen- tative and conclusive analysis. We perform experiments over 3 major axes: (1) environments (Atari Arcade, PyBullet, OpenAl Gym), (2) reinforcement learning training algorithms (Deep-Q Networks, Advantage Actor-Critic, Deep Deterministic Policy Gradients, Proximal Policy Optimization) and (3) quantization methods (post-training quantization, quantization aware training).
We show that quantization induces a regularization effect by increasing exploration during train- ing. This motivates the use of quantization aware training, which we show demonstrates improved performance over post-training quantization and oftentimes even over the full precision baseline. Additionally, We show that deep reinforcement learning models can be quantized to 6-8 bits of precision without loss in quality. Furthermore, we analyze how each axis affects the final perfor- mance of the quantized model develop insights into how to achieve better model quantization. Our results show that some tasks and training algorithms yield models that are more difficult to apply post-training quantization as they widen the spread of the models' weight distribution, yield- ing higher quantization error. To demonstrate the usefulness of quantization for deep reinforcement learning. we 1) use half precision ops to train a Pong model 50% faster than full precision training and 2) deploy quantized reinforcement learning based navigation policy onto an emhedded system and achieve an 18x speedup and a 4x reduction in memory usage over an unquantized policy.
RELATED WORK
Reducing neural network resource requirements is an active research topic. Techniques include quantization (Han et al. 2015: 2016 Zhu et al. 2016 Jacob et al.. 2018 Lin et al. 2019 Polino
ctt al., 2018: Sakr& & Shanbhag 2018). deep compression (Han et al. 22016), knowledge distilla-
tion (Hinton et 2015|Chhe Chen et al. 2017), sparsification (Han et al. al.] [016 Alford et al. 2018 Park
iet al. 2016 Louizos et al. 20 8b) Bellee et al. 2017) and pruning (Alford et al. 2018 Molchanov al. 2016 Li et al. 2016). These methods are employed because they compress reduce storage and memory requirements as well as enable fast and efficient inference and training with specialized operations. We provide background for these motivations and describe the specific techniques that fall under these categories and motivate why quantization for reinforcement learning needs study.
Compression for Memory and Storage: Techniques such as quantization, pruning, sparsifica- tion, and distillation reduce the amount of storage and memory required by deep neural networks. These techniques are motivated by the need to train and deploy neural networks on memory- constrained environments (e.g.. IoT or mobile). Broadly, quantization reduces the precision of net- work weights (Han et al.) [2015] 2016 Zhu et al./ 2016). pruning removes various layers and filters of network (Alford et al. 20118 Molchanov et al. 2016)). sparsification zeros out selective network values (Molchanov et al. [2166] Alford et al. 2018) and distillation compresses an ensemble of net- works into one (Hinton et al. 2015) Chen et al. 2017). Various algorithms combining these core techniques have been proposed. For example, Deep Compression (Han et al., 2015) demonstrated that a combination of weight-sharing, pruning, and quantization might reduce storage requirements by 35-49x. Importantly, these methods achieve high compression rates at small losses in accuracy by exploiting the redundancy that is inherent within the neural networks.
Fast and Efficient Inference/Training Methods like quantization, pruning, and sparsification may also be employed to improve the runtime of network inference and training as well their en- ergy consumption. Quantization reduces the precision of network weights and allows more efficient quantized operations to be used during training and deployment, for example, a "binary" GEMM (general matrix multiply) operation (Rastegari et al. 2016 Courbariaux al. 2016). Pruning speeds up neural networks by removing layers or filters to reduce the overall amount of computation neces- sary to make predictions (Molchanov et al. 20. Finally, Sparsification zeros out network weights and enables faster computation via specialized primitives like block-sparse matrix multiply (Ren) let al. 2018). These techniques not only speed up neural networks but decrease energy consumption by requiring fewer floating-poin operations.
Prior work in quantization focuses mostly on quan- tizing image/ / supervised models. However, there are several key differences between these models and reinforcement learning policies: an agent's 's current input and decision affects its future state and actions, there are many complex algorithms (e-g: DQN, PPO, A2C, DDPG) for training, and there are many diverse tasks. To the best of our knowledge. this is the first work to apply and analyze the performance of quantization across broad of reinforcement learning tasks and training algorithms.
Quantization for Reinforcement Learning:
2

---

Under review as a conference paper at ICLR 2020
3 QUANTIZED REINTORCEMENT LEARNING (QUARL)
We develop QuaRL, an open-source software framework that allows us to systematically apply tradi- tional quantization methods to a broad spectrum of deep reinforcement learning models. We use the QuaRL framework 1) evaluate how effective quantization is at compressing reinforcement learn- ing policies, 2) analyze how quantization affects/is affected by the various environments and training algorithms in reinforcement learning and 3) establish standard on the performance of quantization techniques across various training algorithms and environments.
Environments: We evaluate quantized models on three different types of environments: OpenAI gym (Brockman al. 2016), Atari Arcade Learning (Bellemare et al. (012), and PyBullet (which is an open-source implementation of the MuJoCo). These environments consist of a variety of tasks, including CartPole, MountainCar, LunarLandas Atari Games, Humanoid, etc. The complete list of environments used in the QuaRL framework is listed in Table Evaluations across this spectrum of different tasks provide a robust benchmark on the performance quantization applied to different reinforcement learning tasks.
We study quantization four popular reinforcement learning algorithms, namely Advantage Actor-Critic (A2C) (Mnih et al. 2016), Deep Q-Network (DQN) (Mnih et al. 2013). Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015) and Proximal Policy Optimization (PPO) (Schulman al. 2017). Evaluating these standard reinforcement learning al- gorithms that are well established in the community allows us to explore whether quantization is similarly effective across different reinforcement learning algorithms.
Training Algorithms:
Quantization Methods: We apply standard quantization techniques to deep reinforcement learning models. Our main approaches are post-training quantization and quantization aware training. We apply these methods models trained in different environments by different reinforcement learning algorithms to broadly understand their performance. We describe how these methods are applied in the context of reinforcement lcarning below.
Alporibm
UpenA Cymn
Atari
PsBulltt
Cartpole
MountalnCur
(Condinouus)
BeamRider
Breakout
McPaccan
Pong
Qbert
Seaquod
Spacelavader
HapodalValker
HaltCheetah
Walker2D
DON
PTQ
n/a
PIQ
PTQ
PTQ
PTQ
PT12
P1Q
PTQ
n/a
a/a
n/a
PTO
PTQ
PTQ
P10
P10
PTQ
PTQ
PTQ
A2C
QAT
QAT
QXT
9A7
QAT
QAT
QAT
QNT
PTQ
PTQ
PTQ
PTQ
710
PTQ
PTQ
PTQ
PРО
QAT
QAT
QAT
QAT
QAT
QAI
QAT
QAT
PTQ
n//m
PTQ
n/a
m/m
6/*
m/s
n/a
QAT
QAT
Table 1: Summary of algorithms, environments, and quantization scheme the QuaRL frame- work. PTQ means post-training quantization, QAT refers to Quantization-Awan corresponds to evaluating the policy from 8-bits to 2-bits. The Atari games the frameskip ver- sions with 4 frames stacked as input to the models. n/ means we cannot evaluate the combination due to experiments, conduct over 350 experiments to present deep understanding of how quantization affects deep reinforcement learning. This the first such (comprehensive) study.
Training, BW
incompatibility. All put together, including the individual bitwidth
3.1 POST-TRAINING QUANTIZATION
Post-training quantization takes a trained full precision model (32-bit floating point) and quantizes its weights lower precision values. We quantize weights down to fp16 (16-bit floating point) and int8 (8-bit integer) values. fp16 quantization based on IEEE-754 floating point rounding and int8 quantization uses uniform affine quantization.
Fp16 Quantization: Fp16 quantization involves taking full precision (32-bit) values and mapping them to the nearest representable 16-bit float. The IEEF-755 standard specifies 16-bit floats with the format shown below. Bits are grouped to specify the value of the sign (S), fraction (F) and exponent (E) which then combined with the following formula to yield the effective value of the float:
3

---

Under review as a conference paper at ICLR 2020
Exporent (5 bits)
Fradtion alla)
Sign
F 210) 2
Vine (-1) (1
In subsequent sections, we refer to f quantization using the following notation:
Uniform Affine Quantization: Uniform affine quantization (TensorFlow 2018b) applied to a full precision weight matrix and is performed by 1) calculating the minimum and maximum values of the matrix and dividing this range equally into 2" representable values (where n is the number of bits being quantized to). As each representable value is equally spaced across this range, the quantized value be represented by integer. More specifically, quantization from full precision n-bit integers is given by:
Qn(W) W
+ where 8 mmnnWW,0) + |maz W. 0)]
-m. 0)
Note that ò is the gap between representable numbers and 2 is an offset that exactly repre- sentable. Further note that we use minn (W, 0) and max (W, 0) to ensure that 0 is always represented. To dequantize we perform:
D(W g. 6, 5(W,
In the context of QuaRL, int8 and fp16 quantization applied after training full precision model on an environment, as per [ post training quantization, uniform quantization is applied to each fully connected layer of the model (per-tensor quantization) and is applied to each channel of convolution weights (per-axis quantization); activations are not quantized. We use post-training quantization to quantize to fp16 and int8 values.
Algorithm 1: Post- Dest-Trainin Quantization for Reinforce- ment Learning
Algorithm 2: Quantization Aware Training for Reinforcement Learning
Output: Reward
Input: T: task or environment
Input: T task or environment
Input: L reinforcement learning algorithm
Input: L: reinforcement learning algorithm
Input: quantize bits
Input: A model architecture
Input: A model architecture
Input: Qd quantization delay
Input: n quantize bits (8 or 16)
A, SoeertAfffeeeegghhAAAAAAAtttiitt
2 M, T Tensor inn Mares
Output: Reward
L, Ag.
M TrainnT, L, A)
Qd)
n = 8 Qints
3 M Train TrainWithQuumiizzaiinfff. Me
Q/p16 n = 16 return Eval(Q(MM)
Q
Qtrain)
return Eval(M, Q Qtrain Tensor
3.2 QUANTIZATION AWARE TRAINING
Quantization aware training involves retraining the reinforcement learning policies with weights and activations uniformly quantized to bit values. Importantly, weights are maintained in full fp32 precision except that they are passed through the uniform guantization function before being used in the forward pass. Because of this, the technique is also known as "fake quantization" (Tensor- Flow 2018b). Additionally, to improve training there is additional parameter, quantization delay (TensorFlow 2018a), which specifies the number of full precision training steps before enabling quantization. When the number of steps is less than the quantization delay parameter, the minimum and maximum values of weights and activations are actively monitored. Afterwards, the previously
4

---

Under review as a conference paper at ICLR 2020
captured minimum and maximum values used to quantize the tensors (these values remain static from then on). Specifically:
Qurai (W, Vinin Vman)
W
+ where 8 = Viiin + |Vman
Vmnin
8
2
Where Vmin and Vmar are the monitored minimum and maximum values of the tensor (expanding necessary). Intuitively. the expectation is that the training pro-
V Vmnn and Vmaz to include cess eventually learns to account for the quantization error, yielding a higher performing quantized model. Note that uniform quantization is applied to fully connected weights in the model (per-tensor quantization) and to each channel for convolution weights (per-axis quantization). n bit quantization is applied to each layer's weights and activations:
1k+1 =A(Q (W&, Vinnn Vmaa) a + b) where A is the activation function
ax+1 = Qrrain (IK+1: Vinin Vinax)
During backward propagation, the gradient passed through the quantization function unchanged (also known as the stright-thhoouhh estimator (Hinton 2012)). and the full precision weight matrix W is optimized follows:
A w Qtrain (W, Vinnn Viax)
In context of the QuaRL framework, the policy neural network is retrained from scratch after insert- ing the quantization functions between weights and activations (all else being equal). At evaluation full precision weights are passed through the uniform affine quantizer to simulate quantization error during inference. Algorithm 2 describes how quantization aware training is applied in QuaRL.
4 RESULTS
In this section, first show that quantization has regularization effect reinforcement learning algorithms and can boost exploration. Secondly, We show that reinforcement learning algorithms can be quantized safely without significantly affecting the rewards. To that end, we perform evalua- tions across the three principal axes of QuaRL: environments, training algorithms, and quantization methods.For post-training quantization, we evaluate each policy for 100 episodes and average the rewards. For Quantization Aware Training (QAT), we train atleast three policies and report the mean rewards over hundred evaluations. Table I lists the space of the evaluations explored.
Quantization as Regularization: To further establish the effects of quantization during training, we compare quantization-awwett aware training with traditional regularization techniques (specifically layer-norm (Ba al.) 2016 Kukacka et al. (017)) and measure the amount of exploration these techniques induce. Tt has been show in previous literature (Farebrother et al.] 2018 Cobbe et al.) 2018) that regularization actively helps reinforcement learning training generalize better; here we further reinforce this notion and additionally establish a relationship between quantization, generalization and exploration. We use the variance in action distribution produced by the model as proxy for exploration: intuitively, since the policy samples from this distribution when performing an action, a policy that produces an action distribution with high variance is less likely to explore different states. Conversely, a low variance action distribution indicates high exploration as the policy is more likely to take a different action than the highest scoring one.
We measure the variance in action distribution produced by differently trained models (QAT- 2, QAT-4, QAT-6, QAT-8, with layer norm and full precision) at different stages of the training process. We collect model rewards and the action distribution variance over several rollouts with deterministic action selection (model performs the highest scoring action). Importantly, sure to use deterministic action selection to ensure that the states reached are similar to the the distribution seen by the model during training. To separate signal from noise, we furthermore smooth the action variances with smoothing factor of .95 for both rewards and action variances.
make
5

---

Under review as a conference paper at ICLR 2020
Action Distribution Variance
0.08
80
0.06
60
0.04
40
06712
QAT-4
0.02
OAT6
047-5
20
0.00
Full Fruddces Eussine
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2
0.4
0.6
0.8
1.0
Step
Step
1xt
Figure 1: Exploration with different training processes and rewards achieved by corresponding mod- els. Lower variance in inferred action distribution implies higher exploration. Training with higher quantization levels, like layer norm regularization, induces lower action distribution variance and thus higher exploration. Reward plot indicates that training with quantization achieves similar level of rewards despite more exploration. Note that quantization during training is turned on after 5,000,000 steps (quant delay 5,000,000) and the differences manifest shortly after this point.
Figure shows the variance in action distribution produced by the models at different stages of training. Training with higher quantization levels (c.g: bit vs 4 bit training), like layer norm regularization, induces lower action distribution variance and hence indicates more exploration. Furthermore, figure 4 reward plot shows that despite lower action variance, models trained with quantization achieve a reward similar to the full precision bascline, which indicates that higher ex- ploration is facilitated by quantization and not by a lack of training. Note that quantization is turned on at 5,000,000 steps and we see its effects on the action distribution variance shortly after this point. In summary, data shows that training with quantization, like traditional regularization, in part regularizes reinforcement learning training by facilitating exploration during the training process.
Elfectiveness of Quantization: To evaluate the overall effectiveness of quantization for deep rein- forcement learning, we apply post-training quantization and quantization aware learning to a spec- trum of tasks and record their performance. We present the reward results for post-training quanti- zation in Table 2] We also compute the percentage error of the performance of the quantized policy relative to that of their corresponding full precision baselines pE1p and Emnss). Additionally, we report the mean of the errors across tasks for each of the training algorithms.
The absolute mean of 8-bit and 16-bit relative errors ranges between 2% and 5% (with the exception of DQN), which indicates that models may be quantized to 8/16 bit precision without much loss in quality. Interestingly, the overall performance difference between the 8-bit and 16-bit post-training quantization is minimal (with the exception the DQN algorithm, for reasons we explain Sec- tion|4] We believe this is because the policies weight distribution is narrow enough that 8 bits is able capture the distribution of weights without much error. In few cases, post-training quantization yields better scores than the full precision policy. We believe that quantization injected an amount of noise that was small enough to maintain good policy and large enough to regularize model be- havior; this supports some of the results seen by Louizos et al. (2018)); Bishhpp (1995); Hirose al. 2018); see appendix for plots showing that there is a sweet spot for post-training quantization.
For quantization aware training, we train the policy with fake-quantization operations while main- taining the same model and hyperparameters (see Appendix). Figure 2 shows the results of quantiza- tion aware training on multiple environments and training algorithms to compress the policies down from 8-bits to 2-bits. Generally, the performance relative the full precision bascline is maintained until 5/6-bit quantization, after which there a drop in performance. Broadly, at 8-bits, we see no degradation in performance. From the data, we see that quantization aware training achieves higher rewards than post-training quantization and also sometimes outperforms the full precision baseline.

---

Under review as a conference paper at ICLR 2020
Alquriitm
A2C
DQN
PPO
DDPG
Dntatype
[y32
fple
Fyst
fp
Eavinmmentt
RRdd Rivd
(%]
Rwd
Russ (8)
Red
Rod
(8) Red Ru (7))
Rvd
Rod
E (75)
Red
Rod Red
Rryss
15
Red
Bissh (55
Breakiou
371
2.11
7.65
214
400
368
specenccdees:
787 667
6.97
639
11.86
596
625
4.86
1311
698
662
5.86
684
2.01
Beamitdder
3687 3060
0.87
2793
9.52
922
822
$1.03
721
22,15
1658
1826
-9.97
1697
-2.44
1913 1915
3045
-6.79
1846
Obeet
5002 5082
0.00
5611
-12.18
641
6411
11:00
616
3,90
15010
15810
0.80
14425
3.96
756
CarcPole
500
500
0.00
500
0.00
500
500
0.00
500
0.00
500
500
0.00
500
0.00
Walker20
[8W1
1929
-2.06
1866
HallChretsll
2553
2561
IL18
2473
3,13
BipolaWallkrr
Mountaint'ar
92 92
1L:0
92
0,00
Table 2: Post training quantization error for DQN, DDPG, PPO, and A2C algorithm. The "Rwd" column corresponds to the rewards. The negative error percentage means the quantized policy per- formed better than fp32 policy. We summarize the enror in rewards using arithmetic mean.
Cartpole
BreakOut
SeaQuest
Pong
Reward
650
Reward
400
Reward
2000:
Reward
20
500
200
1000
450
A00
0
20
Fp 8
Fp
8
7
Fp 6'
8
4
8p
4
bit
bit
bit
bit
MsPacman
QBrt
BeamRider
Spacelnvader
Feward
2000
1000
Roward
15000
10000
5000
Reward
3000
2000
1000
Reward
1000
500
Fp
Fp
Fp
Fp
MountainCar
Walker2D
HalhCheetah
BiPodaWWalker
Reward
100
Pleaard
2000
Reward
2000
Reward
100
1000
100
100
8p
8
2
8p
FD
7
Fp
60
bi
68
bit
Figure 2: Quantization aware training (QAT) of PPO, A2C, and DDPG algorithms OpenAl gym, Atari, and PyBullet. FP is achieved by fp32 and 8* is achieved by 8-bit post-training quantization.
Effect of Environment on Quantization Quality: To analyze the task's effect on quantization quality plot the distribution of weights of full precision models trained in three environments (Breakout, Beamridd der and Pong) and their error after applying 8-bit post-training quantization on them. Each model uses the same network architecture, is trained using the same algorithm (DQN) with the same hyperparameters (see Appendix).
Figure 3 shows that the task with the highest emror (Breakout) has the widest weight distribution, the task with the second-highest error (BeamRi der) has a rower weight distribution, and the task with the lowest enror (Pong) has the narrowest distribution. With an affine quantizer, quantizing a narrower distribution yields less enror because the distribution can be captured at a fine granularity; conversely, a wider distribution requires larger gaps between representable numbers and thus in- creases quantization error. The trends indicate the envi- ronment affeets models' weight distribution spread which affects quantization performance: specifically, environ- ments that yield a wider distribution of model weights are more difficult to apply quantization to. This observation suggests that regularizing the training process may yield better performance.
Environment
Ey
10
RearRiddi
10
Breakout
63.55%
10
BeamRider
22.05%
Pong
0%
3
2
-1
0
Figure 3: Weight distribution and cor- responding 8-bit quantized error for models trained on the Breakout, Beamrides and Pong environments with DQN.

---

Under review as a conference paper at ICLR 2020
Frequnny Frequancy Frequennyy
DON
Min Wwight-2222
-2.21
Max Weight
-2.0
-1.
-1.0
40.5
0.0
0.5
1.0
Algorithm
Environment
fp32 Reward
Exuss
Eyats
PPO
Min Weight--..
PPO
Breakout
55%
0.00%
Max Weight 0.58
A2C
Breakout
379
7.65%
2.11%
.2.0
-1.
-1.0
-.5
0.0
0.6
1.0
A2C
Table Rewards for DQN, PPO, and A2C.
T0 3.
Min Weight--
Weight
0.72
-1.5
-1.0
0.0
1,0
weight
Figure 4: Weight distributions for the policies trained using DQN, PPO and A2C. DQN policy weights more spread out and more difficult to cover effectively by 8-bit quantization (yellow lines). This explains the higher quantization error for DQN in Table A negative error indicates that the quantized model outperformed the full precision baseline.
To determine the effects of the reinforce- ment learning training algorithm on the performance of quantized models, we compare the perfor- mance of post-training quantized models trained by various algorithms. Table 3 shows the error of different reinforcement learning algorithms and their corresponding 8-bit post-training quantiza- tion error for the Atari Breakout game. Results indicate that the A2C training algorithm is most conducive to int8 post-training quantization, followed by PPO2 and DQN. Interestingly, we see a sharp performance drop compared to the corresponding full precision baseline when applying 8-bit post-training quantization to models trained by DQN. At 8 bits, models trained by PPO2 and A2C have relative errors of 8% and 7.65%, whereas the model trained by DQN has an error of 64%. To understand this phenomenon,
Effect of Training Algorithm on Quantization Quality:
plot the distribution of model weights trained by each algorithm, shown in Figure 4 The plot shows that the weight distribution of the model trained by DQN is significantly wider than those trained by PPO2 and A2C. A wider distribution of weights indicates higher quantization error, which explains the large error of the 8-bit quantized DQN model. This also explains why using more bits (fp16) is more effective for the model trained by DQN (which reduces error relative to the full precision baseline from ~64% down to 1.4).. These results signify that the choice of RL. algorithms (on-policy vs off- policy) have different objective functions and hence can result in a completely different weight distribution. A wider distribution has more pronounced impact on the quantization error.
CASE STUDIES
To show the usefulness of our results, we use quantization to optimize the training and deployment of reinforcement learning policies. We 1) train pong model 1.5x faster by using mixed precision optimization and 2) deploy a quantized robot navigation model onto a resource constrained embed- ded system (RasPi-3b), demonstrating 4x reduction in memory and an 18x speedup in inference. Faster training time means running more experiments for the same time. Achieving speedup on resource-constrained devices enables deployment of the policies on real robots.
Mixed/Half- Preccision Training: Motivated by that reinforcement learning training is robust to quantization error, we train three policies of increasing model complexity (Policy A, Pol icy B, and Policy ey C) using mixed precision training and compare its performance to that of full precision training (see Appendix for details). In mixed precision training, the policy weights, activations, and gradients are represented in fp16. A master copy of the weights are stored in full precision (fp32) and updates are made during backward pass (Micikevicius et al. 2017). We measure the runtime and convergence rate of both full precision and mixed precision training (see Appendix).
PollcyA
Pollyy B
Polly
Algorithm
Network Parameter
fp32 Runtime (min)
MP Runtime (min)
Speedup
20
10'
10
Policy A
127
156
0,87
Reward
DQN-Pong
Policy
179
172
LO1x
6
-10
-10
Policy
1.61x
-20
F2
-20
FU92
Table Mixed precision training for rein- forcement learning.
Figure 5: Mixed precision v/s fp32 training rewards.
Figure 5 shows that all three policies converge under full precision and mixed precision training. In- terestingly, for Pol Policy B, training with mixed precision yields faster convergence; we believe that
8

---

Under review as a conference paper at ICLR 2020
some amount of quantization error speeds up the training process. Table shows the computational speedup to the training loop by using mixed precision training. While using mixed precision training on smaller networks (Policy A) may slow down training iterations (as overhead of doing fp32 to fp16 conversions outweigh the speedup of low precision ops), larger networks (Pol i cy C) show up to a 60% speedup. Generally, our results show that mixed precision may speed up the training process by up to 1.6x without harming convergence.
Quantized Policy for Deployment: To show the benefits of quantization in deploying of reinforce- ment learning policies, we train multiple point-to-poin navigation models (Policy I, II, and III) for aerial robots using Air Learning (Krishnan et al., 2019) and deploy them onto a RasPi-3b, a cost effective, general-purposs embedded processor. RasPi-3b is used as proxy for the compute platform for the aerial robot. Other platforms on aerial robots have similar characteristics. For each of these policies, we report the accuracies and inference speedups attained by the int8 and fp32 policies.
Table shows the accuracies and inference speedups attained for each corresponding quantized policy. We see that quantizing smaller policies (Policy D) yield moderate inference speedups (1.18x for Policy D), while quantizing larger models (Policies II, III) can speed up inference by up to x8x. This speed up in policy III execution times results in speeding-up the generation of the hardware actuation commands from 5 Hz (fp32) to 90 Hz (int8). Note that in this experiment we quantize both weights and activations to 8-bit integers; quantized models exhibit larger loss in accuracy as activations are more difficult to quantize without some form of calibration to determine the range to quantize activation values to (Choi et al. 2018).
A deeper investigation shows that Policies II and III take more memory than the total RAM capacity of the RasPi-3b, causing numerous accesses to swap memory (refer to Appendix) during inference
(which is extremely slow). Quantizing these policies allow them to fit into the RasPi's s RAM, elim- inating accesses to swap and boosting performance by an order of magnitude. Figure 5 shows the memory usage while executing the quantized and unquantized version of Policy III, and shows how without quantization memory usage skyrockets above the total RAM capacity of the board.
FP-32
Pollcy
Polley
Name
Policy
Paramelers
Network
fpo2
(ass)
susson (5)
fp32
(mnx)
Int8
suecess (%)
IndS
up
Memory
1200
1000
Swap Memory
Policy 1
31. MLB 64 Neskos
0.147
60%
0.134
45%
1.18
(M8)
Policy 1I
3L. MLP.P 256 Nades
133.49
74%
9.53
60%6
14
600
RAM
400
Pulicy III
31. MLP (4096, 512, 1024)
208.115
86%
11,036
75%
18.85 x
200
5000
10000
Figure 6: Table lists the inference speed in milliseconds (ms) on Ras-Pi3b+ and success rate (%) for three policies. The figure shows the memory consumption for Policy III's fp-32 and int8 policies.
In context of real-world deployment of an aerial (or any other type of) robot, speedup in policy execution potentially translates faster actuation commands to the aerial robot which in turn implies faster and better responsiveness in a highly dynamic environment (Falanga et al. 2019). Our case study demonstrates how quantization can facilitate the deployment of a accurate policies trained using reinforcement learning onto resource constrained platform.
CONCLUSION
We perform the first study of quantization effects on deep reinforcement learning using QuaRL, a software framework to benchmark and analyze the effects of quantization various reinforcement learning tasks and algorithms. We analyze the performance in terms of rewards for post-training quantization and quantization aware training applied to multiple reinforcement learning tasks and algorithms with the high level goal of reducing policies' resource requirements for efficient training and deployment. We broadly demonstrate that reinforcement learning models may be quantized down to 8/16 bits without loss of performance. Also, we link quantization performance to the dis- tribution of models' weights, demonstrating that some reinforcement learning algorithms and tasks are more difficult to quantize due to their effect of widening the models' weight distribution. Addi- tionally, show that quantization during training acts regularizer which improve exploration. Finally, we apply our results to optimize the training and inference of reinforcement learning mod- els. demonstrating a 50% training speedup for Pong using mixed precision optimization and up to 18x inference speedup on RasPi by quantizing navigation policy. In summary, our findings

---

Under review as a conference paper at ICLR 2020
indicate that there is much potential for the future of quantization of deep reinforcement learning policies.
REFERENCES
Simon Alford, Ryan Robinett, Lauren Milechin, and Jeremy Kepner. Pruned and Structurally Sparse Neural Networks, arXiv e-prints, art. arXiv: 181.00029,, Sep 2018.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Brief Survey of Deep Reinforcement Learning. arXiv e-printy, art. arXiv: 708.05866, Aug 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. 2016.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert A. Legenstein. Deep rewiring: Train- ing very sparse deep networks. International Conference Learning Representations (ICLR), 2017.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning en- vironment: An evaluation platform for general agents. CoRR, abs/1207.4700 2012. URL
http://arxiv wwtp://///////////// .708
C. M. Bishop. Training with noise is equivalent tikhonov regularization. Neural Computation, (1):108-1144 11608-116,100 Jan 1995. doi: 10.1 10.1ne2/e... wwt.ee....99...... 1995.7.1.108.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01140,, 2016. URL TRRLattpp//arxx>> org/
abs/1606. 01540
Guobin Chen, Choi Wongun, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Process- ing Systems, 2017.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pieree I-Jen Chuang. Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, 2018
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. 2018
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI-Yaniv, and Yoshua Bengio. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1. In Advances in Neural Information Processing Systems, Feb 2016.
Davide Falanga, Suseong Kim, and Davide Scaramuzza. How fast is too fast? the role of perception latency in high-speed sense and avoid. IEEE Robotics and Automation Letters, 2019.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in dqn. 2018.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A. Horowitz, and J. Dally. Eie: Efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Sym- posium on Computer Architecture (ISCA), pp. 243-254, June 2016. doi: 10. 1.11109/SCA.. 200.A.2066333
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 15
2015.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally. Eie: efficient inference engine on compressed deep neural network. In 20/6 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243-254. IEEE, 2016.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https: hub. com/hi 1 11-alaaaae waaastablle. basee nes 2018.
10

---

Under review as a conference paper at ICLR 2020
Geoffrey Hinton.
https:/ /ww::///wwwwwww www. www.:wwwww............pp wwoutubee... htwp:uubbb.oo/wwttthhtttt (ttrrkK_Nh29iineeee. [DCCUUaEESLLLLEEFFOOCCLLLLCCEOE2 2012.
Coursera
lecture.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv e-prints, art, ar3iv:1503.02551, Mar 2015.
Kazutoshi Hirose, Ryota Uematsu, Kota Ando, Kodai Ueyoshi, Masayuki Ikehe, Tetsuya Asai, Quantization error-based regularization
Masato Motomura, and Shinya Takamaeda-Yamas for hardware-aware neural network training. Nonlinear Theory and Its Applications, IEICE, 9(4): 453-465, 2018.
doi: 10.1537/nolta..4455
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arihmetti-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojciech Jaškowski. Viz- doom: A doom-based research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in day. CoRR, foss/1807.00413 2018. URL [tttt://p/aattt wrg. abs / 18077 c0412
Srivatsan Krishnan, Behzad Boroujerdian, William Fu, Aleksandra Faust, and Vijay Janapa Reddi. Air learning: An AI research platform for algorithm hardware benchmarking of autonomous aerial robots. CoRR, abss1900.000) 2019. URL. http:://axxi..orrrrer abs/ .90..00221
Jan Kukacka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. 2017.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. International Conference on Learning Representations (ICLR), 2016.
Timothy Lillicrap, Jonathan J. Hunt, Alexand er Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. Interna- tional Conference on Learning Representations (ICLR), Sep 2015.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299: 02007, 2015.
Lin, Chuang Gan, and Song Han. Defensive quantization: When efficiency meets robustness. International Conference on Learning Representations (ICLR), 2019.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. International Conference on Learning Repre- sentations (ICLR), 2018a.
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through 10 regularization. International Conference Learning Representations (ICLR), 2018b.
Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory F. Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR, abs/1710.037400, 2017. URL /ttt:://axxiv.or//aabb.. 03740
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. CoRR, abs/1602.01783, 2016. URL. http::tttt/ttttt org abs 602 00003008233333

---

Under review as a conference paper at ICLR 2020
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning Convolutional Neural Networks for Resource Efficient Inference. International Conference on Learning Repre- sentations (ICLR), Nov 2016.
Jongsoo Park, Sheng R. Li, Wei Wen, Hai Li, Yiran Chen, and Pradeep Dubey. Holistic sparsecnn: Forging the trident of accuracy, speed, and size. International Conference Learning Represen- tations (ICLR), 2016.
Antonio Polino,
Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. International Conference on Learning Representations (ICLR), 2018.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net. ImageNet Classification Using Binary Convolutional Neural Networks. In European Conference Com- puter Vision (ECCV), Mar 2016.
Mengye Ren, Andrei Pokrovsky, Bin Yang, and Raquel Urtasun. SBNet: Sparse Blocks Network for Fast Inference. In Conference on Computer Vision and Pattern Recognition (CVPR, Jan 2018.
Charbel Sakr and Naresh R. Shanbhag. Per-tensor fixed point quantization of the back Pack-propagation algorithm. International Conference on Learning Representations (ICLR), 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
David Silver, Aja Huang, Christopher Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lilli- crap. Madeleine Leach, Koray Kavukcuoglu, Thore Graepel. and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484-503, 2016. URLhttp: html
7/ //www. nature com http:///...aaaee.....a//////////
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):35,, 2017.
TensorFlow. Quantize create training graph.
wtt.rreeteeeetaannnnnneeeaaaaeee graph 2018a.
TensorFlow. Quantization-aware training.
Lensor /tte00///////ee///333//////////1
2018b.
Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Uperoft, and Peter Corke. Towards Vision-
Based Deep Reinforcement Learning for Robotic Motion Control. In Australasian Conference on Robotics and Automation (ACRA), Nov 2015.
Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, abs/1606.06164 2016. /eLlittp://arxex
orq/ abs/ 116666666600 C6160
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 16Xiv: 2016.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
Here, we list several details that are committed from the first 8 pages due to the limited page count. To the best of our ability, provide sufficient details reproduce results and address common clarification questions.
A
POST TRAINING QUANTIZATION RESULTS
Here we tabulate the post training quantization results listed in Table 2 into four separate tables for clarity. Each table corresponds to post training quantization results for specific algorithm. Table S tabulates the post training quantization for A2C algorithm. Likewise, Table [ tabulates the post training quantization results for DQN. Table and Table lists the post training quantization results for PPO and DDPG algorithms respectively.
Environmentt
fp32
Ip16
fpló
int8
E int8
Breakout
379
371
2.11%
350
7,65%
Spacelmaders
717
667
6.97%
634
11.58%
BeamRider
3087
3060
0.87%
2793
9.52%
MsPacman
1915
1915
0.00%
2045
-6, -6.79%
Qbert
5002
5002
0.00%
-12.
Senguest
782
756
3,32%
753
3.71%
CariPole
500
500
0.00%
500
0.00%
Pong
20
20
0.00%
5.00%
Mean
1,66 %
2.31 %
Table 5: A2C rewards for fp32, fp16, and int8 policies.
Environment
fp32
fpl6
E fp16
int8
E int8
Breakout
214
217
-1.40%
63.55%
Spacelavaders
586
625
-6.66%
509
13.14%
BeamRider
925
823
11.03%
721
22.05%
MsPacman
1433
1429
0.28%
2024
-41.24%
Qbert
641
641
0.004
616
3.90%
Seaguest
1709
1885
-10.30%
1582
7.43%
CariPole
500
500
0.00%
S00
0.00%
Pong
Mean
+0,88%
8.60
Table 6: DQN rewards for fp32. fp16, and int8 policies.
Environment
fp32
fp16
E.fp16
int8
E. int88
Breakow
110015:
8.00%
Spacelmvdders
698
662
5.16%
684
2.01%
MsPacman
1735
1735
0.00%
1845
-6.34%
Qbert
15010
15010
14425
Seaguest
1782
1784
-0.11%
1795
-0.73%
CariPole
0.00%
500
0.00%
Pong
20
20
0.00%
20
0.00%
Mean
8.6%
0.54%
Table 7: PPO rewards for fp32, fp16, and int8 policies.
B DQN HYPERPARAMETER FOR ATARI
For all Atari games in the results section we use standard Layer Conv (128) 128 FC, Hyperpa- rameters are listed in Table
We use stable-baselines (Hill al. 2018) for all the reinforcement learning experiments. We use Tensorflow version 1.14 as the machine learning backend.
C MIXED PRECISION HYPERPARAMETERS
In mixed precision training, we used three policies namely Policy A, Policy B and Policy C respec- tively. The policy architecture for these policies tabulated in To] For measuring the runtimes for fp32 adn fp16 training, we use the time Linux command for each run and add the usr and sys times to measure the runtimes for both mixxed-rrecision training and fp32 training. The hyperparameters used for training DQN-Pong agent listed in Table
13

---

Under review as a conference paper at ICLR 2020
Environment
fp32
fp16
E_fp16
Int8
E,int8
Walker20
1890
1929
.2,06%
1866
1.27%
HalfCheetah
2551
BipedalWalkerr
98
90
8.16%
83
15. 15.31%
MountainCarC Conootiinouus
92
92
0.00%
92
0.00%
1.54%
4.93%
Table 8: DDPG rewards for fp32, fp16, and int8 policies.
Hyperparameter
Value
n 1_timesteps
Million Steps
buffer-size
10000
learning-rate warm up quant delay target.network wtakkkuudaaeeeeeae frequeney
0.0001
10000
500000
1000
exploration stttiiiooniiin.... final.eps eps exploration fraction
0.01
prioritized replay alpha
prioritized replay
True
Table 9: Hyper parameters used for mixed precision training for training DQN algorithm in all the Atari arcade learning environments.
D QUANTIZED POLICY DEPLOYMENT
Here describe the methodology used to train point to point navigation policy in Air Learning and deploy it on an embedded compute platform such as Ras-Pi 3b+. Air Learning is an Al research platform that provides infrastructure components and tools to train a fully functional reinforcement learning policies for aerial robots. In simple environments like OpenAl gym, Atari the training and inference happens in the same environment without any randomization. In contrast to these environments, Air Learning allows to randomize various environmental parameters such such as arena size, number of obstacles, goal position etc.
In this study, fix the arena size to 25 at anytime would be anywhere between one to five and is chosen randmonly on episode to episode basis. The position of these obstacles and end point (goal) are also changed every episode. We train the aerial robot to reach the end point using DQN algorithm. The input the policy is sensor mounted on the drone along with IMU measurements. The output of the policy is one among the 25 actions with different velocity and yaw rates. The reward function we use in this study is defined based on the following equation:
25 20 The maximum number of obstacles
1000 a 100 D -
Here, is binary variable whose value is *1' if the agent reaches the goal else its value is '0', 3 is a binary variable which is set to if the aerial robot collides with any obstacle or runs out of the maximum allocated steps for episode Otherwise, is '0', effectively penalizing the agent for hitting an obstacle or not reaching the end point in time. D, is the distance to the end point from the agent's s current location, motivating the agent to move closer to the goal. is the distance correction which is applied penalize the agent if chooses actions which speed up the agent away from the goal. The distance correction term defined as follows:
De (Vmax Vnow) tmax
(2)
V Vmaa is the maximum velocity possible for the agent which for DQN is fixed at 2.5 m/s. V Vnow is the current velocity of the agent and Imaz is the duration of the actuation.
We train three policies namely Policy I, Policy II, and Policy III. Each policy is learned through cur- riculum learning where we make the end goal farther away as the training progresses. We terminate the training once the agent has finished Million steps. We evaluate the all the three policies fp32 and quantized int8 data types for 100 evaluations in airlearning and report the success rate.
We the maximum allowed steps in an episode as 750. This make the agent finds the end-point (goal) within some finite amount steps.
14

---

Under review as a conference paper at ICLR 2020
Algorithm
Policy Architecture
Policy
Layer Conv (128 Filters) + FC (128)
Policy B
3 Layer Conv (512 Filters) FC(512)
Policy
3 Layer Conv (1024 Filters) + FC (2048)
Table 10: The policy architecture that was used in mixed precision training for training DQN algo- rithm in Atari Pong environment.
We also take these policies and characterize the system performance on Ras-pi 3b platform. Ras-Pi 3b is proxy for the compute platform available on the aerial robot. The hardware specification for Ras-Pu Pi is shown in Table
Embedded System
Ras-Pi 3b
CPU Cores
Cores (ARM A53)
CPU Frequency
1.2 GHz
GPU
None <IW
Power
Cost
S35
Table 11: Specification of Ras-Pi 3b embedded computing platform. Ras-Pi 3b proxy for the on-board compute platform available in the aerial robot.
We allocate region of storage space swap memory. It is the region of memory allocated in disk that is used when system memory utilized fully by a process. Ras-Pi 3b, the swap memory is allocated in Flash storage.
E
POST-TRAN TRAINING QUANTIZATION SWEET SPOT
Figures 7 shows that there sweet spot for fewer bits outperforms higher precision quantization. Each plot was generated by applying post-
quantization. Sometimes, quantizing to
training quantization to the full precision baselines and evaluating over 10 runs.
2500
Aversgge Reward
Average Reward
2000
Average Reward
1500
1500
150
1000
500
50
500
10
15
20
25
30
5
10
15
20
25
30
5
10
15
20 25 30
Post-Train Quantize Bits
Post-Train Quantize A Bits
Post-Train Quantize Bits
Figure 7: Post training quantization sweet spot for DQN MsPacman, DQN SeaQuest, DQN Break- out. We see that post-training quantization sweet spot depends on the specific task at hand. Note that 16-bit in this plot is 16-bit affine quantization, not fp16.
15


Reviews:
[{"review_id": "SJxbS6petH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the impact of using a reduced precision (i.e., quantization) in different deep reinforcement learning (DRL) algorithms. It shows that overall, reducing the precision of the neural network in DRL algorithms from 32 bits to 16 or 8 bits doesn't have much effect on the quality of the learned policy. It also shows how this quantization leads to a reduced memory cost and faster training and inference times.\n\nI don't think this paper contributes with many novel results in the field, with most results being known or expected. The result that is interesting, in my opinion, is not properly explored.  The paper is well-written but it is a bit repetitive. It seems to me that the first 3 pages could be compressed in 1, as the same information is introduced over and over again. \n\nWith respect to the results being known, quantization is known to succeed in supervised learning tasks. In a deep reinforcement learning algorithm, when you apply post-training quantization in a deep reinforcement learning algorithm, mainly when that algorithm uses a value function (e.g., A2C or DQN), the problem is reduced to a regression problem. It is no different than a supervised learning problem. One has the original network’s prediction and they need to match that prediction. The complexities introduced in the reinforcement learning problem (bootstrapping, exploration, stability) don’t exist anymore as they arise during training. Thus, it doesn’t seem to me that these results are novel or surprising. In a sense it is neat to see that eventual errors do not compound, but that’s it. If I were to write this paper I would make this set of experiments much shorter just as a sanity check. One thing that I feel is missing is a notion of the impact of the quantization not in the rewards accumulated but in the policy/value function. How often does the quantized agent take a different action than the original agent, for example? Does it happen often but only when it doesn’t matter, or is it rare?\n\nThe quantization during training is potentially interesting. It was not properly explored though. I wonder if the quantization during training has a regularization effect, which is known to improve agent’s performance in reinforcement learning (e.g., Cobbe et al., 2018, Farebrother et al., 2018). Does the agent generalize better when using a network with fewer bits of precision? How does this change impact training? These are all questions that could potentially make the results in this paper novel (i.e., quantization as a form of regularization), but as it is now, the results are not that surprising.\n\nImportantly, there are important details missing in the paper that make it hard for me to evaluate the validity of the results presented. Are the results reported over multiple runs? What is the version of the Atari games used, is it the one with stochasticity? How much variance do we have if we replicate this process over different networks that perform well? These are questions I would like to see answered because they also inform us about the impact of the proposed idea. For example, if by repeating this experiment multiple times one observe a high variance, it might mean that different models might be impacted in different ways.\n\nThe results in the “real-world” (Pong is not real-world) are not that surprising as well. Basically they show that if one uses a network with lower precision training and inference are faster, which, again, is not surprising. \n\nThere’s also an important distinction in the results that is not discussed in the paper: DQN estimates a value function while methods such as PPO directly estimate a policy. The reason DQN might have a wider distribution is exactly because it is estimating a different objective. These are important details that should be acknowledged and discussed in the paper. In my opinion, for this paper be relevant, it should have a very thorough evaluation of these different dimensions of reinforcement learning algorithms, with explicit discussions about it. Variance, the impact of quantization during learning, the distinction between parametrizing policies versus value functions, etc.\n\nFinally, there are some aspects of the presentation of this paper that could also be improved. Aside from typos, below are some other comments on the presentation.\n- There’s no such thing as Atari environment, it is either Arcade Learning Environment (Bellemare et al., 2013) or Atari games.\n- I’d introduce/explain quantization in the beginning of the second paragraph of the Introduction for those not familiar with the term.\n- No references are provided for the environments used. You should refer to Bellemare et al.’s (2013) work as well as Brockman et al.’s (2016).\n- Is it really necessary to explain Fp16 quantization as it is done now, with even a picture of two bytes? I’d expect most readers are familiar with how numbers are represented in a computer.\n- The equation for Uniform Affine Quantization is pretty much the same as the one in the Section Quantization Aware Training. All these “repetitions”, or discussions that are common-knowledge give the impression that the paper is trying to fill all the pages without necessarily having enough content.\n- The references are not standardized (e.g., sometimes names are shortened, sometimes they are not) and the paper “Efficient inference engine on compressed deep neural network” is cited twice.\n\n\nReferences:\n\nMarc G. Bellemare, Yavar Naddaf, Joel Veness, Michael Bowling: The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res. 47: 253-279 (2013)\n\nGreg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba: OpenAI Gym. CoRR abs/1606.01540 (2016)\n\n\nKarl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, John Schulman: Quantifying Generalization in Reinforcement Learning. CoRR abs/1812.02341 (2018)\n\nJesse Farebrother, Marlos C. Machado, Michael Bowling: Generalization and Regularization in DQN. CoRR abs/1810.00123 (2018)\n\n\n------\n\n\n>>> Update after rebuttal: I stand by my score after the rebuttal. \n\nThe rebuttal did acknowledge some points I made to me the paper took a gradient update towards the right direction. I don't think the paper is quite there yet though. It is repetitive, spending too much time with basic concepts, and it still ignores small details that matter (e.g., calling it Atari Arcade Learning). I strongly recommend the authors to follow my recommendations closely and then submit the paper again to a next conference. The discussion about generalization is potentially interesting, going beyond the regularization for exploration aspect. A better discussion about quantization during learning is also essential. The first three pages could probably be compressed by half.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJeTTeOscB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effect of quantization on training reinforcement learning tasks. Specifically, the paper applies post-training quantization and quantization aware learning to various tasks and record the effects on accuracy and training speed.\n\nOverall, the empirical evaluations suggest that quantization does not significantly hurt the performance of RL training among a wide range of tasks. On several tasks, the authors showed that quantization can significantly reduce memory usage and speed up the inference time. On the other hand, the improved efficiency comes at the cost of accuracy or lower rewards (2% - 5% error as shown in section 4) and (> 5% in terms of success rate as shown in Figure 5).\n\nWhile it is expected that quantization should decrease the accuracy of the trained model, it is not entirely clear how one should evaluate the trade-off presented in the work. Some natural questions that I believe deserve more discussions are:\n-- Are the kinds of accuracy cost the best one could hope for using these methods?\n--  Is there still room for improvement in terms of reducing the cost of accuracy?\n\nDetailed comments:\n-- In the definition of Q_n(W): isn't $\\delta$ equal to |W| / 2^n?\n-- In Figure 5: your results show that the \"int8\" method has a significantly lower success rate than \"fp32\". Could you provide some discussion as to why this is the case?\n-- Typos: Page 4, \"is a applied\"; Page 5, \"full connected weights\"; Page 8, \"of a accurate\".", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "B1ls5I2aYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Training and deployment of DRL models is expensive. Quantization has proven useful in supervised learning, however it is yet to be tested thoroughly in DRL. This paper investigates whether quantization can be applied in DRL towards better resource usage (compute, energy) without harming the model quality. Both quantization-aware training (via fake quantization) and post-training quantization is investigated. The work demonstrates that policies can be reduced to 6-8 bits without quality loss. The paper indicates that quantization can indeed lower resource consumption without quality decline in realistic DRL tasks and for various algorithms.\n\nThe researchers propose a benchmark called QUARL that allows them to evaluate the effectiveness of quantization as well as the impact of quantization across a set of established DRL algorithms (e.g., DQN, DDPG, PPO) and environments (e.g., OpenAI Gym, ALE). Quantizations tested: fp32 -> fp16, int8, uniform affine.\n\nThe idea is simple and carries over from (image-based) supervised learning. The experiments are exhaustive and have to the best of my knowledge not yet been conducted. The conclusions indicate the advantage of quantization, however it is unclear how these results would generalize to real environments (the environments used are after all still simple benchmarks, e.g., half-cheetah or pong). The results are also not entirely surprising or impactful: how is quantization impacting reinforcement learning in a different way than supervised learning? E.g., DQN is supervised learning of a Q-value function against a target. What secondary effects does quantization have on the learning procedure: e.g., does it boost exploration behavior or does it regularize training? We also know that some of these tasks can be solved by extremely small models (https://arxiv.org/abs/1806.01363), while the models used in this work are significantly larger: is quantization working simply because the network capacity is large enough to allow it? These could be investigated in more detail. Furthermore, I'm also missing some experimental setup details: e.g., how many seeds were used for all of the experiments (which is known to greatly affect the results on the benchmarks used in this paper)?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates quantization for speeding up RL. While the reviewers agree that the idea is a good one (it should definitely help), they also have a number of concerns about the paper and presentation. In particular, the reviewers feel that the authors should have provided more insight into the challenges of quantization in RL and the tradeoffs involved. After having read the rebuttals, the reviewers believe that the authors are on the right track, but that the paper is still not ready for publication. If the authors take the reviewer comments and concerns seriously and update their paper accordingly, the reviewers believe that this could eventually result in a strong paper.
Decision: Reject