Quantized Reinforcement Learning (QuaRL)

Recent work has shown that quantization can help reduce the memory , compute , and energy demands of deep neural networks without significantly harming their quality . However , whether these prior techniques , applied traditionally to imagebased models , work with the same efficacy to the sequential decision making process in reinforcement learning remains an unanswered question . To address this void , we conduct the first comprehensive empirical study that quantifies the effects of quantization on various deep reinforcement learning policies with the intent to reduce their computational resource demands . We apply techniques such as post-training quantization and quantization aware training to a spectrum of reinforcement learning tasks ( such as Pong , Breakout , BeamRider and more ) and training algorithms ( such as PPO , A2C , DDPG , and DQN ) . Across this spectrum of tasks and learning algorithms , we show that policies can be quantized to 6-8 bits of precision without loss of accuracy . We also show that certain tasks and reinforcement learning algorithms yield policies that are more difficult to quantize due to their effect of widening the models ’ distribution of weights and that quantization aware training consistently improves results over post-training quantization and oftentimes even over the full precision baseline . Additionally , we show that quantization aware training , like traditional regularizers , regularize models by increasing exploration during the training process . Finally , we demonstrate usefulness of quantization for reinforcement learning . We use half-precision training to train a Pong model 50 % faster , and we deploy a quantized reinforcement learning based navigation policy to an embedded system , achieving an 18× speedup and a 4× reduction in memory usage over an unquantized policy . 1 INTRODUCTION . Deep reinforcement learning has promise in many applications , ranging from game playing ( Silver et al. , 2016 ; 2017 ; Kempka et al. , 2016 ) to robotics ( Lillicrap et al. , 2015 ; Zhang et al. , 2015 ) to locomotion and transportation ( Arulkumaran et al. , 2017 ; Kendall et al. , 2018 ) . However , the training and deployment of reinforcement learning models remain challenging . Training is expensive because of their computationally expensive demands for repeatedly performing the forward and backward propagation in neural network training . Deploying deep reinforcement learning ( DRL ) models is prohibitively expensive , if not even impossible , due to the resource constraints on embedded computing systems typically used for applications , such as robotics and drone navigation . Quantization can be helpful in substantially reducing the memory , compute , and energy usage of deep learning models without significantly harming their quality ( Han et al. , 2015 ; Zhou et al. , 2016 ; Han et al. , 2016 ) . However , it is unknown whether the same techniques carry over to reinforcement learning . Unlike models in supervised learning , the quality of a reinforcement learning policy depends on how effective it is in sequential decision making . Specifically , an agent ’ s current input and decision heavily affect its future state and future actions ; it is unclear how quantization affects the long-term decision making capability of reinforcement learning policies . Also , there are many different algorithms to train a reinforcement learning policy . Algorithms like actor-critic methods ( A2C ) , deep-q networks ( DQN ) , proximal policy optimization ( PPO ) and deep deterministic policy gradients ( DDPG ) are significantly different in their optimization goals and implementation details , and it is unclear whether quantization would be similarly effective across these algorithms . Finally , reinforcement learning policies are trained and applied to a wide range of environments , and it is unclear how quantization affects performance in tasks of differing complexity . Here , we aim to understand quantization effects on deep reinforcement learning policies . We comprehensively benchmark the effects of quantization on policies trained by various reinforcement learning algorithms on different tasks , conducting in excess of 350 experiments to present representative and conclusive analysis . We perform experiments over 3 major axes : ( 1 ) environments ( Atari Arcade , PyBullet , OpenAI Gym ) , ( 2 ) reinforcement learning training algorithms ( Deep-Q Networks , Advantage Actor-Critic , Deep Deterministic Policy Gradients , Proximal Policy Optimization ) and ( 3 ) quantization methods ( post-training quantization , quantization aware training ) . We show that quantization induces a regularization effect by increasing exploration during training . This motivates the use of quantization aware training , which we show demonstrates improved performance over post-training quantization and oftentimes even over the full precision baseline . Additionally , We show that deep reinforcement learning models can be quantized to 6-8 bits of precision without loss in quality . Furthermore , we analyze how each axis affects the final performance of the quantized model to develop insights into how to achieve better model quantization . Our results show that some tasks and training algorithms yield models that are more difficult to apply post-training quantization as they widen the spread of the models ’ weight distribution , yielding higher quantization error . To demonstrate the usefulness of quantization for deep reinforcement learning , we 1 ) use half precision ops to train a Pong model 50 % faster than full precision training and 2 ) deploy a quantized reinforcement learning based navigation policy onto an embedded system and achieve an 18× speedup and a 4× reduction in memory usage over an unquantized policy . 2 RELATED WORK . Reducing neural network resource requirements is an active research topic . Techniques include quantization ( Han et al. , 2015 ; 2016 ; Zhu et al. , 2016 ; Jacob et al. , 2018 ; Lin et al. , 2019 ; Polino et al. , 2018 ; Sakr & Shanbhag , 2018 ) , deep compression ( Han et al. , 2016 ) , knowledge distillation ( Hinton et al. , 2015 ; Chen et al. , 2017 ) , sparsification ( Han et al. , 2016 ; Alford et al. , 2018 ; Park et al. , 2016 ; Louizos et al. , 2018b ; Bellec et al. , 2017 ) and pruning ( Alford et al. , 2018 ; Molchanov et al. , 2016 ; Li et al. , 2016 ) . These methods are employed because they compress to reduce storage and memory requirements as well as enable fast and efficient inference and training with specialized operations . We provide background for these motivations and describe the specific techniques that fall under these categories and motivate why quantization for reinforcement learning needs study . Compression for Memory and Storage : Techniques such as quantization , pruning , sparsification , and distillation reduce the amount of storage and memory required by deep neural networks . These techniques are motivated by the need to train and deploy neural networks on memoryconstrained environments ( e.g. , IoT or mobile ) . Broadly , quantization reduces the precision of network weights ( Han et al. , 2015 ; 2016 ; Zhu et al. , 2016 ) , pruning removes various layers and filters of a network ( Alford et al. , 2018 ; Molchanov et al. , 2016 ) , sparsification zeros out selective network values ( Molchanov et al. , 2016 ; Alford et al. , 2018 ) and distillation compresses an ensemble of networks into one ( Hinton et al. , 2015 ; Chen et al. , 2017 ) . Various algorithms combining these core techniques have been proposed . For example , Deep Compression ( Han et al. , 2015 ) demonstrated that a combination of weight-sharing , pruning , and quantization might reduce storage requirements by 35-49x . Importantly , these methods achieve high compression rates at small losses in accuracy by exploiting the redundancy that is inherent within the neural networks . Fast and Efficient Inference/Training : Methods like quantization , pruning , and sparsification may also be employed to improve the runtime of network inference and training as well as their energy consumption . Quantization reduces the precision of network weights and allows more efficient quantized operations to be used during training and deployment , for example , a ” binary ” GEMM ( general matrix multiply ) operation ( Rastegari et al. , 2016 ; Courbariaux et al. , 2016 ) . Pruning speeds up neural networks by removing layers or filters to reduce the overall amount of computation necessary to make predictions ( Molchanov et al. , 2016 ) . Finally , Sparsification zeros out network weights and enables faster computation via specialized primitives like block-sparse matrix multiply ( Ren et al. , 2018 ) . These techniques not only speed up neural networks but decrease energy consumption by requiring fewer floating-point operations . Quantization for Reinforcement Learning : Prior work in quantization focuses mostly on quantizing image / supervised models . However , there are several key differences between these models and reinforcement learning policies : an agent ’ s current input and decision affects its future state and actions , there are many complex algorithms ( e.g : DQN , PPO , A2C , DDPG ) for training , and there are many diverse tasks . To the best of our knowledge , this is the first work to apply and analyze the performance of quantization across a broad of reinforcement learning tasks and training algorithms . 3 QUANTIZED REINFORCEMENT LEARNING ( QUARL ) . We develop QuaRL , an open-source software framework that allows us to systematically apply traditional quantization methods to a broad spectrum of deep reinforcement learning models . We use the QuaRL framework to 1 ) evaluate how effective quantization is at compressing reinforcement learning policies , 2 ) analyze how quantization affects/is affected by the various environments and training algorithms in reinforcement learning and 3 ) establish a standard on the performance of quantization techniques across various training algorithms and environments . Environments : We evaluate quantized models on three different types of environments : OpenAI gym ( Brockman et al. , 2016 ) , Atari Arcade Learning ( Bellemare et al. , 2012 ) , and PyBullet ( which is an open-source implementation of the MuJoCo ) . These environments consist of a variety of tasks , including CartPole , MountainCar , LunarLandar , Atari Games , Humanoid , etc . The complete list of environments used in the QuaRL framework is listed in Table 1 . Evaluations across this spectrum of different tasks provide a robust benchmark on the performance of quantization applied to different reinforcement learning tasks . Training Algorithms : We study quantization on four popular reinforcement learning algorithms , namely Advantage Actor-Critic ( A2C ) ( Mnih et al. , 2016 ) , Deep Q-Network ( DQN ) ( Mnih et al. , 2013 ) , Deep Deterministic Policy Gradients ( DDPG ) ( Lillicrap et al. , 2015 ) and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) . Evaluating these standard reinforcement learning algorithms that are well established in the community allows us to explore whether quantization is similarly effective across different reinforcement learning algorithms . Quantization Methods : We apply standard quantization techniques to deep reinforcement learning models . Our main approaches are post-training quantization and quantization aware training . We apply these methods to models trained in different environments by different reinforcement learning algorithms to broadly understand their performance . We describe how these methods are applied in the context of reinforcement learning below . 3.1 POST-TRAINING QUANTIZATION . Post-training quantization takes a trained full precision model ( 32-bit floating point ) and quantizes its weights to lower precision values . We quantize weights down to fp16 ( 16-bit floating point ) and int8 ( 8-bit integer ) values . fp16 quantization is based on IEEE-754 floating point rounding and int8 quantization uses uniform affine quantization . Fp16 Quantization : Fp16 quantization involves taking full precision ( 32-bit ) values and mapping them to the nearest representable 16-bit float . The IEEE-754 standard specifies 16-bit floats with the format shown below . Bits are grouped to specify the value of the sign ( S ) , fraction ( F ) and exponent ( E ) which are then combined with the following formula to yield the effective value of the float : Sign Exponent ( 5 bits ) Fraction ( 10 bits ) Vfp16 = ( −1 ) S × ( 1 + F 210 ) × 2E−15 In subsequent sections , we refer to float16 quantization using the following notation : Qfp16 ( W ) = roundfp16 ( W ) Uniform Affine Quantization : Uniform affine quantization ( TensorFlow , 2018b ) is applied to a full precision weight matrix and is performed by 1 ) calculating the minimum and maximum values of the matrix and 2 ) dividing this range equally into 2n representable values ( where n is the number of bits being quantized to ) . As each representable value is equally spaced across this range , the quantized value can be represented by an integer . More specifically , quantization from full precision to n-bit integers is given by : Qn ( W ) = ⌊ W δ ⌋ + z where δ = |min ( W , 0 ) |+ |max ( W , 0 ) | 2n , z = ⌊ −min ( W , 0 ) δ ⌋ Note that δ is the gap between representable numbers and z is an offset so that 0 is exactly representable . Further note that we usemin ( W , 0 ) andmax ( W , 0 ) to ensure that 0 is always represented . To dequantize we perform : D ( Wq , δ , z ) = δ ( Wq − z ) In the context of QuaRL , int8 and fp16 quantization are applied after training a full precision model on an environment , as per Algorithm 1 . In post training quantization , uniform quantization is applied to each fully connected layer of the model ( per-tensor quantization ) and is applied to each channel of convolution weights ( per-axis quantization ) ; activations are not quantized . We use post-training quantization to quantize to fp16 and int8 values . Algorithm 1 : Post-Training Quantization for Reinforcement Learning Input : T : task or environment Input : L : reinforcement learning algorithm Input : A : model architecture Input : n : quantize bits ( 8 or 16 ) Output : Reward 1 M = Train ( T , L , A ) 2 Q = { Qint8 n = 8 Qfp16 n = 16 3 return Eval ( Q ( M ) ) Algorithm 2 : Quantization Aware Training for Reinforcement Learning Output : Reward Input : T : task or environment Input : L : reinforcement learning algorithm Input : n : quantize bits Input : A : model architecture Input : Qd : quantization delay 1 Aq = InsertAfterWeightsAndActivations ( Qtrainn ) 2 M , TensorMinMaxes = TrainNoQuantMonitorWeightsActivationsRanges ( T , L , Aq , Qd ) 3 M = TrainWithQuantization ( T , L , M , TensorMinMaxes , Qtrainn ) 4 return Eval ( M , Qtrainn , TensorMinMaxes )

Summary: This paper investigates the impact of using a reduced precision (i.e., quantization) in different deep reinforcement learning (DRL) algorithms. It shows that overall, reducing the precision of the neural network in DRL algorithms from 32 bits to 16 or 8 bits doesn't have much effect on the quality of the learned policy. It also shows how this quantization leads to a reduced memory cost and faster training and inference times.