Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance

1 INTRODUCTION . Off-policy Deep Reinforcement Learning algorithms aim to improve sample efficiency by reusing past experience . A number of off-policy Deep RL algorithms have been proposed for control tasks with continuous state and action spaces , including Deep Deterministic Policy Gradient ( DDPG ) , Twin Delayed DDPG ( TD3 ) and Soft Actor Critic ( SAC ) ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ; b ) . TD3 introduced clipped double-Q learning , and was shown to be significantly more sample efficient than popular on-policy methods for a wide range of MuJoCo benchmarks . Soft Actor Critic ( SAC ) has similar off-policy structures with clipped double-Q learning , but it also employs maximum entropy reinforcement learning . SAC was shown to provide excellent sample efficiency and asymptotic performance in a wide-range of MuJoCo environments , including the high-dimensional Humanoid environment for which both DDPG and TD3 perform poorly . More recently , Kuznetsov et al . ( 2020 ) proposed Truncated Quantile Critics ( TQC ) , a model-free algorithm which includes distributional representations of critics , truncation of critics prediction , and ensembling of multiple critics . Instead of the usual modeling of the Q-function of the expectation of return , TQC approximates the distribution of the return random variable conditioned on the state and action . By dropping several of the top-most “ atoms ” and varying the number of dropped atoms of the return distribution approximation , TQC can control the over-estimation bias . TQC ’ s asymptotic performance ( that is after a long period of training ) was shown to be better than that of SAC on the continuous control MuJoCo benchmark suite , including a 25 % improvement on the most challenging Humanoid environment . However , TQC is not sample efficient in that it generally requires a large number of samples to reach even moderate performance levels . Chen et al . ( 2021 ) proposed Randomized Ensembled Double Q-learning ( REDQ ) , a model-free algorithm which includes a high Update-To-Data ( UTD ) ratio , an ensemble of Q functions , and in-target minimization across a random subset of Q functions from the ensemble . Using a UTD ratio much larger than one , meaning that several gradient steps are taken for each environment interaction , improves sample efficiency , while the ensemble and in-target minimization allows the algorithm to maintain stable and near-uniform bias under the high UTD ratio . The algorithm was shown to attain much better performance than SAC at the early stage of training , and to match or improve the sample-efficiency of the state-of-the-art model-based algorithms for the MuJoCo benchmarks . How- ever , although REDQ is highly sample efficient for early-stage training , its asymptotic performance is significantly below that of TQC . Is it possible to design a simple , streamlined model-free algorithm which can achieve REDQ ’ s high sample efficiency in early-stage training and also achieve TQC ’ s high asymptotic performance in late state training ? In this paper , we achieve this goal with a new model-free algorithm , Aggressive QLearning with Ensembles ( AQE ) . Like TQC and REDQ , AQE uses an ensemble of Q-functions , and like REDQ it uses a UTD ratio > 1 . However AQE is very simple , requiring neither distributional representation of critics as in TQC nor target randomization and double-Q learning as in REDQ . AQE controls over estimation bias and the standard deviation of the bias by varying the number of ensemble members N and the number of ensembles K ≤ N that are kept when calculating the targets . Through carefully designed experiments , we provide a detailed analysis of AQE . We first show that for the MuJoCo benchmark , AQE provides state-of-the-art performance , surpassing the performance of SAC , REDQ , and TQC at all stages of training . We perform an ablation study , and show that AQE is robust to choices of hyperparameters : AQE can work well with small ensembles consisting of 10-20 ensemble members , and performance does not vary significantly with small changes in the keep parameter K. We show that that AQE performs better than several variations , including using the median of all ensemble members and removing the most extreme minimum and maximum outlier in the targets . In order to improve computational time , we also consider different multihead architectures for the ensemble of critics : consistent with the supervised convolutional network literature , we find that a two-head architecture not only reduces computational time but can actually improve performance for some environments . Additionally , we show that AQE continues to outperform SAC and TQC even when these algorithms are made aggressive with a UTD 1 . To ensure our comparisons are fair , and to ensure our results are reproducible ( Henderson et al. , 2018 ; Islam et al. , 2017 ; Duan et al. , 2016 ) , we provide open source code1 . For all algorithmic comparisons , we use the the authors ’ code . 2 ADDITIONAL RELATED WORK . Overestimation bias due to in target maximization in Q-learning can significantly slow learning ( Thrun & Schwartz , 1993 ) . For tabular Q-learning , van Hasselt ( 2010 ) introduced Double QLearning , and showed that it removes the overestimation basis and in general leads to an underestimation bias . Hasselt et al . ( 2016 ) showed that adding Double Q-learning to deep-Q networks can have a similar effect , leading to a major performance boost for the Atari games benchmark . As stated in the Introduction , for continuous-action spaces , TD3 and SAC address the overestimation bias using clipped-double Q-learning , which brings significant performance improvements ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ; b ) . As mentioned in the Introduction , Kuznetsov et al . ( 2020 ) control the over-estimation bias by estimating the distribution of the return random variable , and then by dropping several of the top-most “ atoms ” from the estimated distribution . The distribution estimate is based on a methodology developed in Bellemare et al . ( 2017 ) ; Dabney et al . ( 2018a ; b ) , which employs an asymmetric Huber loss function to minimize the Wasserstein distance between the neural network output distribution and the target distribution . In this paper , in order to counter over-estimation bias , we also drop the top-most estimates , although we do so solely with an ensemble of Q-function mean estimators rather than with an ensemble of the more complex distributional models employed in ( Kuznetsov et al. , 2020 ) . It is well-known that using ensembles can improve the performance of DRL algorithms ( Faußer & Schwenker , 2015 ; Osband et al. , 2016 ; Lee et al. , 2021 ) . For Q-learning based methods , Anschel et al . ( 2017 ) use the average of multiple Q estimates to reduce variance . Lan et al . ( 2020 ) introduced Maxmin Q-learning , which uses the minimum of all the Q-estimates rather than the average . Agarwal et al . ( 2020 ) use Random Ensemble Mixture ( REM ) , which employs a random convex combination of multiple Q estimates . 1https : //anonymous.4open.science/r/Aggressive-Q-Learning-with-Ensembles-0FC0/ Model-based methods often attain high sample efficiency by using a high UTD ratio . In particular , Model-Based Policy Optimization ( MBPO ) ( Janner et al. , 2019 ) uses a large UTD ratio of 20-40 . Compared to Soft-Actor-Critic ( SAC ) , which is model-free and uses a UTD of 1 , MBPO achieves much higher sample efficiency in the OpenAI MuJoCo benchmark ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) . REDQ ( Chen et al. , 2021 ) , a model-free algorithm , also successfully employs a high UTD ratio to achieve high sample efficiency . 3 ALGORITHM . We propose Aggressive Q-learning with Ensembles ( AQE ) , a simple model-free algorithm which provides state-of-the-art performance for the MuJoCo benchmark for both early and late stage of training . The pseudo-code is shown in Algorithm 1 . As is the case with most off-policy continuouscontrol algorithms , AQE has a single actor ( policy network ) and multiple critics ( Q-function networks ) , and employs Polyak averaging of the target parameters to enhance stability . Building on this algorithmic base , it also employs an update-to-data ratio G > 1 , an ensemble of N ≥ 3 Q-functions ( rather than just two as in TD3 and SAC ) , and targets that average all the Q-functions excluding the Q-functions with the highest N − K values . For exploration , it uses entropy maximization as in SAC , although it could easily incorporate alternative exploration schemes . The design of AQE borrows components from TQC and REDQ , but also excludes components from those algorithms . Like TQC , AQE does not employ clipped double Q-learning to control over estimation bias , but instead drops the highest Q values when calculating the targets . Like REDQ , AQE employs a UTD > 1 to improve sample efficiency during training . However , unlike TQC , AQE uses a UTD > 1 , and does not employ distributional representations of critics but instead the simpler Q-function expectation estimates . Unlike REDQ which employs two randomly chosen ensemble members when calculating the target , AQE does not use target randomization and uses most of the ensemble members when calculating the target . As discussed in the theory Section 5 , this provides AQE more flexibility through the choice of the N . The resulting algorithm is not only simple and streamlined , but also provides state-of-the art performance . AQE has three key hyperparameters , G , N , and K. If we set N = 2 , K = 1 and G = 1 , AQE is simply the underlying off-policy algorithm such as SAC . When N > 2 , K = 1 and G = 1 , then AQE becomes similar to , but not equivalent to , Maxmin Q-learning ( Lan et al. , 2020 ) . 3.1 MULTI-HEAD ENSEMBLE ARCHITECTURE . AQE uses an ensemble of Q networks ( as does REDQ and TQC ) . Employing multiple networks , one for each Q-function output , can be expensive in terms of computation and memory . In order to reduce the computation and memory requirements , we also consider multi-head architectures for generating multiple Q-function outputs . Instead of each network providing a single Q-estimate output , we consider N separate Q networks each with h heads , providing a total of h ·N estimates . The h heads from one network share all of the layers except the final fully-connected layer . In practice , we have found h = 2 heads works well for AQE , consistent with work in ensembles of convolutional neural networks for computer vision tasks ( Lee et al. , 2015 ) . When properly sharing low level weights , multi-headed networks may not only retain the performance of full ensembles , but can sometimes outperform them . We conduct ablation studies on the multi-head architecture in AQE in Section 4.2 . 4 EXPERIMENTS RESULTS . We provide experimental results in this section for AQE , TQC , REDQ and SAC for the five most challenging MuJoCo environments , namely , Hopper , Walker2d , HalfCheetah , Ant and Humanoid . To make a fair comparison , the TQC , REDQ and SAC results are reproduced using the authors ’ open source code , and use the same network sizes and hyperparameters reported in their papers . In particular , TQC employs 5 critic networks with 25 distributional samples for a total of 125 atoms . TQC drops 5 atoms per critic for Hopper , 0 atoms per critic for Half Cheetah , and 2 atoms per critic for Walker , Ant , and Humanoid . For REDQ , we also use the authors ’ suggested values of N = 10 and M = 2 , where M is the number ensemble members used in the target calculation . Algorithm 1 Aggressive Q-Learning with Ensembles 1 : Initial policy parameters θ , N Q-function parameters φi , i = 1 , . . . , N , empty replay buffer D. Set target parameters φtarg , i ← φi for i = 1 , 2 , . . . , N. 2 : repeat 3 : Take one action at ∼ πθ ( ·|st ) . Observe reward rt , new state st+1 . 4 : Add data to replay buffer : D ← D ∪ { ( st , at , rt , st+1 ) } 5 : for G updates do 6 : Randomly sample a mini-batch B = { ( s , a , r , s′ ) } from D. 7 : for each ( s , a , r , s′ ) ∈ B do 8 : Sample ã′ ∼ πθ ( ·|s′ ) . 9 : Determine the K indices from i = 1 , . . . , N that minimize Qtarget , i ( s′ , ã′ ) . 10 : Compute the Q target y : y ( s , a ) = r + γ ( 1 K ∑ i∈K Qφtarg , i ( s ′ , ã′ ) − α log πθ ( ã′|s′ ) ) 11 : for i = 1 , . . . , N do 12 : Update φi with gradient descent using ∇φi 1|B| ∑ ( s , a , r , s′ ) ∈B ( Qφi ( s , a ) − y ( s , a ) ) 2 13 : Update target networks with φtarg , i ← ρφtarg , i + ( 1− ρ ) φi 14 : Update policy parameters θ with gradient ascent using ∇θ 1|B| ∑ s∈B ( 1 N N∑ i=1 Qφi ( s , ãθ ( s ) ) − α log πθ ( ãθ ( s ) |s ) ) ãθ ( s ) ∼ πθ ( ·|s ) 15 : until Convergence The REDQ paper uses G = 20 for the update-to-data ratio , and provides results for up to 300K environment interactions . Using such a high value for G is computationally infeasible in our our experimental setting , since we use 3 million environment interactions for Ant and Humanoid in order to investigate asymptotic performance as well early-stage sample efficiency . In the experiments reported here , we use a value of G = 5 for both REDQ and AQE . For AQE , we use 10 Q-networks each with 2 heads , producing 20 Q-values for each input . The AQE networks are the same size as those in the REDQ paper . AQE keeps 10 out of 20 values for Hopper , all 20 values for half-Cheetah , and 16 out of 20 values for Walker , Ant and Humanoid . Figure 1 shows the training curves for AQE , TQC , REDQ , and SAC . For each algorithm , we plot the average return of 5 independent trials as the solid curve , and plot the standard deviation across 5 seeds as the shaded region . For each environment , we train each algorithm for exactly the same number of environment interactions as done in the SAC paper . We use the same evaluation protocol as in the TQC paper . Specifically , after every epoch , we run ten test episodes with the current policy , record the undiscounted sum of all the rewards in each episode and take the average of the sums as the performance . A more detailed discussion on hyperparameters and implementation details is given in the Appendix . We see from Figure 1 that AQE is the only algorithm that beats SAC in all five environments during all stages of training . Moreover , it typically beats SAC by a very wide margin . Table 1 shows that , when averaged across the five environments , AQE achieves SAC asymptotic performance approximately 3x faster than SAC and 2x faster than REDQ and TQC . As seen from Figure 1 and Table 2 , in the early stages of training , AQE matches the excellent performance of REDQ in all five environments , and both algorithms are much more sample efficient than SAC and TQC . As seen from Figure 1 and Table 3 , in late-stage training , AQE always matches or beats all other algorithms , except for Humanoid , where TQC is about 10 % better . Table 3 shows that , when averaged across all five environments , AQE ’ s asymptotic performance is 26 % , 22 % , and 6 % higher than SAC , REDQ , and TQC , respectively .

Summary: The paper proposes a new method for mitigating the overestimation bias in Q-learning-like algorithms. An ensemble of critics of size N is trained while only the small K individuals are used for computing the bootstrapping targets. Empirical results are provided to support the claim.