Aggressive Q-Learning with Ensembles: Achieving Both High Sample Efficiency and High Asymptotic Performance

1 INTRODUCTION . Off-policy Deep Reinforcement Learning algorithms aim to improve sample efficiency by reusing past experience . A number of off-policy Deep RL algorithms have been proposed for control tasks with continuous state and action spaces , including Deep Deterministic Policy Gradient ( DDPG ) , Twin Delayed DDPG ( TD3 ) and Soft Actor Critic ( SAC ) ( Lillicrap et al. , 2016 ; Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ; b ) . TD3 introduced clipped double-Q learning , and was shown to be significantly more sample efficient than popular on-policy methods for a wide range of MuJoCo benchmarks . Soft Actor Critic ( SAC ) has similar off-policy structures with clipped double-Q learning , but it also employs maximum entropy reinforcement learning . SAC was shown to provide excellent sample efficiency and asymptotic performance in a wide-range of MuJoCo environments , including the high-dimensional Humanoid environment for which both DDPG and TD3 perform poorly . More recently , Kuznetsov et al . ( 2020 ) proposed Truncated Quantile Critics ( TQC ) , a model-free algorithm which includes distributional representations of critics , truncation of critics prediction , and ensembling of multiple critics . Instead of the usual modeling of the Q-function of the expectation of return , TQC approximates the distribution of the return random variable conditioned on the state and action . By dropping several of the top-most “ atoms ” and varying the number of dropped atoms of the return distribution approximation , TQC can control the over-estimation bias . TQC ’ s asymptotic performance ( that is after a long period of training ) was shown to be better than that of SAC on the continuous control MuJoCo benchmark suite , including a 25 % improvement on the most challenging Humanoid environment . However , TQC is not sample efficient in that it generally requires a large number of samples to reach even moderate performance levels . Chen et al . ( 2021 ) proposed Randomized Ensembled Double Q-learning ( REDQ ) , a model-free algorithm which includes a high Update-To-Data ( UTD ) ratio , an ensemble of Q functions , and in-target minimization across a random subset of Q functions from the ensemble . Using a UTD ratio much larger than one , meaning that several gradient steps are taken for each environment interaction , improves sample efficiency , while the ensemble and in-target minimization allows the algorithm to maintain stable and near-uniform bias under the high UTD ratio . The algorithm was shown to attain much better performance than SAC at the early stage of training , and to match or improve the sample-efficiency of the state-of-the-art model-based algorithms for the MuJoCo benchmarks . How- ever , although REDQ is highly sample efficient for early-stage training , its asymptotic performance is significantly below that of TQC . Is it possible to design a simple , streamlined model-free algorithm which can achieve REDQ ’ s high sample efficiency in early-stage training and also achieve TQC ’ s high asymptotic performance in late state training ? In this paper , we achieve this goal with a new model-free algorithm , Aggressive QLearning with Ensembles ( AQE ) . Like TQC and REDQ , AQE uses an ensemble of Q-functions , and like REDQ it uses a UTD ratio > 1 . However AQE is very simple , requiring neither distributional representation of critics as in TQC nor target randomization and double-Q learning as in REDQ . AQE controls over estimation bias and the standard deviation of the bias by varying the number of ensemble members N and the number of ensembles K ≤ N that are kept when calculating the targets . Through carefully designed experiments , we provide a detailed analysis of AQE . We first show that for the MuJoCo benchmark , AQE provides state-of-the-art performance , surpassing the performance of SAC , REDQ , and TQC at all stages of training . We perform an ablation study , and show that AQE is robust to choices of hyperparameters : AQE can work well with small ensembles consisting of 10-20 ensemble members , and performance does not vary significantly with small changes in the keep parameter K. We show that that AQE performs better than several variations , including using the median of all ensemble members and removing the most extreme minimum and maximum outlier in the targets . In order to improve computational time , we also consider different multihead architectures for the ensemble of critics : consistent with the supervised convolutional network literature , we find that a two-head architecture not only reduces computational time but can actually improve performance for some environments . Additionally , we show that AQE continues to outperform SAC and TQC even when these algorithms are made aggressive with a UTD 1 . To ensure our comparisons are fair , and to ensure our results are reproducible ( Henderson et al. , 2018 ; Islam et al. , 2017 ; Duan et al. , 2016 ) , we provide open source code1 . For all algorithmic comparisons , we use the the authors ’ code . 2 ADDITIONAL RELATED WORK . Overestimation bias due to in target maximization in Q-learning can significantly slow learning ( Thrun & Schwartz , 1993 ) . For tabular Q-learning , van Hasselt ( 2010 ) introduced Double QLearning , and showed that it removes the overestimation basis and in general leads to an underestimation bias . Hasselt et al . ( 2016 ) showed that adding Double Q-learning to deep-Q networks can have a similar effect , leading to a major performance boost for the Atari games benchmark . As stated in the Introduction , for continuous-action spaces , TD3 and SAC address the overestimation bias using clipped-double Q-learning , which brings significant performance improvements ( Fujimoto et al. , 2018 ; Haarnoja et al. , 2018a ; b ) . As mentioned in the Introduction , Kuznetsov et al . ( 2020 ) control the over-estimation bias by estimating the distribution of the return random variable , and then by dropping several of the top-most “ atoms ” from the estimated distribution . The distribution estimate is based on a methodology developed in Bellemare et al . ( 2017 ) ; Dabney et al . ( 2018a ; b ) , which employs an asymmetric Huber loss function to minimize the Wasserstein distance between the neural network output distribution and the target distribution . In this paper , in order to counter over-estimation bias , we also drop the top-most estimates , although we do so solely with an ensemble of Q-function mean estimators rather than with an ensemble of the more complex distributional models employed in ( Kuznetsov et al. , 2020 ) . It is well-known that using ensembles can improve the performance of DRL algorithms ( Faußer & Schwenker , 2015 ; Osband et al. , 2016 ; Lee et al. , 2021 ) . For Q-learning based methods , Anschel et al . ( 2017 ) use the average of multiple Q estimates to reduce variance . Lan et al . ( 2020 ) introduced Maxmin Q-learning , which uses the minimum of all the Q-estimates rather than the average . Agarwal et al . ( 2020 ) use Random Ensemble Mixture ( REM ) , which employs a random convex combination of multiple Q estimates . 1https : //anonymous.4open.science/r/Aggressive-Q-Learning-with-Ensembles-0FC0/ Model-based methods often attain high sample efficiency by using a high UTD ratio . In particular , Model-Based Policy Optimization ( MBPO ) ( Janner et al. , 2019 ) uses a large UTD ratio of 20-40 . Compared to Soft-Actor-Critic ( SAC ) , which is model-free and uses a UTD of 1 , MBPO achieves much higher sample efficiency in the OpenAI MuJoCo benchmark ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) . REDQ ( Chen et al. , 2021 ) , a model-free algorithm , also successfully employs a high UTD ratio to achieve high sample efficiency . 3 ALGORITHM . We propose Aggressive Q-learning with Ensembles ( AQE ) , a simple model-free algorithm which provides state-of-the-art performance for the MuJoCo benchmark for both early and late stage of training . The pseudo-code is shown in Algorithm 1 . As is the case with most off-policy continuouscontrol algorithms , AQE has a single actor ( policy network ) and multiple critics ( Q-function networks ) , and employs Polyak averaging of the target parameters to enhance stability . Building on this algorithmic base , it also employs an update-to-data ratio G > 1 , an ensemble of N ≥ 3 Q-functions ( rather than just two as in TD3 and SAC ) , and targets that average all the Q-functions excluding the Q-functions with the highest N − K values . For exploration , it uses entropy maximization as in SAC , although it could easily incorporate alternative exploration schemes . The design of AQE borrows components from TQC and REDQ , but also excludes components from those algorithms . Like TQC , AQE does not employ clipped double Q-learning to control over estimation bias , but instead drops the highest Q values when calculating the targets . Like REDQ , AQE employs a UTD > 1 to improve sample efficiency during training . However , unlike TQC , AQE uses a UTD > 1 , and does not employ distributional representations of critics but instead the simpler Q-function expectation estimates . Unlike REDQ which employs two randomly chosen ensemble members when calculating the target , AQE does not use target randomization and uses most of the ensemble members when calculating the target . As discussed in the theory Section 5 , this provides AQE more flexibility through the choice of the N . The resulting algorithm is not only simple and streamlined , but also provides state-of-the art performance . AQE has three key hyperparameters , G , N , and K. If we set N = 2 , K = 1 and G = 1 , AQE is simply the underlying off-policy algorithm such as SAC . When N > 2 , K = 1 and G = 1 , then AQE becomes similar to , but not equivalent to , Maxmin Q-learning ( Lan et al. , 2020 ) . 3.1 MULTI-HEAD ENSEMBLE ARCHITECTURE . AQE uses an ensemble of Q networks ( as does REDQ and TQC ) . Employing multiple networks , one for each Q-function output , can be expensive in terms of computation and memory . In order to reduce the computation and memory requirements , we also consider multi-head architectures for generating multiple Q-function outputs . Instead of each network providing a single Q-estimate output , we consider N separate Q networks each with h heads , providing a total of h ·N estimates . The h heads from one network share all of the layers except the final fully-connected layer . In practice , we have found h = 2 heads works well for AQE , consistent with work in ensembles of convolutional neural networks for computer vision tasks ( Lee et al. , 2015 ) . When properly sharing low level weights , multi-headed networks may not only retain the performance of full ensembles , but can sometimes outperform them . We conduct ablation studies on the multi-head architecture in AQE in Section 4.2 . 4 EXPERIMENTS RESULTS . We provide experimental results in this section for AQE , TQC , REDQ and SAC for the five most challenging MuJoCo environments , namely , Hopper , Walker2d , HalfCheetah , Ant and Humanoid . To make a fair comparison , the TQC , REDQ and SAC results are reproduced using the authors ’ open source code , and use the same network sizes and hyperparameters reported in their papers . In particular , TQC employs 5 critic networks with 25 distributional samples for a total of 125 atoms . TQC drops 5 atoms per critic for Hopper , 0 atoms per critic for Half Cheetah , and 2 atoms per critic for Walker , Ant , and Humanoid . For REDQ , we also use the authors ’ suggested values of N = 10 and M = 2 , where M is the number ensemble members used in the target calculation . Algorithm 1 Aggressive Q-Learning with Ensembles 1 : Initial policy parameters θ , N Q-function parameters φi , i = 1 , . . . , N , empty replay buffer D. Set target parameters φtarg , i ← φi for i = 1 , 2 , . . . , N. 2 : repeat 3 : Take one action at ∼ πθ ( ·|st ) . Observe reward rt , new state st+1 . 4 : Add data to replay buffer : D ← D ∪ { ( st , at , rt , st+1 ) } 5 : for G updates do 6 : Randomly sample a mini-batch B = { ( s , a , r , s′ ) } from D. 7 : for each ( s , a , r , s′ ) ∈ B do 8 : Sample ã′ ∼ πθ ( ·|s′ ) . 9 : Determine the K indices from i = 1 , . . . , N that minimize Qtarget , i ( s′ , ã′ ) . 10 : Compute the Q target y : y ( s , a ) = r + γ ( 1 K ∑ i∈K Qφtarg , i ( s ′ , ã′ ) − α log πθ ( ã′|s′ ) ) 11 : for i = 1 , . . . , N do 12 : Update φi with gradient descent using ∇φi 1|B| ∑ ( s , a , r , s′ ) ∈B ( Qφi ( s , a ) − y ( s , a ) ) 2 13 : Update target networks with φtarg , i ← ρφtarg , i + ( 1− ρ ) φi 14 : Update policy parameters θ with gradient ascent using ∇θ 1|B| ∑ s∈B ( 1 N N∑ i=1 Qφi ( s , ãθ ( s ) ) − α log πθ ( ãθ ( s ) |s ) ) ãθ ( s ) ∼ πθ ( ·|s ) 15 : until Convergence The REDQ paper uses G = 20 for the update-to-data ratio , and provides results for up to 300K environment interactions . Using such a high value for G is computationally infeasible in our our experimental setting , since we use 3 million environment interactions for Ant and Humanoid in order to investigate asymptotic performance as well early-stage sample efficiency . In the experiments reported here , we use a value of G = 5 for both REDQ and AQE . For AQE , we use 10 Q-networks each with 2 heads , producing 20 Q-values for each input . The AQE networks are the same size as those in the REDQ paper . AQE keeps 10 out of 20 values for Hopper , all 20 values for half-Cheetah , and 16 out of 20 values for Walker , Ant and Humanoid . Figure 1 shows the training curves for AQE , TQC , REDQ , and SAC . For each algorithm , we plot the average return of 5 independent trials as the solid curve , and plot the standard deviation across 5 seeds as the shaded region . For each environment , we train each algorithm for exactly the same number of environment interactions as done in the SAC paper . We use the same evaluation protocol as in the TQC paper . Specifically , after every epoch , we run ten test episodes with the current policy , record the undiscounted sum of all the rewards in each episode and take the average of the sums as the performance . A more detailed discussion on hyperparameters and implementation details is given in the Appendix . We see from Figure 1 that AQE is the only algorithm that beats SAC in all five environments during all stages of training . Moreover , it typically beats SAC by a very wide margin . Table 1 shows that , when averaged across the five environments , AQE achieves SAC asymptotic performance approximately 3x faster than SAC and 2x faster than REDQ and TQC . As seen from Figure 1 and Table 2 , in the early stages of training , AQE matches the excellent performance of REDQ in all five environments , and both algorithms are much more sample efficient than SAC and TQC . As seen from Figure 1 and Table 3 , in late-stage training , AQE always matches or beats all other algorithms , except for Humanoid , where TQC is about 10 % better . Table 3 shows that , when averaged across all five environments , AQE ’ s asymptotic performance is 26 % , 22 % , and 6 % higher than SAC , REDQ , and TQC , respectively .

Summary: **Summary**  This work introduces Aggressive Q-learning with Ensembles (AQE), a simple model-free reinforcement learning algorithm. AQE can be considered as a modified version of the Randomised Ensembled Double Q-Learning (REDQ) algorithm. Specifically, AQE uses a number ($1 \leq K \leq N$) of the ensemble members ($N$) with the lowest Q values for calculating the target, instead of sampling a number ($1 \leq M \leq N$) of ensemble members and keeping the one with the lowest Q values as in the case of REDQ. Similarly to REDQ, AQE also employs a high update-to-data ratio ($G > 1$) to improve the sample efficiency. Empirical analyses have been conducted on five MuJoCo environments showing that the performance AQE is competitive to that of Truncated Quantile Critics (TQC) and improves the sample efficiency performance of REDQ.