Randomized Ensembled Double Q-Learning: Learning Fast Without a Model

1 INTRODUCTION . Recently , model-based methods in continuous action space domains have achieved much higher sample efficiency than previous model-free methods . Model-based methods often attain higher sample efficiency by using a high Update-To-Data ( UTD ) ratio , which is the number of updates taken by the agent compared to the number of actual interactions with the environment . For example , Model-Based Policy Optimization ( MBPO ) ( Janner et al. , 2019 ) , is a state-of-the-art model-based algorithm which updates the agent with a mix of real data from the environment and “ fake ” data from its model , and uses a large UTD ratio of 20-40 . Compared to Soft-Actor-Critic ( SAC ) , which is model-free and uses a UTD of 1 , MBPO achieves much higher sample efficiency in the OpenAI MuJoCo benchmark ( Todorov et al. , 2012 ; Brockman et al. , 2016 ) . This raises the question of whether it is also possible to achieve such high performance without a model ? In this paper , we introduce a simple model-free algorithm called Randomized Ensemble Double Q learning ( REDQ ) , and show that its performance is just as good as , if not better than , MBPO . The result indicates , that at least for the MuJoCo benchmark , simple model-free algorithms can attain the performance of current state-of-the-art model-based algorithms . Moreover , REDQ can achieve this performance using fewer parameters than MBPO , and with less wall-clock run time . Like MBPO , REDQ employs a UTD ratio that is 1 , but unlike MBPO it is model-free , has no roll outs , and performs all updates with real data . In addition to using a UTD ratio that is 1 , it has two other carefully integrated ingredients : an ensemble of Q functions ; and in-target minimization across a random subset of Q functions from the ensemble . Through carefully designed experiments , we provide a detailed analysis of REDQ . We introduce the metrics of average Q-function bias and standard deviation ( std ) of Q-function bias . Our results show that using ensembles with in-target minimization reduces the std of the Q-function bias to close to ∗Equal contribution , in alphabetical order . †Correspondence to : Keith Ross < keithwross @ nyu.edu > . zero for most of training , even when the UTD is very high . Furthermore , by adjusting the number of randomly selected Q-functions for in-target minimization , REDQ can control the average Q-function bias . In comparison with standard ensemble averaging and with SAC with a higher UTD , REDQ has much lower std of Q-function bias while maintaining an average bias that is negative but close to zero throughout most of training , resulting in significantly better learning performance . We perform an ablation study , and show that REDQ is very robust to choices of hyperparameters , and can work well with a small ensemble and a small number of Q functions in the in-target minimization . We also provide a theoretical analysis , providing additional insights into REDQ . Finally , we consider combining the REDQ algorithm with an online feature extractor network ( OFENet ) ( Ota et al. , 2020 ) to further improve performance , particularly for the more challenging environments Ant and Humanoid . We achieve more than 7x the sample efficiency of SAC to reach a score of 5000 for both Ant and Humanoid . In Humanoid , REDQ-OFE also greatly outperforms MBPO , reaching a score of 5000 at 150K interactions , which is 3x MBPO ’ s score at that point . To ensure our comparisons are fair , and to ensure our results are reproducible ( Henderson et al. , 2018 ; Islam et al. , 2017 ; Duan et al. , 2016 ) , we provide open source code1 . For all algorithmic comparisons , we use the same codebase ( except for MBPO , for which we use the authors ’ code ) . 2 RANDOMIZED ENSEMBLED DOUBLE Q-LEARNING ( REDQ ) . Janner et al . ( 2019 ) proposed Model-Based Policy Optimization ( MBPO ) , which was shown to be much more sample efficient than popular model-free algorithms such as SAC and PPO for the MuJoCo environments . MBPO learns a model , and generates “ fake data ” from its model as well as “ real data ” through environment interactions . It then performs parameter updates using both the fake and the real data . One of the distinguishing features of MBPO is that it has a UTD ratio 1 for updating its Q functions , enabling MBPO to achieve high sample efficiency . We propose Randomized Ensembled Double Q-learning ( REDQ ) , a novel model-free algorithm whose sample-efficiency performance is just as good as , if not better than , the state-of-the-art modelbased algorithm for the MuJoCo benchmark . The pseudocode for REDQ is shown in Algorithm 1 . REDQ can be used with any standard off-policy model-free algorithm , such as SAC ( Haarnoja et al. , 2018b ) , SOP ( Wang et al. , 2019 ) , TD3 ( Fujimoto et al. , 2018 ) , or DDPG ( Lillicrap et al. , 2015 ) . For the sake of concreteness , we use SAC in Algorithm 1 . REDQ has the following key components : ( i ) To improve sample efficiency , the UTD ratio G is much greater than one ; ( ii ) To reduce the variance in the Q-function estimate , REDQ uses an ensemble of N Q-functions , with each Q-function randomly and independently initialized but updated with the same target ; ( iii ) To reduce over-estimation bias , the target for the Q-function includes a minimization over a random subsetM of the N Q-functions . The size of the subsetM is kept fixed , and is denoted as M , and is referred to as the in-target minimization parameter . Since our default choice for M is M = 2 , we refer to the algorithm as Randomized Ensembled Double Q-learning ( REDQ ) . REDQ shares some similarities with Maxmin Q-learning ( Lan et al. , 2020 ) , which also uses ensembles and also minimizes over multiple Q-functions in the target . However , Maxmin Q-learning and REDQ have many differences , e.g. , Maxmin Q-learning minimizes over the full ensemble in the target , whereas REDQ minimizes over a random subset of Q-functions . Unlike Maxmin Q-learning , REDQ controls over-estimation bias and variance of the Q estimate by separately setting M and N . REDQ has many possible variations , some of which are discussed in the ablation section . REDQ has three key hyperparameters , G , N , and M . When N = M = 2 and G = 1 , then REDQ simply becomes the underlying off-policy algorithm such as SAC . When N = M > 2 and G = 1 , then REDQ is similar to , but not equivalent to , Maxmin Q-learning ( Lan et al. , 2020 ) . In practice , we find M = 2 works well for REDQ , and that a wide range of values around N = 10 and G = 20 work well . To our knowledge , REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio G 1 . 1Code and implementation tutorial can be found at : https : //github.com/watchernyu/REDQ Algorithm 1 Randomized Ensembled Double Q-learning ( REDQ ) 1 : Initialize policy parameters θ , N Q-function parameters φi , i = 1 , . . . , N , empty replay buffer D. Set target parameters φtarg , i ← φi , for i = 1 , 2 , . . . , N 2 : repeat 3 : Take one action at ∼ πθ ( ·|st ) . Observe reward rt , new state st+1 . 4 : Add data to buffer : D ← D ∪ { ( st , at , rt , st+1 ) } 5 : for G updates do 6 : Sample a mini-batch B = { ( s , a , r , s′ ) } from D 7 : Sample a setM of M distinct indices from { 1 , 2 , . . . , N } 8 : Compute the Q target y ( same for all of the N Q-functions ) : y = r + γ ( min i∈M Qφtarg , i ( s ′ , ã′ ) − α log πθ ( ã′ | s′ ) ) , ã′ ∼ πθ ( · | s′ ) 9 : for i = 1 , . . . , N do 10 : Update φi with gradient descent using ∇φ 1 |B| ∑ ( s , a , r , s′ ) ∈B ( Qφi ( s , a ) − y ) 2 11 : Update target networks with φtarg , i ← ρφtarg , i + ( 1− ρ ) φi 12 : Update policy parameters θ with gradient ascent using ∇θ 1 |B| ∑ s∈B ( 1 N N∑ i=1 Qφi ( s , ãθ ( s ) ) − α log πθ ( ãθ ( s ) |s ) ) , ãθ ( s ) ∼ πθ ( · | s ) 2.1 EXPERIMENTAL RESULTS FOR REDQ . We now provide experimental results for REDQ and MBPO for the four most challenging MuJoCo environments , namely , Hopper , Walker2d , Ant , and Humanoid . We have taken great care to make a fair comparison of REDQ and MBPO . The MBPO results are reproduced using the author ’ s open source code , and we use the hyperparameters suggested in the MBPO paper , including G = 20 . We obtain MBPO results similar to those reported in the MBPO paper . For REDQ , we use G = 20 , N = 10 , and M = 2 for all environments . We use the evaluation protocol proposed in the MBPO paper . Specifically , after every epoch we run one test episode with the current policy and record the performance as the undiscounted sum of all the rewards in the episode . A more detailed discussion on hyperparameters and implementation details is given in the Appendix . Figure 1 shows the training curves for REDQ , MBPO , and SAC . For each algorithm , we plot the average return of 5 independent trials as the solid curve , and plot the standard deviation across 5 seeds as the transparent shaded region . For each environment , we train each algorithm for exactly the same number of environment interactions as done in the MBPO paper . Figure 1 shows that both REDQ and MBPO learn much faster than SAC , with REDQ performing somewhat better than MBPO on the whole . In particular , REDQ learns significantly faster for Hopper , and has somewhat better asymptotic performance for Hopper , Walker2d , and Humanoid . A more detailed performance comparison is given in the Appendix , where it is shown that , averaging across the environments , REDQ performs 1.4x better than MBPO half-way through training and 1.1x better at the end of training . These results taken together are perhaps counter-intuitive . They show that a simple modelfree algorithm can achieve as good or better sample-efficiency performance as the state-of-the-art model-based algorithm for the MuJoCo environments . Does REDQ achieve its sample efficiency using more computational resources than MBPO ? We now compare the number of parameters used in REDQ and MBPO . With REDQ , for each Q network and the policy network , we use a multi-layer perceptron with two hidden layers , each with 256 units . For MBPO , we use the default network architectures for the Q networks , policy network , and model ensembles ( Janner et al. , 2019 ) . The Appendix provides a table comparing the number of parameters : REDQ uses fewer parameters than MBPO for all four environments , specifically , between 26 % and 70 % as many parameters depending on the environment . Additionally , we measured the runtime on a 2080-Ti GPU and found that MBPO roughly takes 75 % longer . In summary , the results in this section show that the model-free algorithm REDQ is not only at least as sample efficient as MBPO , but also has fewer parameters and is significantly faster in terms of wall-clock time .

Summary: The paper proposes three techniques that altogether greatly improves the performance of soft actor-critic (SAC), resulting in a new algorithm called REDQ. (1) A higher update-to-data ratio, which speeds up the critic update. (2) Using the average ensemble Q for the policy gradient, therefore reducing its variance. (3) Taking the min of a small subset of the ensemble Qs to compute the target Q, therefore reducing the Q bias. The paper also performs extensive ablation studies to prove the importance of each technique.