Is High Variance Unavoidable in RL? A Case Study in Continuous Control

1 INTRODUCTION . With the advent of deep learning , reinforcement learning ( RL ) with function approximators has become a promising strategy for learning complex behavior ( Silver et al. , 2017 ; Mnih et al. , 2015 ) . While RL can work remarkably well , deep RL experiments have been observed to exhibit high variance ( Chan et al. , 2020 ; Clary et al. , 2018 ; Lynnerup et al. , 2020 ) . Specifically , the rewards obtained by RL agents commonly vary significantly depending on the random seeds used for training , even if no other modifications are made . Furthermore , implementation level details ( Engstrom et al. , 2019 ) and even hardware scheduling ( Henderson et al. , 2018 ) often have an outsized effect on performance . High variance is problematic for creating reproducible RL research , and achieving statistically significant results over large RL benchmarks is often prohibitively time-consuming ( Colas et al. , 2018 ; Agarwal et al. , 2021 ) . Furthermore , high variance provides an obstacle for real-world applications of RL , where safety and predictability can be critical ( Garcıa & Fernández , 2015 ; Srinivasan et al. , 2020 ) . Especially relevant RL applications include medical testing ( Bastani et al. , 2021 ) , autonomous vehicles ( Bellemare et al. , 2020 ) and communications infrastructure ( Liu et al. , 2020 ) . In this paper , we investigate causes for high variance in RL . To limit the paper ’ s scope and computational footprint we focus on continuous control from pixels with a state-of-the-art ( SOTA ) actorcritic algorithm ( Lillicrap et al. , 2021 ; Yarats et al. , 2021b ) . This is a popular setup that exhibits high variance , and where actor-critic algorithms often achieve SOTA results ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ) . In this setting , we first show that an important cause of variance is outlier runs that fail early and never recover . It is natural to hypothesize that the typical sources of randomness , network initialization , and initial exploration , are to blame . However , we demonstrate that these do in fact have little impact on the final performance . We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment is at fault . We instead demonstrate that unstable network parametrization can drive the variance and lead to excessively large activations that saturate action distributions . We propose several methods to avoid such issues and find that they decrease variance . The most effective method is simply setting penultimate features to unit norm – a strategy we call penultimate normalization – which avoids exploding activations by design . We also find that partially disabling double clipped Q-learning ( Fujimoto et al. , 2018 ) , a strategy we call asymmetrically double clipped Q-learning , reduces variance for many sparse tasks . By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks . The average standard deviation decreases by a factor > 3 and for some tasks by more than an order of magnitude . Our experiments suggest that high variance is not necessarily inherent to RL . Instead , one can directly optimize for lower variance without hurting the average reward . We argue that developing such low-variance agents is an important goal for the RL community – both for the sake of reproducible research and real-world deployment . 2 BACKGROUND . We consider continuous control modeled as a Markov decision process ( MDP ) with a continuous action space A and state space S. In practice , the action space A is often bounded to account for physical limitations , e.g. , how much force robotic joints can exert . A popular choice is simply A = [ −1 , 1 ] dim ( A ) . Within this setting , Deep Deterministic Policy Gradients ( DDPG ) ( Lillicrap et al. , 2021 ) is a performant algorithm that forms the backbone of many state-of-the-art agents ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ; Yarats et al. , 2021b ) . DDPG uses a neural network ( the critic ) to predict the Q-valueQφ ( at , st ) for each state-action pair . The parameters φ of the critic are trained by minimizing the soft Bellman residual : E ( Qφ ( at , st ) − [ rt + γE [ Q̂ ( at+1 , st+1 ) ] ] ) 2 ( 1 ) Here , rt is the reward and E [ Q̂ ( at+1 , st+1 ) ] is the Q-value estimated by the target critic – a network whose weights can e.g . be the exponentially averaged weights of the critic . One can use multi-step returns for the critic target ( Sutton , 1988 ) . Another common trick is double clipped q-learning , i.e . using two Q-networks Q1 , Q2 and taking Q = min ( Q1 , Q2 ) to decrease Q-value overestimation ( Fujimoto et al. , 2018 ) . The policy is obtained via an actor-network with parameters θ , which maps each state st to a distribution πθ ( st ) over actions . The action is obtained by calculating a pre θ ( s ) , the vector output of the actor network . Thereafter a tanh non-linearity is applied elementwise to ensure a ∈ [ −1 , 1 ] n for n = dim ( A ) . The policy is then obtained as : aθ ( s ) = tanh ( a pre θ ( s ) ) + ( 2 ) is some additive noise which encourages exploration , e.g . a Gaussian variable with time-dependent variance σ2t . Various noise types have been employed ( Achiam , 2018 ; Lillicrap et al. , 2021 ; Yarats et al. , 2021b ) . Since the Q-values correspond to the expected discounted rewards , the actions aθ should maximize Qφ ( aθ , s ) . Thus , to obtain a gradient update for the actor one uses the gradient ∇θQφ ( aθ , s ) , effectively calculating gradients through the critic network . In control from pixels , the state s is typically an image ( possibly data augmented ) processed by a convolutional network . 3 BENCHMARKING VARIANCE . Experimental Setup . To limit the paper ’ s scope and computational footprint , we focus on continuous control from pixels with an actor-critic algorithm . This setup is relatively close to real-world robotics , where reliable performance is important . We consider the standard continuous control benchmark deepmind control ( dm-control ) ( Tassa et al. , 2020 ) . Within dm-control , a long line of state-of-the-art models relies on the actor-critic setup ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ) . We consider the recent data-augmentation based DDPG-type agent DRQv2 ( Yarats et al. , 2021b ) which provides state-of-the-art performance and runs fast . We use the default hyperparameters that Yarats et al . ( 2021b ) uses on the medium benchmark ( listed in Appendix A ) throughout the paper . We will refer to this agent as the baseline agent . We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider . To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al . ( 2021b ) . We select the five tasks with the highest relative variance – finger turn hard , walker walk , hopper hop , acrobot swingup , and reacher hard ( see Table 4 in Appendix A for details ) . In Section 4.2 we consider a larger set of tasks . We will refer to these as environments or tasks interchangeably . For each run , we train the agent for one million frames , or equivalently 1,000 episodes , and evaluate over ten episodes . We use 10 or 40 seeds for each method we consider , these are not shared between experiments , but we reuse the runs for the baseline agent across figures . 3.1 VERIFYING VARIANCE . We first aim to verify that the environments indeed exhibit high variance . To this end , we run the baseline agent with 40 random seeds on each task . Figure 1 shows the result of this experiment for three tasks , see Appendix A for the two other tasks . We show both individual runs as well as mean and standard deviations . For some tasks , the variance is dominated by outlier runs that fail to learn early in the training and never recover . E.g . for the walker task , we see several runs which have not improved during the first 500 episodes , and most of these do not improve in the following 500 episodes either . Except for these outliers , the variance of the walker task is relatively small . We also note that these outliers separate themselves relatively early on , suggesting that there are some issues in early training that cause variance . If it was possible to eliminate poor outlier runs , we would expect the variance to go down . Motivated by these observations , and the walker task in particular , we will investigate various potential causes of outlier runs and high variance . 3.2 DOES INITIALIZATION MATTER ? . There are multiple sources of randomness specific to the early training iterations . The agent networks are randomly initialized and the agent is given some initial experience , collected by random actions , to start training on . Could these two initial factors be the cause of the high variance ? To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience , but otherwise have independent randomness ( from e.g . the environment , action sampling , and mini-batch sampling ) . Across all tasks , we repeat this process for ten combinations of initializations and initial experience , from each such combination we start two runsA , B which do not share other sources of randomness . For each time-step we measure the Pearson correlation between score ( A ) and score ( B ) . This correlation , averaged across all timesteps , is shown in Figure 2 ( left ) . We see that the correlation is very small , suggesting that the initialization and initial experience does not have a large effect on the outcome . Specifically , the hypothesis that there are good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 ( left ) . Correlation across time is noisy but centers around zero , see Appendix A . 3.3 ARE THE ENVIRONMENTS TOO SENSITIVE ? . Another cause of variance is the environments . Perhaps the environments require very specific policies to yield rewards , and unless the agent stumbles upon a good policy early on , learning fails . To study the sensitivity of the environments we consider gradually exchanging a good learned policy with a random policy . Specifically , at every timestep t , with probability p , we exchange the policy ’ s action with a random action . We use the final policy of a performant run with the baseline agent as the learned policy . The effect of this , for ten final policies learned for the walker environment , is shown in Figure 2 ( right ) . Performance degrades gracefully as we gradually move to a random policy . This suggests that the environment is relatively stable with regard to actions . Results for other environments are similar and are shown in Appendix A . Furthermore , in Appendix E we verify that the variance arising from evaluating the agent on a finite number of episodes is small compared to the inter-run variance . Thus , environment sensitivity and evaluation methodology do not appear to explain the excessive variance .

Summary: This paper investigates sources of poor outlier runs in a DDPG-based agent on 5 image-based DM control tasks. The empirical analysis suggests that such outlier runs result from saturating tanh nonlinearities due to large-valued activations in the policy network. To prevent saturation of tanh, the paper proposes and evaluates fixes and shows that combining a bunch of them results in better performance with lower variance.