Is High Variance Unavoidable in RL? A Case Study in Continuous Control

1 INTRODUCTION . With the advent of deep learning , reinforcement learning ( RL ) with function approximators has become a promising strategy for learning complex behavior ( Silver et al. , 2017 ; Mnih et al. , 2015 ) . While RL can work remarkably well , deep RL experiments have been observed to exhibit high variance ( Chan et al. , 2020 ; Clary et al. , 2018 ; Lynnerup et al. , 2020 ) . Specifically , the rewards obtained by RL agents commonly vary significantly depending on the random seeds used for training , even if no other modifications are made . Furthermore , implementation level details ( Engstrom et al. , 2019 ) and even hardware scheduling ( Henderson et al. , 2018 ) often have an outsized effect on performance . High variance is problematic for creating reproducible RL research , and achieving statistically significant results over large RL benchmarks is often prohibitively time-consuming ( Colas et al. , 2018 ; Agarwal et al. , 2021 ) . Furthermore , high variance provides an obstacle for real-world applications of RL , where safety and predictability can be critical ( Garcıa & Fernández , 2015 ; Srinivasan et al. , 2020 ) . Especially relevant RL applications include medical testing ( Bastani et al. , 2021 ) , autonomous vehicles ( Bellemare et al. , 2020 ) and communications infrastructure ( Liu et al. , 2020 ) . In this paper , we investigate causes for high variance in RL . To limit the paper ’ s scope and computational footprint we focus on continuous control from pixels with a state-of-the-art ( SOTA ) actorcritic algorithm ( Lillicrap et al. , 2021 ; Yarats et al. , 2021b ) . This is a popular setup that exhibits high variance , and where actor-critic algorithms often achieve SOTA results ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ) . In this setting , we first show that an important cause of variance is outlier runs that fail early and never recover . It is natural to hypothesize that the typical sources of randomness , network initialization , and initial exploration , are to blame . However , we demonstrate that these do in fact have little impact on the final performance . We similarly provide experiments that suggest that neither poor feature representations nor excessive sensitivity of the environment is at fault . We instead demonstrate that unstable network parametrization can drive the variance and lead to excessively large activations that saturate action distributions . We propose several methods to avoid such issues and find that they decrease variance . The most effective method is simply setting penultimate features to unit norm – a strategy we call penultimate normalization – which avoids exploding activations by design . We also find that partially disabling double clipped Q-learning ( Fujimoto et al. , 2018 ) , a strategy we call asymmetrically double clipped Q-learning , reduces variance for many sparse tasks . By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks . The average standard deviation decreases by a factor > 3 and for some tasks by more than an order of magnitude . Our experiments suggest that high variance is not necessarily inherent to RL . Instead , one can directly optimize for lower variance without hurting the average reward . We argue that developing such low-variance agents is an important goal for the RL community – both for the sake of reproducible research and real-world deployment . 2 BACKGROUND . We consider continuous control modeled as a Markov decision process ( MDP ) with a continuous action space A and state space S. In practice , the action space A is often bounded to account for physical limitations , e.g. , how much force robotic joints can exert . A popular choice is simply A = [ −1 , 1 ] dim ( A ) . Within this setting , Deep Deterministic Policy Gradients ( DDPG ) ( Lillicrap et al. , 2021 ) is a performant algorithm that forms the backbone of many state-of-the-art agents ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ; Yarats et al. , 2021b ) . DDPG uses a neural network ( the critic ) to predict the Q-valueQφ ( at , st ) for each state-action pair . The parameters φ of the critic are trained by minimizing the soft Bellman residual : E ( Qφ ( at , st ) − [ rt + γE [ Q̂ ( at+1 , st+1 ) ] ] ) 2 ( 1 ) Here , rt is the reward and E [ Q̂ ( at+1 , st+1 ) ] is the Q-value estimated by the target critic – a network whose weights can e.g . be the exponentially averaged weights of the critic . One can use multi-step returns for the critic target ( Sutton , 1988 ) . Another common trick is double clipped q-learning , i.e . using two Q-networks Q1 , Q2 and taking Q = min ( Q1 , Q2 ) to decrease Q-value overestimation ( Fujimoto et al. , 2018 ) . The policy is obtained via an actor-network with parameters θ , which maps each state st to a distribution πθ ( st ) over actions . The action is obtained by calculating a pre θ ( s ) , the vector output of the actor network . Thereafter a tanh non-linearity is applied elementwise to ensure a ∈ [ −1 , 1 ] n for n = dim ( A ) . The policy is then obtained as : aθ ( s ) = tanh ( a pre θ ( s ) ) + ( 2 ) is some additive noise which encourages exploration , e.g . a Gaussian variable with time-dependent variance σ2t . Various noise types have been employed ( Achiam , 2018 ; Lillicrap et al. , 2021 ; Yarats et al. , 2021b ) . Since the Q-values correspond to the expected discounted rewards , the actions aθ should maximize Qφ ( aθ , s ) . Thus , to obtain a gradient update for the actor one uses the gradient ∇θQφ ( aθ , s ) , effectively calculating gradients through the critic network . In control from pixels , the state s is typically an image ( possibly data augmented ) processed by a convolutional network . 3 BENCHMARKING VARIANCE . Experimental Setup . To limit the paper ’ s scope and computational footprint , we focus on continuous control from pixels with an actor-critic algorithm . This setup is relatively close to real-world robotics , where reliable performance is important . We consider the standard continuous control benchmark deepmind control ( dm-control ) ( Tassa et al. , 2020 ) . Within dm-control , a long line of state-of-the-art models relies on the actor-critic setup ( Laskin et al. , 2020b ; a ; Kostrikov et al. , 2020 ) . We consider the recent data-augmentation based DDPG-type agent DRQv2 ( Yarats et al. , 2021b ) which provides state-of-the-art performance and runs fast . We use the default hyperparameters that Yarats et al . ( 2021b ) uses on the medium benchmark ( listed in Appendix A ) throughout the paper . We will refer to this agent as the baseline agent . We are interested in using a large number of random seeds to obtain accurate statistics and thus need to limit the number of environments we consider . To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al . ( 2021b ) . We select the five tasks with the highest relative variance – finger turn hard , walker walk , hopper hop , acrobot swingup , and reacher hard ( see Table 4 in Appendix A for details ) . In Section 4.2 we consider a larger set of tasks . We will refer to these as environments or tasks interchangeably . For each run , we train the agent for one million frames , or equivalently 1,000 episodes , and evaluate over ten episodes . We use 10 or 40 seeds for each method we consider , these are not shared between experiments , but we reuse the runs for the baseline agent across figures . 3.1 VERIFYING VARIANCE . We first aim to verify that the environments indeed exhibit high variance . To this end , we run the baseline agent with 40 random seeds on each task . Figure 1 shows the result of this experiment for three tasks , see Appendix A for the two other tasks . We show both individual runs as well as mean and standard deviations . For some tasks , the variance is dominated by outlier runs that fail to learn early in the training and never recover . E.g . for the walker task , we see several runs which have not improved during the first 500 episodes , and most of these do not improve in the following 500 episodes either . Except for these outliers , the variance of the walker task is relatively small . We also note that these outliers separate themselves relatively early on , suggesting that there are some issues in early training that cause variance . If it was possible to eliminate poor outlier runs , we would expect the variance to go down . Motivated by these observations , and the walker task in particular , we will investigate various potential causes of outlier runs and high variance . 3.2 DOES INITIALIZATION MATTER ? . There are multiple sources of randomness specific to the early training iterations . The agent networks are randomly initialized and the agent is given some initial experience , collected by random actions , to start training on . Could these two initial factors be the cause of the high variance ? To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience , but otherwise have independent randomness ( from e.g . the environment , action sampling , and mini-batch sampling ) . Across all tasks , we repeat this process for ten combinations of initializations and initial experience , from each such combination we start two runsA , B which do not share other sources of randomness . For each time-step we measure the Pearson correlation between score ( A ) and score ( B ) . This correlation , averaged across all timesteps , is shown in Figure 2 ( left ) . We see that the correlation is very small , suggesting that the initialization and initial experience does not have a large effect on the outcome . Specifically , the hypothesis that there are good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 ( left ) . Correlation across time is noisy but centers around zero , see Appendix A . 3.3 ARE THE ENVIRONMENTS TOO SENSITIVE ? . Another cause of variance is the environments . Perhaps the environments require very specific policies to yield rewards , and unless the agent stumbles upon a good policy early on , learning fails . To study the sensitivity of the environments we consider gradually exchanging a good learned policy with a random policy . Specifically , at every timestep t , with probability p , we exchange the policy ’ s action with a random action . We use the final policy of a performant run with the baseline agent as the learned policy . The effect of this , for ten final policies learned for the walker environment , is shown in Figure 2 ( right ) . Performance degrades gracefully as we gradually move to a random policy . This suggests that the environment is relatively stable with regard to actions . Results for other environments are similar and are shown in Appendix A . Furthermore , in Appendix E we verify that the variance arising from evaluating the agent on a finite number of episodes is small compared to the inter-run variance . Thus , environment sensitivity and evaluation methodology do not appear to explain the excessive variance .

Summary: The paper studies the variance of the DDPG algorithm on continuous control tasks. They identify that the variance in rewards arises in part due to the failure of some runs to learn. They argue that the failure to learn is due to saturating tanh nonlinearities, and suggest two methods to address this problem.