Regularization Matters in Policy Optimization - An Empirical Study on Continuous Control

1 INTRODUCTION . The use of regularization methods to prevent overfitting is a key technique in successfully training neural networks . Perhaps the most widely recognized regularization methods in deep learning are L2 regularization ( also known as weight decay ) and dropout ( Srivastava et al. , 2014 ) . These techniques are standard practices in supervised learning tasks across many domains . Major tasks in computer vision , e.g. , image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) , object detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ) , use L2 regularization as a default option . In natural language processing , for example , the Transformer ( Vaswani et al. , 2017 ) uses dropout , and the popular BERT model ( Devlin et al. , 2018 ) uses L2 regularization . In fact , it is rare to see state-of-the-art neural models trained without regularization in a supervised setting . However , in deep reinforcement learning ( deep RL ) , those conventional regularization methods are largely absent or underutilized in past research , possibly because in most cases we are maximizing the return on the same task as in training . In other words , there is no generalization gap from the training environment to the test environment ( Cobbe et al. , 2018 ) . Heretofore , researchers in deep RL have focused on high-level algorithm design and largely overlooked issues related to network training , including regularization . For popular policy optimization algorithms like Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) , Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , and Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) , conventional regularization methods were not considered . In popular codebases such as the OpenAI Baseline ( Dhariwal et al. , 2017 ) , L2 regularization and dropout were not incorporated . Instead , a commonly used regularization in RL is the entropy regularization which penalizes the high-certainty output from the policy network to encourage more exploration and prevent the agent from overfitting to certain actions . The entropy ∗Equal contribution regularization was first introduced by ( Williams & Peng , 1991 ) and now used by many contemporary algorithms ( Mnih et al. , 2016 ; Schulman et al. , 2017 ; Teh et al. , 2017 ; Farebrother et al. , 2018 ) . In this work , we take an empirical approach to assess the conventional paradigm which omits common regularization when learning deep RL models . We study agent performance on current task ( the environment which the agent is trained on ) , rather than its generalization ability to different environments as in many recent works ( Zhao et al. , 2019 ; Farebrother et al. , 2018 ; Cobbe et al. , 2018 ) . We specifically focus our study on policy optimization methods , which are becoming increasingly popular and have achieved top performance on various tasks . We evaluate four popular policy optimization algorithms , namely SAC , PPO , TRPO , and the synchronous version of Advantage Actor Critic ( A2C ) , on multiple continuous control tasks . Various conventional regularization techniques are considered , including L2/L1 weight regularization , dropout , weight clipping ( Arjovsky et al. , 2017 ) and Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . We compare the performance of these regularization techniques to that without regularization , as well as the entropy regularization . Surprisingly , even though the training and testing environments are the same , we find that many of the conventional regularization techniques , when imposed to the policy networks , can still bring up the performance , sometimes significantly . Among those regularizers , L2 regularization tends to be the most effective overall . L1 regularization and weight clipping can boost performance in many cases . Dropout and Batch Normalization tend to bring improvements only on off-policy algorithms . Additionally , all regularization methods tend to be more effective on more difficult tasks . We also verify our findings with a wide range of training hyperparameters and network sizes , and the result suggests that imposing proper regularization can sometimes save the effort of tuning other training hyperparameters . We further study which part of the policy optimization system should be regularized , and conclude that generally only regularizing the policy network suffices , as imposing regularization on value networks usually does not help . Finally we discuss and analyze possible reasons for some experimental observations . Our main contributions can be summarized as follows : • To our best knowledge , we provide the first systematic study of common regularization methods in policy optimization , which have been largely ignored in the deep RL literature . • We find conventional regularizers can be effective on continuous control tasks ( especially on harder ones ) with statistical significance , under randomly sampled training hyperparameters . Interestingly , simple regularizers ( L2 , L1 , weight clipping ) could perform better than entropy regularization , with L2 generally the best . BN and dropout can only help in off-policy algorithms . • We study which part of the network ( s ) should be regularized . The key lesson is to regularize the policy network but not the value network . • We analyze why regularization may help generalization in RL through sample complexity , return distribution , weight norm , and training noise robustness . 2 RELATED WORKS . Regularization in Deep RL . There have been many prior works studying the theory of regularization in policy optimization ( Farahmand et al. , 2009 ; Neu et al. , 2017 ; Zhang et al. , 2020 ) . In practice , conventional regularization methods have rarely been applied in deep RL . One rare case of such use is in Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2016 ) , where Batch Normalization is applied to all layers of the actor and some layers of the critic , and L2 regularization is applied to the critic . Some recent studies have developed more complicated regularization approaches to continuous control tasks . Cheng et al . ( 2019 ) regularizes the stochastic action distribution π ( a|s ) using a control prior and dynamically adjusts regularization weight based on the temporal difference ( TD ) error . Parisi et al . ( 2019 ) uses TD error regularization to penalize inaccurate value estimation and Generalized Advantage Estimation ( GAE ) ( Schulman et al. , 2016 ) regularization to penalize GAE variance . However , most of these regularizations are rather complicated ( Cheng et al. , 2019 ) or catered to certain algorithms ( Parisi et al. , 2019 ) . Also , these techniques consider regularizing the output of the network , while conventional methods mostly directly regularize the parameters . In this work , we focus on studying these simpler but under-utilized regularization methods . Generalization in Deep RL typically refers to how the model perform in a different environment from the one it is trained on . The generalization gap can come from different modes/levels/difficulties of a game ( Farebrother et al. , 2018 ) , simulation vs. real world ( Tobin et al. , 2017 ) , parameter variations ( Pattanaik et al. , 2018 ) , or different random seeds in environment generation ( Zhang et al. , 2018b ) . There are a number of methods designed to address this issue , e.g. , through training the agent over multiple domains/tasks ( Tobin et al. , 2017 ; Rajeswaran et al. , 2017 ) , adversarial training ( Tobin et al. , 2017 ) , designing model architectures ( Srouji et al. , 2018 ) , adaptive training ( Duan et al. , 2016 ) , etc . Meta RL ( Finn et al. , 2017 ; Gupta et al. , 2018 ; Al-Shedivat et al. , 2017 ) try to learn generalizable agents by training on many environments drawn from the same family/distribution . There are also some comprehensive studies on RL generalization with interesting findings ( Zhang et al. , 2018a ; b ; Zhao et al. , 2019 ; Packer et al. , 2018 ) , e.g. , algorithms performing better in training environment could perform worse with domain shift ( Zhao et al. , 2019 ) . Recently , several studies have investigated conventional regularization ’ s effect on generalization across tasks . ( Farebrother et al. , 2018 ) shows that in Deep Q-Networks ( DQN ) , L2 regularization and dropout are sometime beneficial when evaluated on the same Atari game with mode variations . ( Cobbe et al. , 2018 ) shows that L2 regularization , dropout , BN , and data augmentation can improve generalization performance , but to a less extent than entropy regularization and -greedy exploration . Different from those studies , we focus on regularization ’ s effect in the same environment , yet on which conventional regularizations are under-explored . 3 EXPERIMENTS . 3.1 SETTINGS . Regularization Methods . We study six regularization methods , namely , L2 and L1 weight regularization , weight clipping , Dropout ( Srivastava et al. , 2014 ) , Batch Normalization ( Ioffe & Szegedy , 2015 ) , and entropy regularization . See Appendix A for detailed introduction . Note that we consider entropy as a separate regularization method because it encourages exploration and helps to prevent premature convergence ( Mnih et al. , 2016 ) . In Appendix N , we show that in the presence of certain regularizers , adding entropy on top does not lead to significant performance difference . Algorithms . We evaluate regularization methods on four popular policy optimization algorithms , namely , A2C ( Mnih et al. , 2016 ) , TRPO ( Schulman et al. , 2015 ) , PPO ( Schulman et al. , 2017 ) , and SAC ( Haarnoja et al. , 2018 ) . The first three algorithms are on-policy while the last one is off-policy . For the first three algorithms , we adopt the code from OpenAI Baseline ( Dhariwal et al. , 2017 ) , and for SAC , we use the official implementation at ( Haarnoja , 2018 ) . Tasks . The algorithms with different regularizers are tested on nine continuous control tasks : Hopper , Walker , HalfCheetah , Ant , Humanoid , and HumanoidStandup from MuJoCo ( Todorov et al. , 2012 ) ; Humanoid , AtlasForwardWalk , and HumanoidFlagrun from RoboSchool ( OpenAI ) . Among the MuJoCo tasks , agents for Hopper , Walker , and HalfCheetah are easier to learn , while Ant , Humanoid , HumanoidStandup are relatively harder ( larger state-action space , more training examples ) . The three Roboschool tasks are even harder than the MuJoCo tasks as they require more timesteps to converge ( Klimov & Schulman , 2017 ) . To better understand how different regularization methods work on different difficulties , we roughly categorize the first three environments as “ easy ” tasks and the last six as “ hard ” tasks . Besides continuous control , we provide results on randomly sampled Atari environments ( Bellemare et al. , 2012 ) in Appendix S , which have discrete action space and different reward properties . Our observations are mostly similar to those on continuous control tasks . Training . On MuJoCo tasks , we keep all hyperparameters unchanged as in the codebase adopted . Since hyperparameters for the RoboSchool tasks are not included , we briefly tune the hyperparameters for each algorithm before we apply regularization ( details in Appendix D ) . For details on regularization strength tuning , please see Appendix C. The results shown in this section are obtained by only regularizing the policy network , and a further study on this will be presented in Section 5 . We run each experiment independently with five seeds , then use the average return over the last 100 episodes as the final result . Each regularization method is evaluated independently , with other regularizers turned off . We refer to the result without any regularization as the baseline . For BN and dropout , we use its training mode in updating the network , and test mode in sampling trajectories . During training , negligible computation overhead is induced when a regularizer is applied . Specifically , the increase in training time for BN is ∼ 10 % , dropout ∼ 5 % , while L2 , L1 , weight clipping , and entropy regularization are all < 1 % . We used up to 16 NVIDIA Titan Xp GPUs and 96 Intel Xeon E5-2667 CPUs , and all experiments take roughly 57 days with resources fully utilized . Additional Notes . 1 . Note that entropy regularization is still applicable for SAC , despite it already incorporates the maximization of entropy in the reward . In our experiments , we add the entropy regularization term to the policy loss function in equation ( 12 ) of ( Haarnoja et al. , 2018 ) . 2 . In our experiments , L2 regularization loss is added to the training loss , which is then optimized using Adam ( Kingma & Ba , 2015 ) . ( Loshchilov & Hutter , 2019 ) observes that L2 regularization interacts poorly with Adam and proposes AdamW to decouple weight decay from the optimization steps . However , in policy optimization algorithms , we find that the performance of AdamW with decoupled weight decay is slightly worse than the performance of Adam with L2 loss directly added . Comparisons are shown in Appendix O . 3 . Policy network dropout is not applicable to TRPO because during policy updates , different neurons in the old and new policy networks are dropped out , causing different shifts in the old and new action distributions given the same state , which violates the trust region constraint . In this case , the algorithm fails to perform any update from network initialization .

Summary: This paper conducts a comprehensive study on the effect of different regularization on Deep RL algorithms. Regularization has been mostly neglected in RL as most benefits were believed to be in generalization to unseen test environments in supervised learning settings. However, this paper shows that regularization does provide benefit even though training/testing is done on the same environment in deep RL settings.