Regularization Matters in Policy Optimization

Deep Reinforcement Learning ( Deep RL ) has been receiving increasingly more attention thanks to its encouraging performance on a variety of control tasks . Yet , conventional regularization techniques in training neural networks ( e.g. , L2 regularization , dropout ) have been largely ignored in RL methods , possibly because agents are typically trained and evaluated in the same environment . In this work , we present the first comprehensive study of regularization techniques with multiple policy optimization algorithms on continuous control tasks . Interestingly , we find conventional regularization techniques on the policy networks can often bring large improvements on the task performance , and the improvement is typically more significant when the task is more difficult . We also compare with the widely used entropy regularization and find L2 regularization is generally better . Our findings are further confirmed to be robust against the choice of training hyperparameters . We also study the effects of regularizing different components and find that only regularizing the policy network is typically the best option . We hope our study provides guidance for future practices in regularizing policy optimization algorithms . 1 INTRODUCTION . Regularization , typically referring to methods for preventing overfitting , is a key technique in successfully training a neural network . Perhaps the most widely recognized regularization methods in deep learning are L2 regularization ( also known as weight decay ) and dropout ( Srivastava et al. , 2014 ) . Those techniques are standard practices in supervised learning tasks from many domains . Major tasks in computer vision , e.g. , image classification ( He et al. , 2016 ; Huang et al. , 2017 ) , object detection ( Ren et al. , 2015 ; Redmon et al. , 2016 ) , all use L2 regularization as a default option . In natural language processing , for example , the Transformer model ( Vaswani et al. , 2017 ) uses dropout . and the recently popular BERT model ( Devlin et al. , 2018 ) uses L2 regularization . In fact , it is very rare to see state-of-the-art neural models trained without any regularization in a supervised setting . However , in deep reinforcement learning ( RL ) , those conventional regularization methods are largely absent or underutilized in past research , possibly because in most cases we are maximizing the return on exactly the same task as in training . In other words , there is a lack of generalization gap from the training environment to the test environment ( Cobbe et al. , 2018 ) . Moreover , researchers in deep RL focus more on high-level algorithm designs , which is more closely related to the field of reinforcement learning , and focus less on network training techniques such as regularization . For popular policy optimization algorithms like Asynchronous Advantage Actor-Crtic ( A3C ) ( Mnih et al. , 2016 ) , Trust Region Policy Optimization ( TRPO ) ( Schulman et al. , 2015 ) , Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , and Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) , conventional regularization methods were not considered . Even in popular codebases such as the OpenAI Baseline ( Dhariwal et al. , 2017 ) , L2 regularization and dropout were not incorporated . Instead , the most commonly used regularization in the RL community , is an “ entropy regularization ” term that penalizes the high-certainty output from the policy network , to encourage more exploration during the training process and prevent the agent from overfitting to certain actions . The entropy regularization was first introduced by Williams & Peng ( 1991 ) and now used by many contemporary algorithms ( Mnih et al. , 2016 ; Schulman et al. , 2017 ; Teh et al. , 2017 ; Farebrother et al. , 2018 ) . In this work , we take an empirical approach to questioning the conventional wisdom of not using common regularizations . We study agent ’ s performance on the current task ( the environment which the agent is trained on ) , rather than its generalization ability to different environments as many recent works ( Zhang et al. , 2018a ; Zhao et al. , 2019 ; Farebrother et al. , 2018 ; Cobbe et al. , 2018 ) . We specifically focus our study on policy optimization methods , which are becoming increasingly popular and have achieved top performance on various tasks . We evaluate four popular policy optimization algorithms , namely SAC , PPO , TRPO , and the synchronous version of Advantage Actor Critic ( A2C ) , on multiple continuous control tasks . A variety of conventional regularization techniques are considered , including L2/L1 weight regularization , dropout , weight clipping ( Arjovsky et al. , 2017 ) and Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) . We compare the performance of these regularization techniques to that without regularization , as well as the entropy regularization . Surprisingly , even though the training and testing environments are the same , we find that many of the conventional regularization techniques , when imposed to the policy networks , can still bring up the performance , sometimes significantly . Among those regularizers , L2 regularization , perhaps the most simple one , tends to be the most effective for all algorithms and generally outperforms entropy regularization . L1 regularization and weight clipping can boost performance in many cases . Dropout and Batch Normalization tend to bring improvements only on off-policy algorithms . Additionally , all regularization methods tend to be more effective on more difficult tasks . We also verify our findings with a wide range of training hyperparameters and network sizes , and the result suggests find that imposing proper regularization can sometimes save the effort of tuning other training hyperparameters . Finally , we study which part of the policy optimization system should be regularized , and conclude that generally only regularizing the policy network suffices , as imposing regularization on value networks usually does not help . Our results also show that neural network training techniques such as regularization , can be as important as high-level reinforcement learning algorithms in terms of boosting performance . Our main contributions can be summarized as follows : • We provide the first comprehensive study of common regularization methods in policy optimization algorithms , which have been largely ignored in the RL literature . • We find conventional regularizations can often be very effective in improving the performance on continuous control tasks , espcially on harder ones . Remarkably , the most simpleL2 regularization generally performs better than the more widely used entropy regularization . BN and dropout can only help in off-policy algorithms . • We experiment with multiple randomly sampled training hyperparameters for each algorithm and confirm our findings still hold . The result also suggests that proper regularization can sometimes ease the hyperparameter tuning process . • We study which part of the network ( s ) should be regularized . The key lesson is to regularize the policy network but not the value network . 2 RELATED WORKS . Regularization in Deep RL . Conventional regularization methods have rarely been applied in deep RL . One rare case of such use is in Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2016 ) , where Batch Normalization is applied to all layers of the actor µ network and layers of the critic Q network prior to the action input , and L2 regularization is applied to the critic Q network because it tends to have overestimation bias ( Fujimoto et al. , 2018 ) . Some recent studies have developed more complicated regularization approaches to continuous control tasks . Cheng et al . ( 2019 ) regularizes the stochastic action distribution π ( a|s ) using a suboptimal control prior . The regularization weight at a given state is adjusted based on the temporal difference ( TD ) error . The larger the TD error , the more the action distribution moves towards the prior . Galashov et al . ( 2019 ) introduces a default policy that receives limited information as a regularizer . The information asymmetry between the behavior policy and the default policy helps to accelerate convergence and improve performance . Parisi et al . ( 2019 ) introduces TD error regularization to penalize inaccurate value estimation and Generalized Advantage Estimation ( GAE ) ( Schulman et al. , 2016 ) regularization to penalize GAE variance . However , most of these regularizations are rather complicated ( Galashov et al. , 2019 ) , specifically designed for certain algorithms ( Parisi et al. , 2019 ) , or need prior information ( Cheng et al. , 2019 ) . Also , these techniques consider regularizing the output of the network , while conventional regularization methods mostly directly regularize the parameters . In this work , we focus on studying these simpler but under-utilized regularization methods . Generalization in Deep RL typically refers to how the model perform in a different environment from the one it is trained on . The generalization gap can come from different modes/levels/difficulties of a game ( Farebrother et al. , 2018 ) , simulation vs. real world ( Tobin et al. , 2017 ) , parameter variations ( Pattanaik et al. , 2018 ) , or different random seeds in environment generation ( Zhang et al. , 2018b ) . There are a number of methods designed to address this issue , e.g. , through training the agent over multiple domains/tasks ( Tobin et al. , 2017 ; Rajeswaran et al. , 2017 ) , adversarial training ( Tobin et al. , 2017 ) , designing model architectures ( Srouji et al. , 2018 ) , adaptive training ( Duan et al. , 2016 ) , etc . Meta Reinforcement Learning ( Finn et al. , 2017 ; Gupta et al. , 2018 ; Al-Shedivat et al. , 2017 ) try to learn generalizable agents by training on a set of environments drawn from the same family/distribution . There are also some comprehensive studies on RL generalization with interesting findings ( Zhang et al. , 2018a ; b ; Zhao et al. , 2019 ; Packer et al. , 2018 ) , e.g. , algorithms performing better on the training environment could perform worse under domain shift ( Zhao et al. , 2019 ) . Recently , several studies have investigated conventional regularization ’ s effect on generalization . Farebrother et al . ( 2018 ) shows that in Deep Q-Networks ( DQN ) , L2 regularization and dropout can sometimes bring benefit when evaluated on the same Atari game with mode and difficulty variations . Cobbe et al . ( 2018 ) shows that L2 regularization , dropout , data augmentation , and Batch Normalization can improve generalization performance , but to a less extent than entropy regularization and -greedy action selection , when evaluated with ( Espeholt et al. , 2018 ) . Different from those studies , we focus on regularization ’ s effect in the same environment , a more direct goal compared with generalization , yet on which conventional regularizations are under-explored . 3 REGULARIZATION METHODS . There are in general two kinds of common approaches for imposing regularization , one is to discouraging complex models ( e.g. , weight regularization , weight clipping ) , and the other is to inject certain kind of noise in the activations ( e.g. , dropout and Batch Normalization ) . Here we briefly introduce those regularization methods we investigate in our experiments . L2 / L1 Weight Regularization . Large weights are usually believed to be a sign of overfitting to the training data , since the function it represents tend to be complex . One can encourage small weights by adding a loss term penalizing the norm of the weight vector . Suppose L is the original empirical loss we want to minimize . SGD updates the model on a mini-batch of training samples : θi ← θi− η · ∂L∂θi , where η is the learning rate . When applying L2 regularization , we add an additional L2-norm squared loss term 12λ||θ|| 2 2 to the training objective . Thus the SGD step becomes θi ← θi − η ∂L∂θi − η · λ · θi . Similarly , in the case of L1 weight regularization , the additional loss term is λ||θ||1 , and the SGD step becomes θi ← θi − η · ∂L∂θi − η · λ · sign ( θi ) . Weight Clipping . Weight clipping is an extremely simple idea : after each gradient update step , each individual weight is clipped to range [ −c , c ] , where c is a hyperparameter . This could be formally described as θi ← max ( min ( θi , c ) , −c ) . In Wasserstein GANs ( Arjovsky et al. , 2017 ) , weight clipping is used to satisfy the constraint of Lipschitz continuity . This plays an important role in stabilizing the training of GANs ( Goodfellow et al. , 2014 ) , which were notoriously hard to train and often suffered from “ mode collapse ” before . Weight clipping could also be seen as a regularizor since it drastically reduce the complexity of the model space , by preventing any weight ’ s magnitude from being larger than c. Dropout . Dropout ( Srivastava et al. , 2014 ) is one of the most successful regularization techniques developed specifically for neural networks . The idea is to randomly deactivate a certain percentage of neurons during training ; during testing , a rescaling operation is taken to ensure the scale of the activations is the same as training . One explanation for its effectiveness in reducing overfitting is they can prevent “ co-adaptation ” of neurons . Another explanation is that dropout acts as a implicit model ensemble method , because during training a different model is sampled to fit each mini-batch of data . Batch Normalization . Batch Normalization ( BN ) ( Ioffe & Szegedy , 2015 ) is invented to address the problem of “ internal covariate shift ” , and it does the following transformation : ẑ = zin−µB√ σ2B+ ; zout = γẑ + β , where µB and σB are the mean and standard deviation values of input activations over B , γ and β are trainable affine transformation parameters ( scale and shift ) which provides the possibility of linearly transforming normalized activations back to any scales . BN turns out to be able to greatly accelerate the convergence and bring up the accuracy . It has become a standard component , especially in convolutional networks . BN also “ acts as a regularizer ” ( Ioffe & Szegedy , 2015 ) : since the statistics µB and σB are dependent on the current batch , BN subtracts and divides different values in each iteration . This randomness can encourage subsequent layers to be robust to such variation of input . Entropy Regularization . In a policy optimization framework , the policy network is used to model a conditional distribution over actions , and entropy regularization is widely used to prevent the learned policy from overfitting to one or some of the actions . More specifically , in each step , the output distribution of the policy network is penalized to have a high entropy . Policy entropy is calculated at each step as Hsi = −Eai∼π ( ai|si ) log π ( ai|si ) , where ( si , ai ) is the state-action pair . Then the per-sample entropy is averaged within the batch of state-action pairs to get the regularization term LH = 1 N ∑ si Hsi . A coefficient λ is also needed , and λL H is added to the policy objective J ( θ ) to be maximized during policy updates . Entropy regularization also encourages exploration due to increased stochasticity in actions , leading to better performance in the long run .

Summary: This paper investigates the use of conventional regularizers for neural networks in the reinforcement learning setting. Contrary to the standard practice of foregoing regularizers in deep RL, the paper finds that their addition can improve the performance of policy gradient algorithms on a standard suite of continuous control tasks. Various regularizers are tried, including l2/l1 regularization, entropy regularization and dropout in a combination with a few standard deep RL algorithms such as TRPO, PPO and SAC. Other experiments also verify the impact of these regularizers on the sensitivity of other hyperparameters and whether regularization should be applied to the value or policy networks.