Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper investigates the use of conventional regularizers for neural networks in the reinforcement learning setting. Contrary to the standard practice of foregoing regularizers in deep RL, the paper finds that their addition can improve the performance of policy gradient algorithms on a standard suite of continuous control tasks. Various regularizers are tried, including l2/l1 regularization, entropy regularization and dropout in a combination with a few standard deep RL algorithms such as TRPO, PPO and SAC. Other experiments also verify the impact of these regularizers on the sensitivity of other hyperparameters and whether regularization should be applied to the value or policy networks. 

Overall, I find this paper to be a solid empirical study of regularization in deep reinforcement learning. The experiments are thorough, with various aspects being examined in more detail. I find several of the findings interesting, such as the importance of regularizing solely the policy network and that batch norm/dropout are effective for off-policy methods but not on-policy ones. There were certain points which warranted some further clarification. 

I would be willing to increase my score based on the authors' response to the following points:
1) In section 6, the last two sentences ("For A2C, TRPO, and PPO ... so further regularization is unnecessary.") are unclear to me.
	- "rewards are already normalized using running mean filters." I thought that rewards are also normalized for SAC, so I'm not sure how this could explain the difference between the on-policy algorithms and the off-policy ones.
	- "mitigates the overestimation bias...further regularization is unncessary." Could you clarify the connection between regularization and overestimation bias? Related to this point, in section 2 of the paper, it is written that "L2 regularization is applied to the critic Q network because it tends to have overestimation bias (Fujimoto et al., 2018)" but I was not able to find such an explanation in the cited paper though I may have missed it.

2) In section 7, in the paragraph on BN/Dropout, could you clarify the point starting from "1) For both BN and dropout,..."? In particular, which discrepancy between the sampling policy and the optimization policy is being referred to here? 

3) Did you consider trying weight decay ("Fixing Weight Decay Regularization in Adam", Loschilov et al. 2018) as a regularizer? Given the success of L2 regularization, it could be possible that weight decay is even more effective.

4) For the hyperparameter sensitivity plots, where one hyperparameter is varied at a time, why are the step sizes for the policy and value networks not included in these experiments? They are usually a critical hyperparameter.


Minor comments and typos:
- On p.5, when defining "hurting", perhaps it could be better to choose a looser definition such as "\mu_r < \mu_b" or "\mu_r - \sigma_r < \mu_b - \sigma_b". This way, there could be a larger distinction between the most effective methods. Currently, both l2 and entropy regularization achieve 0.0% and the next best two regularizers are also under 10%. 
- In abstract: "regularizing the policy network is typically enough." Rephrase perhaps? The experiments seem to show that applying a regularizer to only the policy network is better than on both.
- In abstract: "large improvement" -> "large improvements"
- p.2, par. 2: "those regularizations" -> "those regularizers"
- p.3, Weight Clipping: "This greatly stablizes" -> "This greatly stabilizes". This sentence could be rephrased since "This" seems to refer to only weight clipping, but is not the only change in WGANs. 
- p.3, Dropout: "regularization technique" -> "regularization techniques"
- p.4, par. 1: "due to more stochasticity" -> "due to increased stochasticity"
- p.4, 2nd to last par.: "during policy update" -> "during policy updates"
- p.5, 2nd to last par.: "sometimes help" -> "sometimes helps", "easier ones baseline is" -> "easier ones the baseline is"
- p.8, 2nd to last par.: "it naturally accepts" -> "they naturally accept", "been shown effective" -> "been shown to be effective"
- p.8, last sentence: "policy network without the value network." -> "policy network but not the value network."



review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper provides an empirical study of regularization in policy optimization methods in multiple continuous control tasks. The paper focuses on the effect of conventional regularization on performance in training environments, not generalization ability to different (but similar) testing environments. Their findings suggest that L2 and entropy regularization can improve the performance, be robust to hyperparameters on the tasks studied in the paper. 

Overall, the paper is well written. However, I am leaning to reject this paper because (1) the experimental finding is not well justified (2) the experiments are missing some details and do not provide convincing evidence.  

First, the paper does not well justify why regularization methods improve performance in training environments. One potential reason is discussed in Section 7: regularization can improve generalization to unseen samples. However, the improvement can simply due to better hyperparemer optimization. When we introduce more hyperparemers and computation compared to baselines, it’s not surprising to see a better performance, especially in deep RL where using a different seed or using a different implementation can have significant difference in performance [1]. Moreover, it is unclear that inability to generalize to unseen samples is a problem in the continuous control tasks evaluated in the paper. I think the paper should demonstrate that this is indeed a problem. If it is not a problem, why would you expect regularization to help?

There are some missing details which makes it difficult to draw conclusion:
1. How was \sigma_{env,r} computed? Is it the standard error of the mean return, or the standard deviation of the return? 
2. What does the average rank mean (in Table 2 and 3)? the average ranking over 5 seeds and all environments? If so, does it make sense to compare these numbers? e.g. Algorithm A with rank 1, 1, 7, 7 and Algorithm B with rank 4, 4, 4, 4 have the same average rank, but totally different performance. 
3. The experiment in Figure 3 seems very interesting, however, what’s the conclusion here? 
4. Why do you use difference hyperparamer ranges (lambda for L2, L1 and entropy regularization) for different algorithms in appendix A? 

Minor comment which does not impact the score:
1. It would have been better if there’s a brief description of each algorithm (before section 4 or in appendix). 

[1] Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: An interesting paper on the role of regularization in policy optimization


In this paper, the authors study a set of existing direct policy optimization methods in the field of reinforcement learning. The authors provide a detailed investigation of the effect of regulations on the performance and behavior of agents following these methods.

The authors present that regularization methods mostly help to improve the agents' performance in terms of final scores. Specifically, they show that direct regularizations on model parameters, such as the standard case of L2 or L1 regularization, generally improve the agent performance. They also show that these regularizations, in their study, is more proper than entropy regularization. The authors also show that, in the presence of such regularizations, the learning algorithms become less sensitive to the hyperparameters. 

Few comments:
1) The paper is well written and easy to follow. I appreciate it. I found the writing of the paper has a bit of repetition. The authors might find it slightly more proper to remove some of the repetitions (e.g. section 4.2)

2) While I appreciate the clear writing and reasoning in this paper, I might suggest a slight change in the second paraphrase of the intro. I agree with the authors' reason on the first three lines, but I think it would be useful to also emphasize the role of the questions the researchers investigate to answer. I might also add one the main reason that the researchers in the field of DRL have spent less time on regulation or architecture search was their focus on more high-level algorithm design which is in the more immediate step of relevance and specialty to the field of reinforcement learning. 

3) I would suggest rephrasing the last two sentences of the second paragraph in related work: "Also, these techniques consider ...". Regularizing the output also regularizes the parameters, I think the authors' point was "directly regularize" the parameters. 

4) In the "Entropy Regularization" part of section 3, I guess the Hs has not been defined. 

5) Repeated "the" in the last paragraph of section 4.1 (despite it already incorporates the the maximization of)

6) The authors used the term "not converge" multiple times. While it is hard from the plots to see whether the series converges or not, I have a strong feeling that by this term the authors mean the algorithm does not converge to a resealable solution rather than being divergent up to a bandwidth. Maybe clarifying would be helpful.

7) In section 5, the authors study the sensitivity to the hyperparameters. In this section, I had a hard time to understand the role of term 3
"BN and dropout hurts on-policy algorithms but can bring improvement only for the off-policy SAC algorithm." Does it mean that deploying BN, results in a more sensitive algorithm? or it means that the performance degrades (which is a different topic than section 5 is supposed to serve)?

8) In section 7, the authors put out a hypothesis "
However, there is still generalization between samples: the agents are only trained on the limited" but the provided empirical study might not fully be considered to be designed to test this hypothesis. In order to test this hypothesis, the author might be interested in training the models with bigger sample sizes, more training iteration, different function classes, and more fitting in order to test this hypothesis.


9) Section 7 on "Why do BN and dropout work only with off-policy algorithms?" while I agree with the authors on their first reason which is quite commonly known, I might hesitate to make the second statement (2)



Generally, I found this paper an interesting paper and appreciate the authors for their careful empirical study. But I found the contribution of this work to be not significant enough. Most of the statements and claims in this paper are well know in the community, especially among deep learning practitioners. While I acknowledge the scientific value of this study, its concreteness, and appreciate the contribution of this paper, due to the low acceptance rate of this conference, I might be reluctant in accepting this paper. 

review_title: Official Blind Review #1

Meta-review: This paper proposes an analysis of regularization for policy optimization. While the multiple effects of regularization are well known in the statistics and optimization community, it is less the case in the RL community. This makes the novelty of the paper difficult to judge as it depends on the familiarity of RL researchers with the two aforementioned communities.

Besides the novelty aspect, which is debatable, reviewers had doubts on the significance of the results, and in particular on the metrics chosen (based on the rank). While defining a "best" algorithm is notoriously difficult, and could be considered outside of the scope of this paper, the fact is that the conclusions reached are still sensitive to that difficulty.

I thus regret to reject this paper as I feel not much more work is necessary to provide a compelling story. I encourage the authors to extend their choice of metrics to be more convincing in their conclusions.
Decision: Reject