Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper conducts a comprehensive study on the effect of different regularization on Deep RL algorithms. Regularization has been mostly neglected in RL as most benefits were believed to be in generalization to unseen test environments in supervised learning settings. However, this paper shows that regularization does provide benefit even though training/testing is done on the same environment in deep RL settings. 
The paper studies L1/L2 regularization, dropout, weight clipping, and batch normalization on four different deep RL policy optimization algorithms. Results show that regularization improves performance especially L2, that regularization brings more benefit on harder tasks that have higher sample complexity, and that it makes algorithms more robust to training hyperparameter variations. The paper conducts rigorous statistical significance test, and analyzes the benefit of regularization through four ways: sample complexity, reward distribution, weight norm, and training noise robustness.

Overall, I think the paper is well-written giving a comprehensive evaluation on a widely neglected area in reinforcement learning. Many different RL algorithm implementations have used regularization with and without acknowledging its use in the paper, and this paper sheds light that using regularization in deep RL algorithms does have significant impact and warrants further study. 

I wish the paper conducted more than 5 runs (it is shown that mujoco environments have high variance due to random seeds and that same alg. performs significantly differently based on different groups of random seeds -- see Henderson et al. 2018 (https://arxiv.org/pdf/1709.06560.pdf)), but the authors also perform significance testing to validate the performance improvements.

I think the paper's analysis section can be further improved. For example, the bar chart in Figure 3 does not indicate how many runs it has done, or show any error bars to show statistical significance. And I think 'return' would be the correct terminology instead of 'reward' to indicate cumulative sum of rewards (For Figure 2, 3, and Table 5)

Also, I think the claim that 'BN and dropout work only with off-policy algorithm', or 'BN and dropout can only help in off-policy algorithm' is quite strong. Only a single off-policy algorithm SAC has been tested, and although BN and dropout helped a lot, its improvement may have been due to algorithm-specific properties of SAC. The authors hypothesize plausible reasons, but I think the findings only show that BN and dropout does not work well for on-policy algorithms.

There is a minor typo in page 4: decompled -> decoupled
review_title: Good benchmark paper that comprehensively evaluates the impact of regularization

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work empirically studies the widely used regularization techniques for training deep neural networks, such as $L_2$/$L_1$ regularizer, Batch Normalization (BN), Weight Clip, and Dropout, in policy optimization algorithms (A2C, SAC, TRPO, PPO). The experimental results demonstrate that these Deep Learning (DL) regularizations actually can help policy optimization.

Pros:
1. The combination of DL regularizations and Reinforcement Learning (RL) algorithms seems to be a reasonable and under-explored idea. The motivation is convincing. 
2. The authors conducted substantive experiments, which I appreciate.
3. The work is presented clearly, and the paper is well written.

Cons:
1. The explanations for why these DL regularizers work or not are hand-waving. As an empirical study paper, I understand theory is not the main focus. But since this paper focused on policy optimization, I expected some insights or explanations from RL perspectives, which are important to guide future research, but they are not provided here (or I was missing something).

    (1a) The DL regularizers studied in this paper have proved to help training neural networks. As neural networks are used as function approximations in RL, it is as expected sometimes they should have some improvements.

    (1b) The main reason the authors claimed for why some DL regularizers work is from the generalization perspective, which makes sense in DL. However, for policy optimization, more explanations are needed from the perspectives of learning better agents (e.g., exploration vs. exploitation, and better objective landscape), which make more sense in RL. An interpretation from an RL perspective is lacking in this paper, which seems necessary since policy optimization is the main topic of this paper.

2. Experimental results are not enough to provide useful conclusions. Since the main focus is on the empirical side, I would expect more on this part, but it seems some conclusions have been made in this paper without sufficient investigations.

    (2a) The comparison of DL regularizers with entropy regularization actually does not seem reasonable to me. 

    First, the entropy regularization is provable to increase exploration (see [1] to the end) and help convergence in policy optimization (see [2,3]), which is not claimed to help generalization. Second, DL regularizers help generalization as claimed in the paper. Therefore, they help agent learning in different ways, and I did not see the reason to compare them and what we can conclude from the results.

    (2b) The conclusion that DL regularizers do not work very well for value function (comparing with policy optimization) is lack of support.

    There is a number of regularizers/tricks of training in value functions (e.g., replay buffer, multi-step roll-out, distributional RL, double-Q, etc, see [4]). The authors did not do experiments (or did not mention) using those well-known ideas in RL and made this conclusion, which seems hasty to me.

    (2c) The conclusion and explanation that BN does not work for on-policy methods and works better for off-policy methods seem quite interesting. But also the study here is not enough. There is an amount of RL techniques for off-policy training (e.g., corrections, see [5]). I would suggest more investigation and deeper explanation than the discussion of the paper in this direction.

Overall, the idea of using DL regularizers in RL seems reasonable and the experimental results look promising. However, the theory part is not solid and insightful, and some of the conclusions are lack support.

References:
    [1] "Making sense of reinforcement learning and probabilistic inference", O’Donoghue et al.
    [2] "Understanding the impact of entropy on policy optimization", Ahmed et al.
    [3] "On the global convergence rates of softmax policy gradient methods", Mei et al.
    [4] "Rainbow: Combining Improvements in Deep Reinforcement Learning", Hessel et al.
    [5] "Safe and Efficient Off-Policy Reinforcement Learning", Munos et al.


======Update======
Thank you for the rebuttal, which resolved most of my concerns. I increased my score. 
review_title: good experimental investigation but lack of insights

final_score: rating: 7: Good paper, accept
review_content: The paper studies how different regularizations affect common RL algorithms in continuous control tasks. 

The paper is very well written and organized. The authors clearly state the scope of the paper and its placement with respect to RL literature.
Despite having no theoretical novelty, the paper has a good empirical contribution by being the first comprehensive study of the subject.
I appreciate the comments and explanations the authors give about the results, and I believe that the findings of this paper can help the RL community in being more aware of the importance of regularization, which is often overlooked. This is crucial especially when it comes to reimplementing existing algorithms without relying too much on handtuned hyperparameters.

Overall, I find this paper to be a good empirical study and I am leaning to accept it. However, my major concern is the number of seeds per experiment.
As shown by Henderson et al. ("Deep Reinforcement Learning that Matters"), 5 seeds are definitely not enough to get accurate statistical results out of RL experiments. I understand that the amount of total experiments is extremely large, given all the different hyperparameters you are testing, but you should have aimed for at least 10 seeds, especially because the paper is based on empirical contributions.

As a further suggestion, I invite you to include DDPG, maybe in the final version of the paper. First, because it is another off-policy algorithm (you have 3 on-policy and 1 off-policy). Second, because (as you also wrote) DDPG was one of the few algorithms to include BN, while TRPO, PPO and SAC rely on more sophisticated regularizations (entropy, KL, or surrogate). This could lead to very interesting results.

Another suggestion (for future work, as it may be a bit out of scope) is to test all algorithms on sparse-reward environments. In this case, it is known that common algorithms either prematurely convergence to local optima, or do not learn anything at all. Some of the regularizations (eg, entropy) should be beneficial in this case, but an extensive study is still missing.

** EDIT **
The authors have addressed my concernes and I have increased my score.
review_title: Good empirical study

final_score: rating: 7: Good paper, accept
review_content: **Pros**

The paper investigates a worthwhile question. Given the prevalence and success of regularization in deep learning, it is curious why we haven't consistently observed the same in deep RL. 

**Cons**

My main concern with the work, especially since it is an empirical study, concerned the lack of statistical significance in the results. Here are the problems I found.

1. Welch's t-test requires normality assumption, and I'm not sure why returns would be normally distributed; in any case, no verification of this assumption was attempted. 
2. Even if the normality issue is ignored, low p-values in the t-test suggests that the means are different, but doesn't necessarily say anything about the direction or magnitude of difference. It could be that regularisation worsens performance, for all we know. 
3. The learning curves are uninformative for determining the direction and magnitude of any performance difference, given the high variance of RL and the number of seeds used (5 seeds). Indeed, in the Henderson et al. work cited, Figure 5 shows how a small number of random seeds can be misleading (they even show that for the same algorithm and different seeds, a t-test can give a small p-value). 
4. Similar concerns apply for ranking metrics, given the small sample size. 

**Summary**

Based on the concerns with statistical significance I have highlighted above, I recommend rejection of this paper. The results presented in the paper are certainly suggestive, but do not I think meet the level of rigour required at ICLR. I do very much like the motivation of the study, and would have liked to have seen results of greater statistical significance. 

I recognize that the results presented in the text already took 57 days on a substantial computational setup, but it worries me that we are using massive computational power to little avail if we cannot produce reliable results. It would perhaps be better to focus on a smaller subset of algorithms and environments in order to guarantee reliable results.

** Edit after author response **
I feel that the authors have sufficiently addressed the statistical significance concerns I had in the comments below. I now recommend acceptance because I think the work provides strong evidence that regularization is beneficial in continuous control. Although, as other reviewers have pointed out, further analysis of why regularization is beneficial--especially from a theoretical standpoint--would be helpful, I think the empirical contribution of the paper still stands. 
review_title: Statistical significance is an issue

Meta-review: The reviewers all  appreciated the insights drawn from this study as well at its thoroughness. I want to commend both authors for running additional experiments to strengthen the paper and reviewers for updating the scores accordingly.

Congratulations.
Decision: Accept (Spotlight)