Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The main contribution of this paper is a normalization scheme to avoid saturating the squashing function typically used to constrain actions within a bounded range in continuous control problems. It is argued that algorithms like DDPG and TD3 suffer from such saturation, which prevents proper exploration during training, while maximum entropy algorithms like Soft Actor-Critic (SAC) avoid it thanks to their entropy bonus. The main reason behind the success of SAC would thus be its ability to keep exploring throughout training, by avoiding saturation. A second contribution is a new experience replay sampling scheme, named Emphasizing Recent Experience (ERE), based on the idea that most recently added transitions should be given higher weights when sampling mini-batches from the replay bufffer. Combining both ideas leads to the SOP (Streamlined Off-Policy)+ERE algorithm, which is shown to consistently outperform SAC on Mujoco tasks.

Although this paper presents interesting insights and very good empirical results, I am currently leaning towards rejection mostly due to missing some important empirical comparisons, which hopefully can be added in a revised version.

The first key missing comparison (IMO) is to the Inverting Gradients approach from Hausknecht & Stone (2016), which the authors know about since it is cited in the related work section. Note that in that paper, the problem of saturating squashing functions preventing proper exploration was already mentioned, although not investigated in as much depth as in this submission («(…) squashing functions quickly became saturated. The resulting agents take the same discrete action with the same maximum/minimum parameters each timestep »). Their proposed Inverting Gradients technique was found to work significantly better than squashing functions, which is why I believe it should be an obvious baseline to compare to.

The other important experiments which I think need to be added are simply to implement the proposed normalization scheme within DDPG & TD3 to demonstrate its usefulness as a standalone improvement over existing algorithms. This would strengthen the claim that « algorithms such as DDPG and TD3 based on the standard objective with additive noise exploration can be greatly impaired by squashing exploration ». Without this comparison on the same benchmark, it is difficult to fully grasp the impact of this normalization.

Finally, regarding the ERE sampling scheme, I would appreciate to see SAC+ERE as well, to (hopefully) show that it can benefit SAC too (since this second contribution is orthogonal to the SOP algorithm).

Minor points:
•	I would tone down a bit the claims for « the need to revisit the benefits of entropy maximization in DRL », since better exploration has always been put forward as a major benefit (« the maximum entropy formulation provides a substantial improvement in exploration and robustness », as written in « Soft Actor-Critic Algorithms and Applications »). To me, what this submission shows is essentially that naive implementation of additive noise exploration in e.g. DDPG is very bad for exploration, more than uncovering some novel properties of SAC.
•	Below eq. 1: « the optimal policy is deterministic » => should be replaced with « there exists an optimal policy that is deterministic »
•	« principle contribution » => principal
•	The normalization scheme does not appear in Alg. 1
•	In Alg. 1 there are a Q_phi,i and a Q_phi,1 that should probably be Q_phi_i and Q_phi_1
•	The results from section E in the Appendix should be mentioned in the main text
•	In Fig. 4f the y axis’ label is a bit clipped

Update based on new revision: thank you for adding more results. From what I can see, it is difficult to conclude on the benefits of SOP over IG, which I find really problematic. It seems to me that the most impactful result is related to the improvements brought by the ERE sampling scheme, which could probably be worth a paper on its own (by showing its benefits over a wider range of algorithms, e.g. TD3 & DQN+variants), but this would be a different paper. As a result I am sticking to "Weak Reject".
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 
# Summary
The paper identifies a problem with TD3 related to action clipping. The authors notice that SAC alleviates this problem by means of entropy regularization. Given the insight that action clipping is crucial, the authors propose an alternative approach to avoid action clipping in TD3, which is empirically shown to yield the same results as SAC. Surprisingly, with this improvement, even several parts from TD3 can be removed, such as delayed policy updates and the target policy parameters. In addition, a straightforward-to-implement experience replay scheme is proposed, that emphasizes recent experiences, which propels the proposed algorithm to achieve state-of-the-art results on MuJoCo.

# Decision
This is a great paper: accept. The proposed Streamlined Off Policy (SOP) algorithm is thoroughly evaluated, ablation studies performed, code made available. Nevertheless, there are a few suggestions below that may further improve the paper.

# Suggestions
1) It is said that entropy regularization leads to action not being saturated in SAC. I feel that this causal relation is very indirect. Maybe SAC with entropy just discovers better policies that do not go crazy between extremes? For example, if you would leave the entropy term but remove tanh saturation from SAC, don't you think you would also get a bang-bang policy? Adding such an ablation study could further strengthen the argument that entropy leads to no constraint violation, if it turns out true.

2) The Emphasizing Recent Experience (ERE) replay scheme seems reminiscent of sampling according to a distribution exponentially decaying into the past. It is said that physically shrinking the allowed sampling range by dropping old experiences is better because then very old experiences cannot be used at all. It would be interesting to see a comparison to sampling according to exponential distribution from the replay queue.

# AFTER REBUTTAL
Taking into account the concerns of other reviewers and the newly added evaluations, I lower my score to weak accept. Since now it seems that ERE is quite crucial, the argument of SPO outperforming SAC becomes weaker. Therefore, the authors should tone down the claims of outperforming SAC. Nevertheless, I still find the contribution of the paper valuable and think that it should be accepted, albeit with the aforementioned modifications in the camera-ready version.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors investigate the role of entropy maximization in SAC and show that entropy regularization does not do what is usually thought: in the examples they investigate, where the output of the policy network needs to be squashed to fit in the action space domain, squashing would result in having only action at the boundaries, but entropy regularization maintains some intermediate values, hence exploration. From this insight, the authors replace entropy regularization by a simpler normalization process and show equivalent performance with their simpler Streamlined Off-Policy (SOP) algorithm. Then they introduce a second "Emphasizing Recent Experience" mechanism and show that SOP+ERE performs better than SAC.

A good point for the paper is that the entropy regularization  study is very nice, more papers in the field should show similar detailed analyses of internal processes. But the paper suffers from a few serious weaknesses:

- The TD3 mechanism goes beyond the Double Q-learning (or DDQN) mechanism of Van Hasselt et al: it takes the min over two critics. This should be explained properly.
- the title, abstract and introduction insist more on SOP, but performance improvement seem to result more from ERE. If this is possible, studying the performance of SAC + ERE would disambiguate the relative contribution of both mechanisms.

About gradient squashing issues, the authors main mention de gradient inverter idea from this paper:

@article{hausknecht2015deep,
  title={Deep reinforcement learning in parameterized action space},
  author={Hausknecht, Matthew and Stone, Peter},
  journal={arXiv preprint arXiv:1511.04143},
  year={2015}
}

The authors should also probably also cite (and read the latest arxiv version of):
@inproceedings{ahmed2019understanding,
  title={Understanding the impact of entropy on policy optimization},
  author={Ahmed, Zafarali and Le Roux, Nicolas and Norouzi, Mohammad and Schuurmans, Dale},
  booktitle={International Conference on Machine Learning},
  pages={151--160},
  year={2019}
}


More local points:
- "without performing a careful hyper-parameter search": so how did you choose these hyper-parameters? I see what you mean, but this is a very vague and slippery statement.
- I do not find the 23*4 images in Appendix B much useful
- Fig 3 seems to be repeated in Fig 4. Can't you just remove Fig 3?
review_title: Official Blind Review #3

Meta-review: The paper studies the role of entropy in maximum entropy RL, particularly in soft actor-critic, and proposes an action normalization scheme that leads to a new algorithm, called Streamlined Off-Policy (SOP), that does not maximize entropy, but retains or exceeds the performance of SAC. Independently from SOP, the paper also introduces Emphasizing Recent Experience (ERE) that samples minibatches from the replay buffer by prioritizing the most recent samples. After rounds of discussion and a revised version with added experiments, the reviewers viewed ERE as the main contribution, while had doubts regarding the claimed benefits of SOP. However, the paper is currently structured around SOP, and the effectiveness of ERE, which can be applied to any off-policy algorithm, is not properly studied. Therefore, I recommend rejection, but encourage the authors to revisit the work with an emphasis on ERE.
Decision: Reject