Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This work introduces a particular parametrization of a stochastic policy (a uniform mixture of deterministic policies). They find this parametrization, when trained with stochastic value gradient outperforms DDPG on several OpenAI gym benchmarks.

This paper unfortunately misses many significant pieces of prior work training stochastic policies. The most relevant is [1] which should definitely be cited. The algorithm here can be seen as SVG(0) with a particular parametrization of the policy. However, numerous other works have examined stochastic policies including [2] (A3C which also used the Torcs environment) and [3].

The wide use of stochastic policies in prior work makes the introductory explanation of the potential benefits for stochastic policies distracting, instead the focus should be on the particular choice and benefits of the particular stochastic parametrization chosen here and the choice of stochastic value gradient as a training method (as opposed to many on-policy methods).

The empirical comparison is also hampered by only comparing with DDPG, there are numerous stochastic policy algorithms that have been compared on these environments. Additionally, the DDPG performance here is lower for several environments than the results reported in Henderson et al. 2017 (cited in the paper, table 2 here, table 3 Henderson) which should be explained.

While this particular parametrization may provide some benefits, the lack of engagement with relevant prior work and other stochastic baselines significant limits the impact of this work and makes assessing its significance difficult.

This work would benefit from careful copyediting.

[1] Heess, N., Wayne, G., Silver, D., Lillicrap, T., Erez, T., & Tassa, Y. (2015). Learning continuous control policies by stochastic value gradients. In Advances in Neural Information Processing Systems (pp. 2944-2952).

[2] Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., ... & Kavukcuoglu, K. (2016, June). Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning (pp. 1928-1937).

[3] Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.


review_title: Lack of engagement with relevant prior work

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors investigate a simple method for improving the performance of networks trained with DDPG: instead of outputting a single action, they output several actions (through distinct output layers), and choose one uniformly at random. The selected action is updated using deterministic policy gradient. The critic Q is updated with a Bellman backup where the the choice of the action is marginalized out. Authors show improved performance on a large number of standard continuous control environment (openAI gym and TORCS).

The paper is well written, and the idea seems to work perhaps surprisingly well. The authors do a good job of investigating the behavior of their algorithm (in particular the increase of standard deviation in states where multiple optimal actions exist). 

Similar ideas (mixture of gaussians for action distribution in policy gradient setups, or multi-modal action distribution through the use of latent variables) are often difficult to make work - I am curious why this particular method works so well.
In particular, it would be interesting to investigate how the algorithm avoids collapsing all actions into the same one; as implied by section 3.2, in a state with multiple optimal actions, there is no difference in loss between having all actions be nearly identical (and optimal), and all actions being distinct optimal actions. Furthermore, as the loss does not encourage diversity, once two actions are set to be similar in a state, intuitively it would be hard for the actions to become distinct again. Imagine for instance the cart pole problem with M=2. If both action layers start with the same 'tendency' (towards clock-wise or counter clock-wise motion), it is likely that the same tendency would be reinforced for both, and the network with M=2 would end up having a similar behavior to a classical network with M=1.

Is this problem avoided by using a large value of M? It would be interesting to investigate the behavior of the algorithm in a toy environment (perhaps a simple 2d navigation  task with distinct 'paths' with same cost) where the number of distinct basins of optimality is know for various states, and investigate in more details how diversity is maintained (perhaps as a function of M).


Minor: 
- typo rho -> $\rho$
- Given the paper fits comfortably within the page limit, it would have been worthwhile to give mathematical details to the Algorithm 1 box (even if they are easy to find in text or appropriate references)
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper describes an approach to stochastic control using RL that extends DDPG with a stochastic policy.  A standard DDPG setup is extended such that the actor now produces M actions at each timestep.  Only one of the M actions will be executed in the environment using a uniform sampling.  The sampled action is the only that will receive a gradient update from the critic network. The authors demonstrate that such a stochastic policy performs better on average in a series of benchmark control tasks.

I find the general idea of the work compelling, but the particular approach is rather poor.  The fact that we are choosing the number of modes in the uniform distribution is a bit underwhelming (a more compelling architecture could have proposed a policy conditioned on gaussian noise for example, thus having better coverage of the distribution). I found the proposed apprach to be under-analyzed and the stochastic aspects of the policy are undervalued.   The main claim being argued in the paper is that the proposed stochastic policy has better final performance on average than a deterministic policy, but the only practical difference seems to be a slightly more structured approach to exploration.  
However, very little attention is paid to trying different exploration methods with the deterministic policy (incidentally, Ornstein-Uhlenbeck process noise is not something I'm familiar with, a citation to the use of this noise for exploration as well as a more explicit explanation would have been appreciated).  One interpretation is that each of the M sub-policies follows a different mode of the Q-value distribution over the action space.  But is this indeed the case?  There is a brief analysis of this with cartpole, but a more complete look at how actions are clustered in the action space would make this paper much more compelling.  Even in higher-dimensional action spaces, you could look at a t-SNE projection or cluster analysis to try and see how many modes the agent is reasoning over.  Additionally, the baseline agent should have used additional exploration methods as these can quickly change the performance of the agent.

I also think that better learning is not the only redeeming aspect of a stochastic policy.  In the face of a non-stationary environment, a stochastic policy will likely be much more robust.  Additionally, it will have much better performance against adversarial environments.  Given the remaining available space in the paper it would have been interesting to provide more insight into the proposed methods gains in these areas.
review_title: Interesting idea but insufficient analysis.

Meta-review: All of the reviewers agree that the paper presents strong experimental results on continuous control benchmarks. The reviewers raised concerns regarding the analysis of the behavior of the algorithm, the possible impact of the technique, and requested more references and comparison with related work. The paper has significantly improved since the initial submission, but still not able fully satisfactory to the reviewers, partly due to the large extent of the changes needed.

Decision: Reject