Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces smoothed Q-values, defined as the value of drawing an action from a Gaussian distribution and following a given policy thereafter.  It demonstrates that this formulation can still be optimized with policy gradients, and in fact is able to dampen instability in this optimization using the KL-divergence from a previous policy, unlike preceding techniques.  Experiments are performed on an simple domain which nicely demonstrates its properties, as well as on continuous control problems, where the technique outperforms or is competitive with DDPG.

The paper is very clearly written and easy to read, and its contributions are easy to extract.  The appendix is quite necessary for the understanding of this paper, as all proofs do not fit in the main paper.  The inclusion of proof summaries in the main text would strengthen this aspect of the paper.

On the negative side, the paper fails to make a strong case for significant impact of this work; the solution to this, of course, is not overselling benefits, but instead having more to say about the approach or finding how to produce much better experimental results than the comparative techniques.  In other words, the slightly more stable optimization and slightly smaller hyperparameter search for this approach is unlikely to result in a large impact.

Overall, however, I found the paper interesting, readable, and the technique worth thinking about, so I recommend its acceptance.
review_title: The paper introduces smoothed Q-values, a tweak on standard Q-functions. It demonstrates some nice theoretical properties and reasonably successful experiments. The paper is interesting and correct, but unlikely to make a big impact.

final_score: rating: 6: Marginally above acceptance threshold
review_content: I think I should understand the gist of the paper, which is very interesting, where the action of \tilde Q(s,a) is drawn from a distribution. The author also explains in detail the relation with PGQ/Soft Q learning, and the recent paper "expected policy gradient" by Ciosek & Whiteson. All these seems very sound and interesting.

Weakness:
1. The major weakness is that throughout the paper, I do not see an algorithm formulation of the Smoothie algorithm, which is the major algorithmic contribution of the paper (I think the major contribution of the paper is on the algorithmic side instead of theoretical). Such representation style is highly discouraging and brings about un-necessary readability difficulties. 

2. Sec. 3.3 and 3.4 is a little bit abbreviated from the major focus of the paper, and I guess they are not very important and novel (just educational guess, because I can only guess what the whole algorithm Smoothie is). So I suggest moving them to the Appendix and make the major focus more narrowed down.
review_title: seems to be a good paper, however, I do not even see an exact algorithm formulation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper explores the idea of using policy gradients to learn a stochastic policy on complex control problems.  The central idea is to frame learning in terms of a new kind of Q-value that attempts to smooth out Q-values by framing them in terms of expectations over Gaussian policies.

To be honest, I didn't really "get" this paper.
* As far I understand, all of the original work policy gradients involved stochastic policies.  Many are/were Gaussian.
* All Q-value estimators are designed to marginalize out the randomness in these stochastic policies.
* As far as I can tell, this is equivalent to a slightly different formulation, where the agent emits a deterministic action (\mu,\Sigma) and the environment samples an action from that distribution.  In other words, it seems that if we just draw the box a bit differently, the environment soaks up the nondeterminism, instead of needing to define a new type of Q-value.

Ultimately, I couldn't discern /why/ this was a significant advance for RL, or even a meaningful new perspective on classic ideas.

I thought the little 2-mode MOG was a nice example of the premise of the model.

While I may or may not have understood the core technical contribution, I think the experiments can be critiqued: they didn't really seem to work out.  Figures 2&3 are unconvincing - the differences do not appear to be statistically significant.  Also, I was disappointed to see that the authors only compared to DDPG; they could have at least compared to TRPO, which they mention.  They dismiss it by saying that it takes 10 times as long, but gets a better answer - to which I respond, "Very well, run your algorithm 10x longer and see where you end up!"  I think we need to see a more compelling demonstration of why this is a useful idea before it's ready to be published.

The idea of penalizing a policy based on KL-divergence from a reference policy was explored at length by Bert Kappen's work on KL-MDPs.  Perhaps you should cite that?

review_title: This doesn't seem to actually work

Meta-review: Thank you for submitting you paper to ICLR. Two of the reviewers are concerned that the paper’s contributions are not significant enough —either in terms of the theoretical or experimental contribution -- to warrant publication. The authors have improved the experimental aspect to include a more comprehensive comparison, but this has not moved the reviewers.

Summary: The approach is very promising, but more experimental work is still required to demonstrate significance. 
Decision: Reject