Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Comment: The paper proposes a simple extension to DDPG that uses a distributional Bellman operator for critic updates, and introduces two simple modifications which are the use of N-step returns and parallelizing evaluations. The method is evaluated on a wide variety of many control and robotic talks. 

In general, the paper is well written and organised. However I have some following major concerns regarding the quality of the paper:

- The proposal, D4PG, is quite straightforward which is simply use the idea of distributional value function by Bellemare et al. (previously used in DQN). Two modifications are also simple and well-known techniques. It would be nicer if the description in Section 3 is less straightforward by giving more justifications and analysis why and how distributional updates are necessary in the context of policy search methods like DDPG. 

- A positive side of the paper is a large set of evaluations on many different control and robotic tasks. For many tasks, D4PG performs better than the variant that does not use distributional updates (D3PG), however by not much. There are some tasks showing no-difference. On the other hand, the choice of N=5 in comparisons is hard to understand and lacks further experimental justifications. Different setting and new performance metrics (e.g. data efficiency, number of episodes in total) might also reveal more properties of the proposed methods.



* Other minor comments:

- Algorithm 1 consists of two parts but there are connection between them. It might be confused for ones who are not familiar with the actor-critic framework.

- It would be nicer if all expectation operators in Section 3 comes with corresponding distributions. 

- page 2, second paragraph: typos in "hence my require less samples to learn"

- it might be better if the reference on arXiv should be changed to relevant publication conferences with archival proceeding: work by Marc G. Bellemare at ICML 2017
review_title: the proposed method is simple

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: A DeepRL algorithm is presented that represents distributions over Q values, as applied to DDPG,
and in conjunction with distributed evaluation across multiple actors, prioritized experience replay, and 
N-step look-aheads. The algorithm is called Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG.
SOTA results are generated for a number of challenging continuous domain learning problems,
as compared to benchmarks that include DDPG and PPO, in terms of wall-clock time, and also (most often) in terms
of sample efficiency.

pros/cons
+ the paper provides a thorough investigation of the distributional approach, as applied to difficult continuous
  action problems, and in conjunction with a set of other improvements (with ablation tests)
- the story is a bit mixed in terms of the benefits, as compared to the non-distributional approach, D3PG
- it is not clear which of the baselines are covered in detail in the cited paper:
  "Anonymous. Distributed prioritized experience replay. In submission, 2017.", 
   i.e., should readers assume that D3PG already exists and is attributable to this other submission?

Overall, I believe that the community will find this to be interesting work.

Is a video of the results available?

It seems that the distributional model often does not make much of a difference, 
as compared to D3PG non-prioritized.  However, sometimes it does make a big difference, i.e., 3D parkour; acrobot.
Do the examples where it yields the largest payoff share a particular characteristic?

The benefit of the distributional models is quite different between the 1-step and 5-step versions. Any ideas why?

Occasionally, D4PG with N=1 fails very badly, e.g., fish, manipulator (bring ball), swimmer.
Why would that be? Shouldn't it do at least as well as D3PG in general?

How many atoms are used for the categorical representation?
As many as [Bellemare et al.], i.e., 51 ?
How much "resolution" is necessary here in order to gain most of the benefits of the distributional representation?

As far as I understand, V_min and V_max are not the global values, but are specific to the current distribution.
Hence the need for the projection. Is that correct?

Would increasing the exploration noise result in a larger benefit for the distributional approach?

Figure 2: DDPG performs suprisingly poorly in most examples. Any comments on this,
or is DDPG best avoided in normal circumstances for continuous problems? :-)

Is the humanoid stand so easy because of large (or unlimited) torque limits?

The wall-clock times are for a cluster with K=32 cores for Figure 1?

"we utilize a network architecture as specified in Figure 1 which processes the terrain info in order to reduce its dimensionality"
Figure 1 provides no information about the reduced dimensionality of the terrain representation, unless I am somehow failing to see this.

"the full critic architecture is completed by attaching a critic head as defined in Section A"
I could find no further documenation in the paper with regard to the "head" or a separate critic for the "head".
It is not clear to me why multiple critics are needed.

Do you have an intuition as to why prioritized replay might be reducing performance in many cases?

review_title: Thorough investigation of distributional policy gradients for continuous problems

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper investigates a number of additions to DDPG algorithm and their effect on performance. The additions investigated are distributional Bellman updates, N-step returns, and prioritized experience replay.

The paper does a good job of analyzing these effects on a wide range of continuous control tasks, from the standard benchmark suite, to hand manipulation, to complex terrain locomotion and I believe these results are valuable to the community.

However, I have a concern about the soundness of using N-step returns in DDPG setting. When a sequence of length N is sampled from the replay buffer and used to calculate N-step return, this sequence is generated according a particular policy. As a result, experience is non-stationary - for the same state-action pair, early iterations of the algorithm will produce structurally different (not just due to stochasticity) N-step returns because the policy to generate those N steps has changed between algorithm iterations. So it seems to me the authors are using off-policy updates where strictly on-policy updates should be used. I would like some clarification from the authors on this point, and if it is indeed the case to bring attention to this point in the final manuscript.

It would also be useful to evaluate the effect of N for values other than 1 and 5, especially given the significance this addition has on performance. I can believe N-step returns are useful, possibly due to effectively enlarging simulation timestep, but it would be good to know at which point it becomes detrimental.

I also believe "Distributional Policy Gradients" is an overly broad title for this submission as this work still relies on off-policy updates and does not tackle the problem of marrying distributional updates with on-policy methods. "Distributional DDPG" or "Distributional Actor-Critic" or variant perhaps could be more fair title choices?

Aside from these concerns, lack of originality of contributions makes it difficult to highly recommend the paper. Nonetheless, I do believe the experimental evaluation if well-conducted and would be of interest to the ICLR community. 
review_title: good evaluation, but lacking in originality

Meta-review: As identified by most reviewers, this paper does a very thorough empirical evaluation of a relatively straightforward combination of known techniques for distributed RL. The work also builds on "Distributed prioritized experience replay", which could be noted more prominently in the introduction.
Decision: Accept (Poster)