Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a modification to Policy Prediction Networks (PPN) in which the learned transition-, reward- and value function models are used at test-time in a planning procedure. 
A second contribution is the "pi-Q-backup" which uses the geometric mean of both the policy and the value function as maximisation target of the planning step.

 Overall, I find the idea interesting and the experimental evaluations promising. However, I am voting for "weak reject" for the reasons outlined below. If some (or all) of them are address, I'd be happy to raise my score. 

- I found the paper hard to understand. In particular, the algorithm PPN on which this work is build is not explained at all, requiring the reader to read the original PPN paper. Including a description of PPN, including it's main features, would greatly help the paper. Second, I am still not sure I correctly undestand when each component is used. As I currently understand it, the main usage of the model during training time is to compute the Advantages in equation (2)? Or are those computed based on rollouts? If so, where is the model actually being used during training?
- Figure 1 is taken directly from the PPN paper without any reference or citation (as far as I can tell).
- For the comparison in Figure 5, it would be great if PPN could also be tested with the newly introduced parameter \beta_i. At the moment, it is hard to tell whether the performance gains are due to \beta or due to the proposed planning scheme.
- I'm confused about Theorem 1: Wouldn't we want an upper bound on the difference of means?  Also, what does 'worst-case' mean? Is that for the 'worst' Q-function we could choose? 

Edit:
Thank you for your comments and updated manuscript.

I think the writing has improved significantly, but could still be further improved and clarified. In particular, the question of how the model and other components at various points in time could be made more obvious. I found the authors' response to R3 here helpful as well. 
At least for me some of the confusion arises not due to the complexity of the proposed approach, but just because combining real and 'simulated' transitions can be used and mixed in so many different ways that it's important to be clear about it. Also, at least personally, I found the explanation "Learning is done with a model-based approach that follows the behaviour policy’s rollout trajectory. However, the test policy follows a model-based approach" still not very helpful. 
Overall, I think the presentation is on a good way but needs some more work.

With that being said and now having a better understanding of the algorithm I think this is very interesting work. However, I share R1's concerns about the computation of the \pi-Q backup, in particular that it seems arbitrary and doesn't handle negative values. I'm also not convinced that adding |min(Q)| is a good solutions as a) we don't always have access to that value and b) If I'm not wrong, than \pi_F is not invariant under a shift of Q. 
I'm wondering why the authors decided to take the geometric mean instead of following the more typically used approach of using exp(Q/Temperature)*pi to combine Q-function and a policy distribution (see e.g. the "Control as Inference" literature, the "Maximum a Posteriori Policy Optimisation" or "Soft Actor Critic" algorithms, in particular the "Soft Value functions". I think this should at the very least be an ablation study, and could be even performing better and at the very least be robust to negative values.

Overall, I think this is very interesting work and could become a very strong paper, but I will remain to recommend a "weak reject" because I think it needs some more work to get there. 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper claims to present a method for combining model based and model free approaches. The paper I find very poorly written hence my certainty about understanding the method cannot be very high. In training the method seems to build up a backup tree using transition operators and a policy and using them as targets for learning. In training it is not quite as clear what they are doing. The paper seems novel and sensible and has some experimental results that are not trivial but the writing is so difficult to follow that it makes it impossible for me to assess the contributions and even check correctness. I also think that readers would find it too difficult to understand as well. This is making a complete rewrite mandatory. I have added some initial pointers that would help making this more readable but implementing these would only allow us to assess what is being done rather than guarantee acceptance. 

Since the rebuttal is not intended as a deadline extension I recommend rejecting this paper!

Major points:
* I find the related work quite badly written. There is content but what the reader cares about it situating the paper in the landscape of existing methods. There is none of that here: why do we care in this work about PPO and not say Q(\lambda). It should build up the components that were existing in the literature not just present some other methods. It needs to tell us roughly what is similar in this work to what was previously existing (roughly at least).
* If PPN is so central it has to be presented before PTN and notation should be introduced there. Introducing formulas without explaining notation like eq (1-4) serves only to alienate the reader and the (well-intentioned) reviewer.
* Please separately present how inference works and present the learning all in one place instead of losses in 3.1 and how to construct targets spread out until 3.2.2
* The fact that you need so many "note that " should be a red flag that the writing is not right (12 times).
* The algorithm was the most useful thing in the paper but even there it should be much clearer e.g. what is Q, how come we can write Q[j] = in consecutive lines. The second one should probably be Q[j] +=. I can't be sure because everything else is so hard to track.
* To me if you have a network that takes in previous step and action and produces a latent next step that is an explicit transition model. How is it not ?
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents Policy Tree Network, a novel approach to use an implicit dynamics model to perform decision-time planning in continuous action spaces. The experiments show that the proposed method performs better than the underlying model-free RL algorithm in standard MuJoCo environments. 

The writing quality is low and I don't understand the proposed method, especially the backed-up Q-value. The notation is also confusing. It seems to me that pi(a|s) is the density of action a, so what does sqrt(value * density) mean? What if the value is negative? 

Besides, I have some questions.
1. Figure 1 looks the same as Figure 1 in https://arxiv.org/pdf/1909.07373.pdf, but I don't find any reference in the paper. Could you please state the difference between two figures, or explain why you want to put this figure here without any description or reference? 
2. How is PTN compared to model-based RL algorithms? The only baseline here is PPO, which is model-free. More importantly, note that the policy in PPO is stochastic, so how is PTN compared to the deterministic policy? 
3. How is the proposed pi-Q-backup method compared to classical control method, e.g. MPC? Does the proposed planning algorithm work for model-free algorithms? 
4. As this paper talks about planning with implicit dynamics models, how is the proposed method compared with explicit dynamics models? 

Minor comments:
1. In Algorithm 1 Line 11, could you please check the brackets? 
2. Page 4, "Thus, cumulative density function (cdf) of pi_F is given by ...": Could you please check the correctness of the equation? 
3. What does "worst-case" in Theorem 1 mean? 
4. How is correlation in Table 2 calculated? 
5. In Algorithm 1, is the return value a scalar or a vector? 
6. The paper states that "Intuitively, the branching factor b can be thought of as interpolating how much confidence we have in pi and Q0". One can have infinite b but sample a uniformly to optimize Q (and then pi_F becomes maxQ policy), so I don't think b can be simply characterized as the confidence. 

review_title: Official Blind Review #1

Meta-review: The consensus amongst the reviewers is that the paper discusses an interesting idea and shows significant promise, but that the presentation of the initial submission was not of a publishable standard. While some of the issues were clarified during discussion, the reviewers agree that the paper lacks polish and is therefore not ready. While I think Reviewer #3 is overly strict in sticking to a 1, as it is the nature of ICLR to allow papers to be improved through the discussion, in the absence of any of the reviewers being ready to champion the paper, I cannot recommend acceptance. I however have no doubt that with further work on the presentation of what sounds like a potentially fascinating contribution to the field, the paper will stand a chance at acceptance at a future conference.
Decision: Reject