Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: #rebuttal responses
Thanks for the clarification!  However, I will keep my original score for these reasons:
(1) Only 3 random seeds are used for each environment, which is not convincing as the variance of MAAC is large in some figures.
(2) Baselines are only trained with 10^5 steps and do not converge. Thus it not fair to say that MAAC matches the asymptotic performance of model-free algorithms.

#review
This paper constructs a model-based policy optimization algorithm (MAAC) that uses the pathwise derivative of the learned model and policy across future timesteps. The terminal value function is used to improve stability.  The theoretical guarantee of the error of the model-based gradient is presented. Experimental results show that MAAC outperforms SAC, STEVE, and MBPO on four environments in terms of the sample efficiency.

The experimental results are strong and I appreciate the plots of the gradient error in a simple task, shown in Figure 3. But I want to see the comparison of the final performance of each algorithm in these environments, and I doubt that the baselines do not converge.

However, the paper is badly written. First of all, the authors claim that the pathwise derivate method is applied to optimize the objective function. But the detail of the method is missing. Secondly, I can not follow the procedure of how Q function is learned in the MAAC algorithm. Thirdly, Theorem 4.1 gives a performance improvement bound of the new policy w.r.t J_pi without the entropy term. But the MAAC algorithm optimizes the objective with the policy entropy term.

Also, MAAC applies many techniques in other papers. The paper does not clearly show the advantage of each component:
(1) I want to see the experimental results of MAAC optimizing the objective without the entropy.  
(2) The SVG(1) algorithm should be compared as a baseline as the SVG(1) also uses the gradient of the learned model to optimize the policy.
(3) The policy and the Q function are optimized on both the real samples and the generated samples. I want to see the ablation study or justification on whether training on real samples or both samples.

Questions:
1. How many independent runs are used in experiments?
2. Does the computation of the pathwise derivate method cost much time？ Is MAAC much slower than SAC?

I am happy to increase my score if the authors justify these questions.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents MAAC, a model-based reinforcement learning algorithm which makes use of path-wise derivative to optimize the policy. The learned Q function is used to estimate the terminal rewards and reduce instability and is trained by a STEVE-like style and Clipped Double Q. The dynamics model is learned by the same method in PETS. The policy is trained by directly computing the gradient to improve model-boosted Q. The theory shows that if learned model and learned Q function have similar derivative as the real one, the improvement of policy can be lower bounded. Experiments show that MAAC achieves the state-of-the-art and can be further improved by adding MPC. 


The paper is well-written in general. The emperical results are very good and are claimed to be the new state-of-the-art. 


Questions:

1. Could you please elaborate how J_Q(phi) is calculated? I found one in preliminary but I suppose the one used in Algorithm 1 is different as the paper states that MAAC also uses the model to train Q. 
2. How is H(pi_theta) estimated? What's underlying state distribution of H(pi_theta)? 
3. The objective of policy network is similar to SAC's objective, but the optimization is different. SAC optimizes the objective by minimizing the KL divergence between pi and exp(Q). Do you have any comparison between BPTT and KL divergence minimization? 
4. How do you "use the cross-entropy method with our stochastic policy as our initial distributions" in MPC? 
5. Does MAAC work for Humanoid? 
6. How many samples are used to estimate the expectation in J_pi(theta)? For the single sample experiment, what will happen if the batch size is increased? 
7. In Algorithm 1, how does "Sample trajectories T from \hat{f}_\phi" work? 
8. To my understanding, when H = 0, the policy optimization doesn't use the model. Assuming Q function optimization uses the same H, the only usage of learned dynamics model is that pi and Q are trained in a different state distribution (D vs D_env). The ablation study shows that in this case, MAAC still outperforms SAC. So is it possible that D_model provides data augmentation? 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: After seeing the clarifications made to address the other reviewers and my own reservations, I lean towards accepting this paper.  It is a simple yet novel way to incorporate a model, and I appreciate the thorough results with multiple baselines and additional ablations to show importance of each component of the method.

-------------------------------------

The authors suggest back-propagating through a learned dynamics model and provide a derivation on monotonic improvement of the proposed objective. They show increased sample efficiency, asymptotic performance matching model-free methods, and ability to scale to long horizons. They provide bounds on the error of the gradient when using the learned model and Q function and the total variation distance between policies trained using true dynamics versus the learned dynamics model. This leads to a theorem showing monotonic improvement of the policy under their algorithm, MAAC.

Decision: Weak Accept. The work contains both theoretical and empirical results on back-propagating through a learned dynamics model compared to other model-based and model-free methods. However, I'm unsure about the novelty of the method itself. How is this different from other planning through back propagation methods? This should be an additional section in related work. As examples, there is Universal Planning Networks [1], Differentiable MPC [2], and Path Integral Networks [3], [4] present a way to differentiate through path integral optimal control. I will increase my score if these concerns are addressed.

Nits:
Missing C in "Contrary to these methods" in Related Work section
Conclusion: analized -> analyzed

[1] Universal Planning Networks - Srinivas et al. 2018
[2] Differentiable MPC for End-to-end Planning and Control - Amos et al. 2018
[3] Path integral networks: End-to-end differentiable optimal control - Okada et al. 2017
[4] Mpc-inspired neural network policies for sequential decision making - Pereira et al. 2018
review_title: Official Blind Review #1

Meta-review: The authors propose a novel model-based reinforcement learning algorithm. The key difference with previous approaches is that the authors use gradients through the learned model. They present theoretical results on error bounds for their approach and a monotonic improvement theorem. In the small sample regime, they show improved performance over previous approaches.

After the revisions, reviewers raised a few concerns:
The results are only for 100,000 steps, which does not support the claim that the models achieves the same asymptotic performance as model – free algorithms would.
The results would be stronger as the experiments were run with more than 3 random seats.
In the revised version of the text, it's unclear if the authors are using target networks.

Overall, I think the paper introduces some interesting ideas and shows improved performance over existing approaches. I recommend acceptance on the condition that the authors tone down their claims or back them up with empirical evidence. Currently, I don't see evidence for the claim that the method achieves similar asymptotic performance to model free algorithms or the claim that their approach allows for longer horizons than previous approaches.
Decision: Accept (Poster)