Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: ==Summary==

DDPG is a popular RL method for continuous control problems. It is more widely applicable than traditional model-based approaches like MPC, since it doesn't require differentiable models of the dynamics. However, in many environments, dynamics are differentiable. This paper proposes a method for extending DDPG to exploit simulator gradients. In particular, the Bellman error objective (which is defined in terms of critic values) used for training the critic is augmented with additional terms defined in terms of gradients of the critic. This leads to faster convergence in practice on a range of benchmarks.

==Overall Assessment==

I recommend acceptance. The paper's contribution is well-motivated, works reasonably well, and is relatively easy to implement.

==Comments==

It would be good to add an argument explaining to readers that accurately estimating Q using Q_\phi does not mean that the gradients of Q_\phi will be good approximations of the true gradients of Q. I found Fig 1 of arxiv.org/pdf/1705.07107.pdf informative.

Can you justify the choice of euclidean norm in line 10? In terms of the critic helping teach the actor, the direction of the gradient may be more important than the norm. What if you used cosine sim?

You argue that DRL is better than MPC because DRL explores better. Could you use the simulator gradients somehow to improve exploration?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper shows how the derivatives from a differentiable
environment can be used to improve the convergence rate of
the actor and critic in DDPG.
This is useful information to use as most physics simulators
have derivative information available that would be useful
to leverage when training models.
The empirical results show that their method of adding
this information (D3PG) slightly improves DDPG's
performance in the tasks they consider.
As the contribution of this work is empirical is nature,
I think a very promising future direction fo work is to
add derivative information to and evaluate similar
variants of some of the newer actor-critic methods
such as TD3 and SAC.

I have two minor questions:
1) Figure 2(a) shows the convenrgence of regularizing states,
   actions, and both states and actions and the text
   describing the figure states that this is
   "expected to boost the convergence of Q."
   However the figure shows that regularizing both states and
   actions results in a slower convergence than doing
   them separately. Why is this?
2) How should I interpret the visualization of the
   learned Q surface in Figure 2(f) in comparison to
   the true Q function in Figure 2(g)?
   It does not look like a good approximation.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies optimal control problems where a physical simulator of the system is available, which outputs the gradient of the dynamics. Using the gradients proposed by the model, the authors propose to add two additional terms in the loss function for critic training in DDPG, where these to terms corresponding to the prediction error of $\nabla_{a} Q(s,a)$ and $\nabla_b Q(s,a)$, respectively. However, my main concern is that the form of gradient given in equation (2) might contains an error.

1. Equation (2). Note that in DDPG, the action is given by a deterministic policy. Thus, we have $a_t = \pi(s_t)$ for all $t\geq 0$. For critic estimation, it seems you are basing on the Bellman equation 
$ Q(s,a) = r(s,a) + Q(s', \pi(s'))$, where $s'$ is the next state following $(s,a)$. Then, it seems that Equation (2) is obtained by taking gradient with respect to $(s,a)$. However, I cannot understand what $\nabla_{\pi} Q$ stands for. If it is $\nabla_a Q(s_{i+1}, a_{i+1}) \cdot \nabla_s \pi(s_{i+1}) $, then that makes sense. 

2. Based on the experiments, it seems that the proposed method does not always outperform MPC or DDPG, even in a small-scale control problem Mountaincar. Moreover, it seems that the performance is similar to that of the DDPG. 

3. Here the model-based gradient in equation (2) is defined by only unroll one-step forward by going from $s_i, a_i$ to $s_{i+1}$. It would be interesting to see how the number of unroll steps affect the algorithm, which is a gradient version of TD($\lambda$).

4. Missing reference: Differential Temporal Difference Learning https://arxiv.org/abs/1812.11137
review_title: Official Blind Review #3

Meta-review: This paper proposes a hybrid RL algorithm that uses model based gradients from a differentiable simulator to accelerate learning of a model-free policy.  While the method seems sound, the reviewers raised concerns about the experimental evaluation, particularly lack of comparisons to prior works, and that the experiments do not show a clear improvement over the base algorithms that do not make use of the differentiable dynamics. I recommend rejecting this paper, since it is not obvious from the results that the increased complexity of the method can be justified by a better performance, particularly since the method requires access to a simulator, which is not available for real world experiments where sample complexity matters more.
Decision: Reject