Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: I would definitely love to have this and use it for my research. A great tool.

However, the paper lacks detail. In particular, I feel that it is impossible for someone to reimplement the research-mostly because of the lack of detail. However, replicability is a crucial part of science. Other publications proposing software (e.g. the tensorflow, theano and edward papers) come along with open source code. This is not the case here and therefore the picture is quite incomplete.

I am not convinced that ICLR is the right venue: robotics conferences such as IROS and ICRA might appreciate it much more. Nevertheless, this is just an encouragement to the authors to interact with those communities.
review_title: Great and refreshing idea, but some lack of detail

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper creates a physics simulator using theano, and uses it to learn a neural network policy by back propagating gradients through the simulation. The approach is novel, and is motivated by being able to learn policies for robotics.

My two key reservations with the paper are as follows:
1. The method is motivated by learning policies for robotics. However, the proposed method is *only* useful for robotics if the learned policy can transfer the real world. Transferring policies from simulation to real-world is an open research problem, and is particularly challenging with less realistic simulators.
2. They key novelty/benefit of this approach over other model-based approaches is that the simulator is differentiable. However, the only empirical comparison in the paper is to a model-free approach (CMA-ES). To appropriately demonstrate the approach, it should be compared to other model-based approaches, which do not require analytic derivatives of the model.

For the reader to fully understand the pros and cons of the approach, the paper should also include quantitative comparison between the speed of the proposed simulator, and that of standard simulation platforms.

Because the idea is interesting and novel, I think it lies above the acceptance threshold. However, it would be significantly improved with the aforementioned comparisons.

Lastly, the writing of the paper could be improved, as it is rather informal and/or imprecise in a number of places. Here are some examples:
-- “we model the use of a neural network as a general controller for a robot” - can be more concisely phrased as something like “we model the robot controller using a neural network” or “the robot controller is modeled using a neural network"
-- “In previous research, finding a gradient…” - This is a run-on sentence.
-- “We basically jam this entire equation into” - This sentence is informal and imprecise.
-- “deep learning neural network” - the word “learning” should be omitted
-- “one big RNN, where we unfold over time” - should be “…RNN, which we unfold over time” or “…RNN, unfolded over time”

The writing would also be improved by making it more concise and fitting the paper into 8 pages.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: A differentiable physics engine is indeed a wonderful thing to have. 

The key selling point of the proposed software is its speed, however there is no comparison to other physics engines. Besides describing the engine's speed in rather creative units (e.g. "model seconds per day"), the reader has no idea if this is fast or slow. Todorov's engine (my simulator of choice) computes a dynamics step and its derivatives wrt both states and controls (using finite-differences) in less than 1ms for a *full humanoid* model (his code is available here mujoco.org/book/programming.html#saDerivative). I think this actually faster than the engine described in this paper, but I can't be sure.

Because this engine is so limited in what it can collide (sphere/sphere and sphere/plane), it would be trivial to build the example models in several other popular engines (e.g. ODE, Bullet and MuJoCo) and simply compare the performance. Until this comparison is done I consider the paper to be incomplete.
review_title: nice idea, missing information

final_score: rating: null
review_content: Following the ICLR reviewing guidelines, here are a few pre-review questions:
1. How does this method compare with model-based reinforcement learning methods, where the model is learned?

2. Is exploration and local optimum a problem in your policy search approach? If so, do you do anything to deal with it (e.g. analogous to exploration strategies used in RL)?

3. How does the speed and fidelity of the dynamics simulation compare to standard simulation platforms, such as Bullet, MuJoCo, etc.?

4. The experiments mention that CMA-ES fails to converge for a small neural network. This seems to disagree with the benchmark results in Duan et al. ICML 2016 (https://arxiv.org/abs/1604.06778). Why is this task harder than the suite of tasks in Duan et al.? How would other deep RL methods compare on the tasks in this paper?

5. As discussed in the paper, the performance of the policy learned in this simulator under other conditions is crucial for this method to be applicable for learning behavior in the real-world. Does the policy transfer, particularly to more complex/realistic simulators?
review_title: pre-review questions

final_score: rating: null
review_content: Very nice work.

Some questions.

1) Will the engine be made available? 

2) How straightforward is it to also integrate unknowns in the model? Say, we want to tune our physical model to match real world observations of the true system. E.g. a robot arm of which we have an accurate model, but still believe it could be improved from data? I suppose that this is what one wants to do ultimately. 

review_title: identify unknowns in the model?

Meta-review: Originality, significance:
  The paper implements a physics-based simulator directly using Theano. This avoids the type of finite differentiation that physics engines such as MuJoCo use to compute derivatives. It is quite an interesting idea, and is demonstrated using learned control for several models. 
 
 Quality, clarity:
  The original version was somewhat loosely written; the current version is improved.
 
 Pros:
 - The nice idea of implementing a physics engine in a language such as Theano, and showing that this is quite feasible.
 - May inspire further work in this direction.
 
 Cons:
 - The speed is not systematically evaluated, as compared to finite-difference-based engines. It is thought to be "in the same ballpark" as other more full-featured engines. It is not clear that those using simulators will care whether it uses the true derivatives or finite differences.
Decision: Invite to Workshop Track