Title: A Differentiable Physics Engine for Deep Learning in Robotics
Authors: Jonas Degrave
Michiel Hermans
Joni Dambre
Francis wyffels
Venue: ICLR 2017
Keywords: Deep learning

Abstract: One of the most important fields in robotics is the optimization of controllers. Currently, robots are often treated as a black box in this optimization process, which is the reason why derivative-free optimization methods such as evolutionary algorithms or reinforcement learning are omnipresent. When gradient-based methods are used, models are kept small or rely on finite difference approximations for the Jacobian. This method quickly grows expensive with increasing numbers of parameters, such as found in deep learning. We propose an implementation of a modern physics engine, which can differentiate control parameters. This engine is implemented for both CPU and GPU. Firstly, this paper shows how such an engine speeds up the optimization process, even for small problems. Furthermore, it explains why this is an alternative approach to deep Q-learning, for using deep learning in robotics. Finally, we argue that this is a big step for deep learning in robotics, as it opens up new possibilities to optimize robots, both in hardware and software.

TL;DR: We wrote a framework to differentiate through physics and show that this makes training deep learned controllers for robotics remarkably fast and straightforward

Review discussion:
Following the ICLR reviewing guidelines, here are a few pre-review questions:
1. How does this method compare with model-based reinforcement learning methods, where the model is learned?
2. Is exploration and local optimum a problem in your policy search approach? If so, do you do anything to deal with it (e.g. analogous to exploration strategies used in RL)?
3. How does the speed and fidelity of the dynamics simulation compare to standard simulation platforms, such as Bullet, MuJoCo, etc.?
4. The experiments mention that CMA-ES fails to converge for a small neural network. This seems to disagree with the benchmark results in Duan et al. ICML 2016 (https://arxiv.org/abs/1604.06778). Why is this task harder than the suite of tasks in Duan et al.? How would other deep RL methods compare on the tasks in this paper?
5. As discussed in the paper, the performance of the policy learned in this simulator under other conditions is crucial for this method to be applicable for learning behavior in the real-world. Does the policy transfer, particularly to more complex/realistic simulators?
Hi,
I apologize for the late reply.
1. Since the model is implemented rather than learned, it does not require data acquisition for learning the model.
However, I foresee more use of this approach as a (very) strong prior on the model.
2. In this paper, we did the machine learning and optimization process as vanilla as possible. Exploration is very likely a problem for some applications, but this could be resolved in obvious ways if it were a problem for a specific application. Local optima are unlikely in deep neural networks, but again, if it were a problem, it could be dealt with by appropriate deep learning techniques. I want to stress that the goal of the paper was not to provide an in depth, thorough method dealing with all corner cases, but rather to explain that we see a different way in which deep learning could be combined with robotics.
3. Speed and fidelity are all lower than standard simulation platforms, as is stated in the paper.
On the other hand, it is differentiable.
4. This is an excellent point. We think that the task is hard, but it is indeed surprising that we saw no convergence at all. We note that in Duan et al. ICML 2016 (https://arxiv.org/abs/1604.06778), CMA-ES did not manage to solve all tasks either.
Indeed, it seems that the many parameters are not the only issue at hand. We will look into why this is the case and correct our statement in the paper.
We do not know how other deep RL methods would compare in general. If we look at the results of Schulman et al. (https://arxiv.org/pdf/1506.02438v5.pdf) for the quadruped task, it seems that we require less iterations (500 instead of 1000) with less model-time (8s instead of 2000s). There are however so many differences, that we would draw no conclusions from this observation.
5. This is indeed a question we hope to tackle in our next paper. We found comparing with other simulators not useful, as results would obviously differ. We rather hope to tackle this problem in a different way, as described in the paper. We hope that training our policies with parameter noise (in both hardware and policy) will make them robust against the errors caused by the simulator, and thus more readily transferable to real-world applications.
Given the questions, I would like to stress that we do not want to claim that we have a new or even a good way to use policies, or to outperform other approaches in robotics. The claim we would like to make, is that the physics engines used in robotics rely on 'nearly'-linear equations, and are therefore differentiable in closed form, and that this closed form can be evaluated quickly. We claim that current day AD libraries are powerful enough to remove the laborious task of differentiation by hand, as was common in the time of adjoint optimization. We claim that this is a new approach to deep learning in robotics, one which deserves more attention. We tried to illustrate this claim by training a vanilla policy network in a vanilla way, and showing how this conceptually simple approach already optimizes fast.
Therefore, we ended the paper with a list of the possible use cases we see.
Very nice work.
Some questions.
1) Will the engine be made available?
2) How straightforward is it to also integrate unknowns in the model? Say, we want to tune our physical model to match real world observations of the true system. E.g. a robot arm of which we have an accurate model, but still believe it could be improved from data? I suppose that this is what one wants to do ultimately.
1) It will be made available. I am currently in the process of cleaning up the source. I hope to release it by January.
2) This is indeed something we will try in our next paper. We are confident it would not work, although we cannot exclude the possibility that the gradient on these parameters could become intractable.
This paper creates a physics simulator using theano, and uses it to learn a neural network policy by back propagating gradients through the simulation. The approach is novel, and is motivated by being able to learn policies for robotics.
My two key reservations with the paper are as follows:
1. The method is motivated by learning policies for robotics.
However, the proposed method is *only* useful for robotics if the learned policy can transfer the real world. Transferring policies from simulation to real-world is an open research problem, and is particularly challenging with less realistic simulators.
2. They key novelty/benefit of this approach over other model-based approaches is that the simulator is differentiable.
However, the only empirical comparison in the paper is to a model-free approach (CMA-ES). To appropriately demonstrate the approach, it should be compared to other model-based approaches, which do not require analytic derivatives of the model.
For the reader to fully understand the pros and cons of the approach, the paper should also include quantitative comparison between the speed of the proposed simulator, and that of standard simulation platforms.
Because the idea is interesting and novel, I think it lies above the acceptance threshold.
However, it would be significantly improved with the aforementioned comparisons.
Lastly, the writing of the paper could be improved, as it is rather informal and/or imprecise in a number of places. Here are some examples:
-- “we model the use of a neural network as a general controller for a robot” - can be more concisely phrased as something like “we model the robot controller using a neural network” or “the robot controller is modeled using a neural network"
-- “In previous research, finding a gradient…” - This is a run-on sentence.
-- “We basically jam this entire equation into” - This sentence is informal and imprecise.
-- “deep learning neural network” - the word “learning” should be omitted
-- “one big RNN, where we unfold over time” - should be “…RNN, which we unfold over time” or “…RNN, unfolded over time”
The writing would also be improved by making it more concise and fitting the paper into 8 pages.
Thank you for your remarks. We have indeed removed informal and imprecise wording from our paper to the best of our capabilities, including the ones you have mentioned. Some details have been moved to the appendix and other paragraphs have been made more concise to fit the paper into 8 pages.
However, as long as the engine is not feature complete, we feel it is no use to perform quantitative speed comparisons with other engines. The comparison would not be a fair one. We believe the novelty lies indeed in the following:
* it is feasible to evaluate exact gradients of these robotic frameworks within a reasonable time. Exact gradients should form a favorable approach to finite differences when the number of states of the system rises drastically, for instance when manipulating cloth.
* the recent automatic differentiation libraries are powerful enough to become an important tool in the differentiation of robot models.
Instead, we have made more clear how fast our engine is on our small models.
In order to add more novelty to this approach in this paper, we have added a fourth example to this paper. In this example, the controller relies on a differentiable camera in the differentiable physics engine in order to control a system. To the best of our knowledge, differentiable cameras are a new approach to learning vision in robotics as well.
A differentiable physics engine is indeed a wonderful thing to have.
The key selling point of the proposed software is its speed, however there is no comparison to other physics engines.
Besides describing the engine's speed in rather creative units (e.g. "model seconds per day"), the reader has no idea if this is fast or slow. Todorov's engine (my simulator of choice) computes a dynamics step and its derivatives wrt both states and controls (using finite-differences) in less than 1ms for a *full humanoid* model (his code is available here mujoco.org/book/programming.html#saDerivative). I think this actually faster than the engine described in this paper, but I can't be sure.
Because this engine is so limited in what it can collide (sphere/sphere and sphere/plane), it would be trivial to build the example models in several other popular engines (e.g. ODE, Bullet and MuJoCo) and simply compare the performance. Until this comparison is done I consider the paper to be incomplete.
Thank you for agreeing with this idea. We updated the paper, in order to clarify the speed of the engine. The dynamics step in our engine for a 100 state model is 1ms on GPU, so both engines are in the same ballpark. This comparison is however not fair, and we believe also not the point of our paper. We chose not to compare directly with other engines for the following reasons:
* we believe that such a comparison would not be fair, as our engine currently is less feature complete. We think that comparing to systems which do have full collision detection is pointless for this reason. Comparing with engines which cannot differentiate is equally pointless.
* on the other hand, our gradient is analytically exact.
* our focus was on efficient parallelization of computation, as we think this is a more relevant requirement for deep learning. This is not the case for these other engines.
* nevertheless, as we claimed in the paper: "Our implementation and our model can probably be made more efficient." We did not (yet) work to have this approach be as fast as possible.
We believe that our paper deserves publication for the following novel ideas:
* it is feasible to evaluate exact gradients of these robotic frameworks within a reasonable time. Exact gradients should form a favorable approach to finite differences when the number of states of the system rises drastically, for instance when manipulating cloth.
* the recent automatic differentiation libraries are powerful enough to become an important tool in the differentiation of robot models.
In order to add more novelty to this approach in this paper, we have added a fourth example to this paper. In this example, the controller relies on a differentiable camera in the differentiable physics engine in order to control a system. We believe that the use of differentiable cameras is a new approach to learning vision in robotics. We hope that in your view, this pushes this paper above the acceptance threshold.
I would definitely love to have this and use it for my research. A great tool.
However, the paper lacks detail.
In particular, I feel that it is impossible for someone to reimplement the research-mostly because of the lack of detail.
However, replicability is a crucial part of science. Other publications proposing software (e.g. the tensorflow, theano and edward papers) come along with open source code. This is not the case here and therefore the picture is quite incomplete.
I am not convinced that ICLR is the right venue: robotics conferences such as IROS and ICRA might appreciate it much more.
Nevertheless, this is just an encouragement to the authors to interact with those communities.
Thank you for your kind review. The goal is indeed to open source this library.
However, as it currently stands, the reviews are around the acceptance threshold. I would rather not jeopardize this publication at possibly other conferences by opening the code before it is accepted for publication. If it were not for the open review process, the code would already have been included with the paper. As it stands now, all I can give is my word that it will be made open source.
I am however unfamiliar with an open review process, and the reason I see no way around this issue might by my inexperience. Perhaps I could invite you personally to inspect the repository if you mail me an anonymous github account? Perhaps you see another possibility?
> I am not convinced that ICLR is the right venue: robotics conferences such as IROS and ICRA might appreciate it much more.
We have presented work at those robotics conferences before. In our experience, they usually have little - although growing - focus on machine learning results. We think that this physics engine could form a strong prior for machine learning in robotics, and that it is not yet common to use machine learning libraries this way. Or to formulate a prior for a neural network in this way. We thought this falls within the scope of `learning representations'.
On top of that, we have  presented this work at the deep reinforcement learning workshop at NIPS on invitation of the organizers. For these reasons, we believe that this work does fall within the context of ICLR.
On a more personal level: I am quite environment minded, and ICLR is reachable by train this year, while both IROS and ICRA require intercontinental flights.

Meta-review:
role: PC
data: comment: Originality, significance:
  The paper implements a physics-based simulator directly using Theano. This avoids the type of finite differentiation that physics engines such as MuJoCo use to compute derivatives. It is quite an interesting idea, and is demonstrated using learned control for several models. 
 
 Quality, clarity:
  The original version was somewhat loosely written; the current version is improved.
 
 Pros:
 - The nice idea of implementing a physics engine in a language such as Theano, and showing that this is quite feasible.
 - May inspire further work in this direction.
 
 Cons:
 - The speed is not systematically evaluated, as compared to finite-difference-based engines. It is thought to be "in the same ballpark" as other more full-featured engines. It is not clear that those using simulators will care whether it uses the true derivatives or finite differences.

Decision: Invite to Workshop Track