Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: - Summary

This paper presents a framework for performing both differentiable physics simulations and differentiable rendering. This fully differentiable simulation and rendering pipeline is then employed to perform system identification tasks, directly from video frames, being able to match or outperform both visual-based and state-based baselines. Moreover, the potential of this framework to be applied for visuomotor control is also demonstrated.
 

- Pros

This method unified advances in the differentiation of both physics simulation and rendering.

The experimental results demonstrate a good ability to perform system identification for diverse parameters and control directly from videos.
The ability to identify parameters or direct control tasks directly from images is useful, since it reduces the need for direct supervision/annotation in the form of state information.

The presented simulator supports a variety of "domains", such as rigid and deformable body dynamics, cloth simulation, and these are efficient enough to be run faster than real time (at least for simple tasks).



- Cons

Overall, the proposed method is mostly a unification of pre-existing techniques from different fields, such as differentiable rigid and deformable body dynamics, differentiable rendering.

The paper itself admits that a limitation of this method is that it currently "has limited capability to handle contact-rich motion that introduces a large number of discontinuities", which limits its applicability to real-world scenes. It cannot also currently handle joints. All of these would be important for possible robotic applications, for example.

The tasks demonstrated in the experiments are simple, and issues from model mismatch does not seem to have been thoroughly evaluated (see comments below for more).



- Reasons for score

[Edit: Score updated, see discussion below]

Overall, given the "pros" described above, notably the interesting results achieved for system identification and control directly from video frames by combining differentiable physics and rendering into a single framework, I recommend this paper for acceptance. Given some of the concerns raised in the "cons" and in more detail in the comments below, I for now will score this paper as a little above the acceptance threshold.

 

- Additional comments

The scenarios used for the system identification and control tasks are fairly simple, with usually only a single object and few contact points.
Was the ground truth for the scenarios in the system identification tasks generated using gradsim itself? If so, isn't it unfair that it is compared to other models (e.g., pybullet), for which there would be model mismatch? (While not mismatch would be present for gradsim)

Along the same direction, the experiments present a section on "Impact of imperfect dynamics and rendering models". It would also be interesting to see a quantification of the impact of model mismatch (possibly both while using the same renderer, i.e. only dynamics mismatch, or also different renderers)

In the experiments section, it is said that "Inference ... is done by picking an initial guess of the mass (at random)". From what distribution is this random initial guess picked from? What are these starting guesses in relation to the true parameters?

The section on "Impact of shading and texture cues" seems a little too short, which renders it hard to understand in detail what is going on.


review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This work presents a fully differentiable physics simulation coupled with neural rendering such that input video can be used to estimate object properties or find control policies to move those objects by trying to generate the same video at the output.

The paper is well motivated by presenting a natural progression of ideas from this literature and it does a thorough job discussing related work. The paper is light on details in section 3 and it is necessary to refer to the appendix to get a complete picture. Overall, the technical contribution is solid and thus worth accepting the paper even if the validation is with relatively simpler experiments since they are sufficient to motivate this direction to be further researched. Below are a few comments to aid in improving the current work:

- All experiments use what I am guessing are input (desired) videos from the same pipeline and then later hiding some parameters (to be learned). While this is a good validation the learning done here is still 'in distribution'. It would be useful to see if video (even simplistic) from a different simulator or simplified from a real world video could be applied. To what extent is this possible and are there any fundamental limitations that prevent this at the moment?

- Analysis is mostly with one object in an empty scene. Are there technical limitations to handling realistic scenes where there are multiple objects and those objects interact with each other as well the environment? How does this affect performance wrt forward and backward pass timings? With such experiments, it would be helpful to understand if the released code can be easily extended to such (more complex) settings or if someone would need to start a new implementation from scratch.

- The scale on the loss landscape is quite small, '0.4 pixelwise mse'. How good does the initial guess need to be to stay in the range, do the curves in fig 3 continue the trend beyond these values for larger error?

- Reality gap: while this is discuss in reference to visual appearances, since the current experiments deal with synthetic scenes, the more relevant topic to discuss is the reality gap wrt physics and object motions. Experiments designed to study this would boost confidence in this approach.


Other comments:

- How much does the performance depend on good initial guess?

- Currently a single impulse is used to set things in motion, can this be extended to handle more continuous actions?

- Presenting qualitative results for baselines would be helpful

- Some baselines not clearly explained: average, random, ConvLSTM

- How does performance scale with the length of the video?
review_title: Interesting research direction that will spur follow up work despite preliminary nature of evaluation

final_score: rating: 7: Good paper, accept
review_content: This work focuses on the problem of estimating object physical properties from video sequences. 

The proposed framework combines differentiable physical simulations and differentiable rendering to map physical parameters into images differentiably. This paradigm is then used to recover physical parameters from image sequences by means of gradient based optimisation.

Validation of the proposed method is carried through two main synthetic applications, parameter identification and visuomotor control. 

Although the proposed approach still requires 3D ground truth information to yield reliable estimates, it is and encouraging step towards unsupervised physics understanding from image/video data. 

Positive:

-Crucially and differently from previous attempts, the proposed approach does not require 3D supervision - except for geometry and appearance of the static scene (i.e. at t=0). 

-Approach is clever, simple and yields interpretable representation

-First step towards physics understanding from videos

Negative:

- I would improve the quality of the visualisations and plots in the paper (e.g. I found Figure 6 impossible to read)

-  How to differentiate through the physical simulator was not obvious to me. I would have appreciated a more detailed explanation of how that is done in practice for one of the physical problems studied in the paper to be included in the main manuscript, in an effort to make the paper more readable. 


review_title: gradSim: Differentiable simulation for system identification and visuomotor control

Meta-review: This paper presents a framework for joint differentiable simulation of physics and image formation for inverse problems. It brings together ideas from differentiable physics and differentiable rendering in a compelling framework.
Decision: Accept (Poster)