Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a model for learning physical parameters from video. The model takes in a video with a foreground object and background, and predicts the motion of the object in the video. The experiments are done on a set of videos of damped pendulums. 
main_review: Strength:

- The proposed method deals with video prediction in physics-aware fashion by incorporating implicit physics models, which seems to be novel and innovative. The direction that this paper is heading towards is interesting.
- The presentation of the paper is clear and easy to follow.

Weakness:

- The whole framework is built upon an assumption that the video can be (near) perfectly decomposed into foreground objects and background, which is a very toy assumption and cannot be used in any complicated real video data.
- This paper assumes knowing the underlying physics dynamics (in this case pendulum), which is an unreasonable assumption. Other dynamics, if it exists in the video, will not be able to be modeled. 
- The experiments are very weak. 1) only one physics dynamics model of the pendulum is shown; 2) For the pendulum, only one real video data is evaluated; 3) the other experiments are done on synthetically generated data, which are also very weak.
- How about there is more than one pendulum in the video? How about viewing the pendulum from another viewpoint such that the motion pattern is not a perfect "swing"? These are not shown in the paper at all.


summary_of_the_review: Though the direction that this paper is heading towards is interesting, the framework only works on a toy case. The experiments are very weak. This paper does not reach of the bar of the conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to infer from a single video, the underlying physical parameters of a system. To do this, NeuralODE is combined with a neural radiance field to fit the desired video. The physical parameters learned by a NeuralODE may then be reutilized for simulation of the video under novel dynamics
main_review: The paper studies the interesting problem of inferring the underlying physics of a scene given a video.

The paper is well written and the details clear to follow.

However, I have several different major concerns about the approach both in terms of novelty and evaluated empirical results, .

The underlying setting specified in this paper -- that of learning physics from a single video is more of a limitation in my opinion than a strength. To really learn the underlying physics of objects, more than a single video is needed. For example, if I were only to see a block fall once, there is only so much physics information I can gain. Instead, I think the much more interesting question is how we may learn physics of objects more generally from a dataset of different videos.

The underlying novelty of the project is also rather limited. There are variety of different works in NERF that have been fit to different dynamic scenes and video. This paper appears to build largely on top of [1], but swaps a neuralODE to represent the propogation of dynamics, which has also been done before [3].

The synthetic pendulum results seem very toy, with the pendulum rendered on a fake background. In general, the evaluation is the paper is insufficient -- only pendulum scenes are consider, and in that only a single video of the real world.

The underlying approach is also very domain specific. The underlying ground truth ODE equations for the pendulum are given.Furthermore, utilizing a NeuralODE seems limiting when modeling physical scenes. Many physical scenes do not conserve physics.

The paper does not show any downstream applications of the physics that is learned by the approach. Nor is the approach shown to generalize in any way to a new dataset

I would like to see a comparison where a non continuous neural network is used to predict the underlying dynamics of the approach.

The paper is also missing references several additional implicit representations for modeling dynamic scenes which are relevant [2-4]. [3] for example, also utilizes an NeuralODE to parameterize the underlying dynamics of a scene.

[1] Ost et. al, Neural scene graphs for dynamic scenes. CVPR 2021.   
[2] Xian et. al, Space-time  Neural  Irradiance  Fields  for  Free-Viewpoint Video, CVPR 2021.  
[3] Du et. al, Neural Radiance Flow for 4D View Synthesis and Video Processing, ICCV 2021.  
[4] Li et. al, Neural Scene Flow Fields for Space-Time ViewSynthesis of Dynamic Scenes, CVPR 2021. 
summary_of_the_review: The paper has a variety of issues which do not (in my opinion) make it fit for presentation at this time.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a novel method for inferring a physical model from a single video. Their approach estimates the physical parameters and initial conditions of an ODE that describe the dynamics of the object, using neural implicit representations to render a video sequence based on the physical parameters. The authors perform experiments using simulate and real-world pendulum videos and compare to a Lagrangian VAE baseline. 
main_review: The paper fails to provide sufficient empirical support for the advantages of their approach. In particular, I feel like critical baselines are missing (e.g. Jaques et al. 2020) and pendulums are the only dynamics tested. 
summary_of_the_review: As it stands currently, my feeling is that the evidence for the contribution and improvement of the proposed method over prior work is lacking.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method which is capable of obtaining physical models of motion using only visual observations of the motion from a single video (no training data). This is done by combining a differentiable ODE solver with a coordinate-based network which reconstructs the video frames based on the ODE solution. Since this system is fully differentiable, the synthesized video loss can be used to infer the physical parameters and initial condition of the physical model along with the coordinate-based network parameters which represent the video. The work demonstrates that the physical parameters learned using this method are more accurate than those of baseline methods, and can be used to make long-term predictions of future frames of the video more accurately.
main_review: I am not especially familiar with literature or work in physical parameter inference and the state-of-the-art and practices in the field. Thus my comments focus more on the usage of implicit neural representations, or coordinate-based networks, in this application. 

In my opinion, the strengths of the paper are as follows:
1. It seems like the qualitative results significantly outperform the baseline method. Figure 3 shows that the proposed method generalizes to unseen frames significantly better than the baseline method. Figure 4 shows that this is especially the case when more training frames can be used to better infer physical parameters. 
2. The paper is written very clearly (minus a few pieces of information I could not find, see weaknesses), and is overall enjoyable to read. I feel like I learned something about inferring physical parameters from only visual observations. I feel like this is an important problem, and especially could enable synthesis of realistic new visual content.

In my opinion, the weaknesses of the paper are as follows:
1. It seems like the results are very limited in terms of the types of situations they can be applied to. Only results with a pendulum overlaid over a synthetic background are shown, with one single real pendulum example? Is this formulation easy to extend to other physical models? If so, showing this would make the paper appear much stronger as this could be a more general tool for learning dynamics from only visual data.
2. Some clarity of explanation that seems to be missing
    - I don’t see any explanation of the “baseline method” which is compared to. I think much more detail needs to be given to this in the paper to understand if the method is providing a meaningful contribution or not. Similarly, why are no other inference methods that are mentioned in the related work compared to? It is hard to tell the magnitude of the contribution without this.
    - I think that the methods section on “representation of dynamic objects” could contain more information which makes it more clear how the method from Ost et al. (2021) is adapted to this application. What does the dynamic object coordinate-based network model, pixel colors on a coordinate domain? I understand they are warped in the overall scene conditioned on the physical parameters and initial conditions, but it’s not immediately clear how this is translated into an image which can then be composited on to the background.
3. I think it’s not immediately clear why a coordinate-based representation of the scene is really the useful tool here. Is it because rendering from the coordinate-based network is differentiable, allowing gradients to propagate back through the ODE solver and physical parameters? Because if not, I could see a potential baseline foregoing the neural representation of the video as a whole and optimizing physical parameters based on some deterministic rendering algorithm which warps pixels. For example, the background representation network seems unnecessary since it is completely static, why not just have a static image instead of a network to represent it? It would help the paper to have some explanation on this, and potentially ablate some of these contributions.

summary_of_the_review: Overall, the paper describes a method which uses a coordinate-based network and ODE solver in conjunction to learn physical parameters (and their initial conditions) from a single video. This method is shown to outperform the “baseline” significantly, and seems to generate good results in terms of physical parameter estimation and video synthesis. Thus, I believe that this contribution is impactful, as I haven’t seen coordinate-based networks used in this way before. However, the amount of evaluation seems limited (only on the case of a pendulum), the comparisons to competing methods seem limited, and it’s not immediately clear to me why the coordinate-based network needs to be used for learning from a single video. With some clarification of this, I would be happy to raise my score, but based on my limited understanding I believe that the paper is borderline or slightly above.

**Update after Author Response**:
See response comment, I have chosen to retain my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Summary

This paper represents a video as a canonical implicit representation and a set of object motions that are described with an ordinary differential equation. It renders the representation into images and optimize its parameters by minimizing the difference between rendered images and target images. This work utilize an image pyramid for efficient training. It performs experimentes on a nonlinear damped pendulum in both synthetic and real-world settings.

Contributions
1. This paper represents nonlinear object motions with an ordinary differential equation, which enables them to recover interpretable physical parameters from videos.
2. It performs several experiments to validate the proposed approach.
main_review: Strengths
1. The proposed approach is novel. Although there are methods that represents a dynamic object as a canonical object and object motions, this work innovatively models object motions as an ordinary differential equation, which can produce interpretable physical parameters.
2. The paper is well-written. I can understand the approach easily.
3. The technical details are complete.

Weaknesses
1. The validation experiments are not sufficient. The paper says that it can learn physical models from videos. However, there are only experiments on a nonlinear damped pendulum model. It is not sure that the proposed approach can work on other physical models.
2. The comparison experiments only compare the proposed approach with one baseline. I am not sure if there are other methods in the setting, but it would be better to construct some baselines to validate the effectiveness of the proposed components.
3. The ablation studies are not sufficient. There are some ablation studies that can be conducted, such as the iamge pyramid scheme, validating the decomposition of foreground and background, and the accuracies of the approach for captured objects in different motion states.
summary_of_the_review: The proposed approach is novel and interesting. But there need to be more experiments to validate the approach.
review_title: null

Meta-review: The paper proposes a framework for learning the physical parameters of a physical system’s dynamics from a video. The model combines a differentiable neural ODE solver (NODE) with neural implicit representations through a local coordinate-based network which reconstruct the frames based on the ODE solution. Both the static background and the moving objects are modeled via implicit representations. The system being differentiable, it can be trained to recover the physical parameters and the initial conditions of the ODE. Experiments are performed on two toy problems (pendulum and masses that are connected by a spring).

The reviewers agree on the originality of the approach. They however all consider that the paper falls short to demonstrate the potential of the proposed approach because of limited experiments, limited ablation analyses and comparison with baselines. The authors added a new experiment during the rebuttal, but this was not found sufficient to change the reviewers’ opinion.
Decision: Reject