Neural Implicit Representations for Physical Parameter Inference from a Single Video
Authors: Florian Hofherr
Lukas Koestler
Florian Bernard
Daniel Cremers
Venue: ICLR 2022 Submitted (2022)
Keywords: neural implicit representations
physics learning
video interpretation
physical parameter estimation

Abstract: Neural networks have recently been used to model the dynamics of diverse physical systems. While existing methods achieve impressive results, they are limited by their strong demand for training data and their weak generalization abilities. To overcome these limitations, in this work we propose to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) in order to obtain interpretable physical models directly from visual observations. Our proposed model combines several unique advantages: (i) It is trained from a single video, and thus overcomes the need for large training datasets. (ii) The use of neural implicit representation enables the processing of high-resolution videos and the synthesis of photo-realistic imagery. (iii) The embedded neural ODE has a known parametric form that allows for the identification of interpretable physical parameters, and (iv) long-term prediction in state space. (v) Furthermore, the photo-realistic rendering of novel scenes with modified physical parameters becomes possible. 

Under review as a conference paper at ICLR 2022
NEURAL IMPLICIT REPRESENTATIONS FOR PHYSICAL PARAMETER INFERENCE FROM A SINGLE VIDEO
Anonymous authors Paper under double-blind review
ABSTRACT
Neural networks have recently been used model the dynamics of diverse phys- ical systems. While existing methods achieve impressive results, they are limited by their strong demand for training data and their weak generalization abilities. To overcome these limitations, in this work we propose to combine neural im- plicit representations for appearance modeling with neural ordinary differential equations (ODEs) in order to obtain interpretable physical models directly from visual observations. Our proposed model combines several unique advantages: (i) Contrary to existing approaches that require large training datasets, we are able identify physical parameters from only single video (ii) The use of neu- ral implicit representations enables the processing of high-resolution videos and the synthesis of photo-realistic imagery. (iii) The embedded neural ODE has a known parametric form that allows for the identification of interpretable physi- cal parameters, and (iv) long-term prediction in state space. (v) Furthermore, the photo-realistic rendering of novel scenes with modified physical parameters be- comes possible.
syoth
real
synth
real
synth real
synth real
synth real
synth real
pendulum length
Figure 1: Our method infers physical parameters directly from real-world videos, like the shown pendulum motion. Separated by the red line, the right half of each image shows the input frame, and the left half shows our reconstruction based on physical parameters that we estimate from the input. We show 6 out of 10 frames that were used for training. The proposed model can precisely recover the metric length of the pendulum from the monocular video (relative error to length less than 2.5%). Best viewed on screen with magnification. Please also consider the supplementary video.
INTRODUCTION
The physics of many real-world phenomena can be described concisely and accurately using dif- ferential equations. However, such equations are usually formulated in terms of highly abstracted quantities that are typically not directly observable using commodity sensors, such as cameras. For example, pendulum is physically described by the deflection angle, the angular velocity, the damp- ing coefficient, and the pendulum's length, but automatically extracting those physical parameters directly from video data is challenging. Thus, due to the complex relationship between the physical process and images of respective scenes, measuring such quantities often necessitates a trained ex- pert operating customised measuring equipment. While for many physical phenomena humans able to infer (a rough estimation of) physical quantities from a given vidco, physical understanding from videos is open problem in machine learning.

---

Under review as a conference paper at ICLR 2022
Recently, the combination of deep learning and physics has become popular, particularly in the con- text of video prediction. While earlier works (Lutter et al. 2019 Greydanus et al., 2019 Cranmer et al. 2020 Zhong et al. 2020) require coordinate data, 1.c. already abstracted physical quantities, more recent works directly use image data (Levine et al. 2020 Zhong & Leonard (020). A major downside is that all these approaches rely massive amounts of training data, and,
experi- mentally confirm in App. F they exhibit poor generalization abilities. In contrast, in our work we address this shortcoming by proposing solution that extracts semantic physical parameters directly from single video, see Figure I Therefore, we alleviate the need for large data and furthermore facilitate interpretation due the semantics of the inferred parameters in respective physical equa- tions. Additionally, the six previously mentioned works model physical systems using Lagrangian or Hamiltonian energy formulations, which elegantly guarantee the conservation of energy. but can therefore not easily model dissipative systems that are much more common in the real world (Gal-) ley 2013). The proposed model effectively transforms the camera into a physical measuring device with which can observe quantities such the length or the damping coefficient of a pendulum.
To achieve the learning physical models from a single video, we propose to utilise physics- based neural implicit representations in an neural ordinary differential equations for representing abstract physics of visual scenes. Overall, we summarize our main contributions as follows:
manner, where the latter relies on
1. We present the first method that is able to identify physical parameters from a single video using neural implicit representations.
2. Our approach infers parameters of an underlying ODE-based physical model that directly allows for interpretability and long-term predictions.
3. The unique combination of powerful neural implicit representations with rich physical models allows to synthesize high-resolution and photo-realistic imagery. Moreover, it enables physical editing by rendering novel scenes with modified physical parameters.
Contrary to existing learning-based approaches that require large corpora of training data, we propose per-scene model, so that only single short video clip that depicts the physical phe- nomenon is necessary.
2
RELATED WORK
The combination of machine learning and physics has been addressed across extremely broad range of topics. For example, machine learning was used to aid physics research (Bogojeski et al. 2020 Leclerc al.) 2020). or physics was used within machine learning models, such as for auto- matic question answering from videos (Chen et al.] 202T Bear et al., 202T). In this work we focus specficially extracting physical models from single videos, so that in the following we discuss related works that we consider most relevant in this context.
Physies in the context of learning. While neural networks have led to many remarkable results across diverse domains, the inference of physical principles, such as energy conservation, is still a major challenge and requires additional constraints. A general way to endow models with physics- based prior is to use generalized energy functions. For example. Greydanus et al. (2019) and Toth (2020) use neural network parameterize the Hamiltonian of a system, which yields relation between the energy of the system and the change of the state. Hence, they are able to infer the dynamics of systems with conserved energy, such as a pendulum or multi-bodi system.
One disadvantage of using the Hamiltionian is that canonical coordinates need to be used. To eliminate this constraint, other works use the Lagrangian to model the system's 's energy. Since this formalism is more complex, Lutterr et al. (2019) and & Leonard (2020) restrict the Lagrangian to the case of rigid-body dynamics to model systems with multiple degrees of freedom, such as a pole cart, robotic arm. Cranmer et al. (2020) use a neural network parameterize general Lagrangian, which they use to infer the dynamics of a relativistic particle in a uniform potential.
While being able to model many relevant systems, the aforementioned energy- approaches cannot easily be extended to dissipative systems that are much more common in the real world (Galley) 2013). Furthermore, they do not allow for semantic interpretation of individual learned system parameters. PhyDNet, introduced by Guen & Thome (2020), learns dynamics in the form of general PDE in latent space, which, like the aforementioned works, prohibits interpretation
2

---

Under review as a conference paper at ICLR 2022
of the learned physical model. In contrast, in the context of incorporating physical phenomena into learning frameworks, there are also approaches that make the underlying dynamics explicit. For example, Daques et al. (2020) unroll the Euler integration of the ordinary differential equation of bouncing balls, as well as balls connected by a spring, to identify the physical parameters like the spring constant. Kandukuri al. (2020) and de Avila Belbute-Peres linear complementarity problem to differentiably simulate rigid multi-body dynamics that can also handle object interaction and friction. For our method, we also rely on the advantages of modelling the underlying physics explicitly in order to obtain interpretable parameter estimates.
al. (20T8) propose use
Inferring physical properties from video. While many approaches work with trajectories in state space, there are also some works that operate directly on videos. In this case, the information about physical quantities is substantially more abstract, so that uncovering non-linear dynamics from video data is a significantly more difficult problem. Traditionally, such inverse problems are often phrased in terms of optimization problems, for example for deformable physics inference (Weiss) et al., 2020). among many more. While respective approaches can successfully estimate wide range of relevant physical quantities from video data, they often require rich additional information, such 3D information in the form of depth images in combination with 3D template mesh (Weiss) ct al. 2020), which may limit their practical applicability.
More recently, several end-to-end learning approaches have been proposed. de Avila Belbute- et al. (2018) use an encoder to extract the initial state of several objects from the combination of images, object masks and flow frames. After propagating the physical state over time, they decode the state back into images to allow for end-to-end training. Jaques et al. (2020) and Kandukuri et al.) (2020) use an encoder network extract object positions from object masks for individual frames. After estimating initial velocities from the positions they integrate the state over time and use a carefully crafted cerdinate-consistenn decoder, which is based on spatial transformers, to obtain predicted images. [Zhon & Leonard (2020) extend this idea to their variational autoencoder (VAE) architecture to obtain a coordinate-awarr encoder which they use to infer parameters of the latent distribution of generalized coordinates for each frame. Toth et al. (2020) use VAE structure to predict the parameters of a posterior over the initial state from a sequence of videos. All of these approaches require large amounts of data to train the complex encoder and decoder modules. In contrast, our approach does not rely on trainable encoder or decoder structures, but instead uses a non-trainable fixed neural ODE solver in combination with trainable neural implicit representation, and is thus able to infer physical models from a single video.
Implicit representations.
Recently, neural implicit representations have gained popularity due to their theoretical elegance and performance in novel view synthesis. The idea is to use a neural network to parametrize function that maps spatial location to a spatial feature. For example, to
represent geometric shapes, using occupancy values (Mescheder et al. 2019)] Chen & Zhang 2019 Peng et al. 2020), or signed distance functions (Park et al.) 2019; Gropp et al. [020] Atzzmon & Lipman (020). In the area of multiview 3D surface reconstruction as well as novel view synthesis, implicit geometry representations, such as density or signed distance, are combined with implicit
color fields to represent shape and appearance (Sitzmann et al. 2019 Mildenhall et al. 2020 Yariv jet al. 2020 Niemeyer et al.] 20200 Azinovic et al.) 2021). To model dynamic scenes, there have been several approaches that parametrize a displacement field and model the scene in reference
configuration (Niemeyer et al.
2019 Park ct al. 2021] Pumarola et al. 202Ip. On the other hand, 2021 Li et al. 2021 Du et al. (021) include the time an input to the neural representation and regularize the network using constraints based on appearance, geom- etry, and pre-trained depth or flow networks however, none of these methods uses physics- based constraints, e.g. by enforcing Newtonian motion. While the majority of works on implicit represen- tations focuses on shape. Siizzmann et al. (2020) show the generality of implicit representations by representing images and audio signals. Our work contributes to the neural implicit representation literature by combining such representations with explicit physical models.
several approaches (Xian et al.
ESTIMATING PHYSICAL MODELS WITH NEURAL IMPLICIT REPRESENTATIONS
Our main goal is the estimation of physical parameters from a single video, where we specifically focus on the setting of a static background and dynamic objects that are moving according some
3

---

Under review as a conference paper at ICLR 2022
physical phenomenon. With that, we model the dynamies of the objects using an ordinary differ- ential equation (ODE). Our objective is now to estimate the unknown physical parameters, as well as the initial conditions. of this ODE. Hence, we additionally learn video generation model that is able to render a video that depicts objects which follow a specific physical model depending on re- spective physical parameters. For estimating these physical parameters directly from an input video, we utilise photometric loss that imposes that the generated video is similar to the input video.
3.1 MODELING THE DYNAMICS
For most of the dynamics that can be observed in nature, the temporal evolution of the state can be described by an ODE. For example, for a pendulum the state variables are the angle of deflection and the angular velocity, and two dimensional first-order ODE can be used to describe the dynamies.
In general, we write 2 = f (z. t; 0) to describe the ODE where z E R" denotes the state variable, t E IR denotes time and IR." m are the unknown physical parameters. Using the initial conditions 2o RR" at the initial time lo. we can write the solution of the ODE as
(1; zo; 0) Zo + L./ (2(T),
Note that the solution curve z (t: Zo. 8) C IR" depends both on the unknown initial conditions Zg, as well as on the unknown physical parameters 0.
In practice, the solution to Eq. is typically approximated by numeric integration. In our context of physical parameter estimation from videos, build upon the recent work by Chen et al. (2018), who proposed an approach to compute gradients of the solution curve of an ODE with respect to its parameters. With that, it becomes possible differentiate through the solution in Eq. (1) and therefore we can use gradient-basse based methods to estimate zo and 0.
3.2 DIFFERENTIABLE RENDERING OF THE VIDEO FRAMES
To render the video frames, we draw inspiration from the recent advances in neural implicit represen- tations. To this end, we use static representation to model the background, which we combine with a appearance and shape representation of dynamic foreground objects. By composing the learned background with the dynamic foreground objects, whose poses are determined by the solution of the ODE encoding the physical phenomenon, we obtain a dynamic representation of the overall scene. Doing so allows us to query the color values a pixel grid, so that we are able to render video frames in a differentiable manner. Fig. 2 shows an overview of the approach.
Representation of background. The static background is modeled by function F(; 0og) that maps 2D location to appearance value E R where C denotes the number of appearance channels (e.g. RGB colors). The function F(-: fog) encodes the appearance of the background and is represented neural network with learnable parameters Pgg. To improve the ability of the neural network to learn high frequency variations in appearance. we use Fourier features (Tancik et al, 2020), so that the input location where Nyvarier is the numbers of frequencies used. The full representation of the background then reads Oog (x) = F(y (x) :.gg). For more detailed discussion of the architecture, we refer App. A
R22 is mapped to a higher-frequency vector 7 (x) R 1N
To compose the static background and the dynamically moving objects into the full scene, we draw inspiration from Ost et al. (202T) who use implicit representations to represent color and shape in a scene graph to decompose dynamic scene into a background representation and dynamically moving local representations. A drawback of their work is that they do not use a physical model to constrain the dynamics, and therefore strong supervisory signals like the trajectories and the dimensions of the bounding boxes are essential. our case, each dynamic object is represented in terms of local neural implicit representation, which is then placed in the overall scene based on the time-dependent spatial transformation T, = T (z (t; Zo, Oode) 0+). This transformation is parameterized by the unknown initial condition Zo. the physical parameters Bute of the ODE, and possibly additional parameters 04. As such, these parameters determine the transformation from the global coordinate system of the background to the local coordinate system.
Representation of dynamic objects.
W...o.. we only consider first-order ODEs here, since it is always possible to reduce the order one by introducing additional state variables.
4

---

Under review as a conference paper at ICLR 2022
Input:
Video Sequence
Loss
Rendering
Oode
0-
Solution ODE
Oodz
Parametrize
Local Representation G(::Om))
Transformation T
Background Representation F(.;Oua)
Figure 2: Overview of our approach. The dynamics in the video are modelled by an ordinary differ- ential equation, which is solved depending on unknown initial conditions Zo and unknown physical parameters Oode- The solution curve z (t; Zo, Oake) is used to parametrize a time-dependent trans- formation (z (t: Zu, Oade) ..) from the global coordinates XY of the background coordinates zy of the moving object. The functions F(-: 0bg) and G(-; Ooby) are neural networks that model the appearance of the background and the object, respectively. We can estimate unknown physical parameters for given video based on rendering loss which penalizes the dis- crepancy between the input video frames and the rendered video. All estimated parameters and network weights shown in green in the figure.
the local
Similarly as the background, the appearance each individual dynamic object modelled in terms of implicit neural representation (in the local coordinate system). In contrast to the background, we augment the color output e E RC of the dynamic object representation with an additional opacity value o E [o, 1]. which allows us model objects with arbitrary shape. We write the representation of a dynamic object in the global coordinate system as (Cay (x) o (a:) = G(7 (a') ; Oow)). where G(:; (oy)) is represented as neural network with weights Bobj. Y denotes the mapping Fourier features, and x' = (x) ) is the local coordinate representation of the (global) 2D location æ.
Differentiable rendering. pixel grid, where we use the opacity value the local object representation to blend the color of the background and the dynamic objects. we evaluate the equation
For Fenderingg evaluate the composed scene appearance regular
To obtain the final color, for all positions of the pixel grid
c(x, f) = (1 - o(x)) (xg(z) + o(x) Cot))...
Note that due the time dependence of the transformation T., the color value for pixel is also time dependent, which allows us to render the frames of the sequence over time.
3.3 LOss FUNCTION
We jointly optimize for the parameters of the neural implicit representations tog and Goby and estimate the physical parameters fuee; and 0+ of the dynamics and the transformation. To this end, we use a simple photometric loss defined over the pixel values, which reads
Σ LET PEI
Σ
((x
(3)
I
5

---

Under review as a conference paper at ICLR 2022
B: Overfi
B: Full
Ours
GT
4
7
10
13
16
19
22
25
Figure 3: Two masses spring system in which MNIST digits are connected by an (invisible) spring. Reconstruction and prediction for test sequence 6. The arrow indicates the prediction start. For the spring constant and equilibrium distance (k, the different methods achieve the following relative errors: (2.7%, 81.0%) (B: Overfit), (3.7%, 1.8%) (B: Full), and (0.6%, 1.7%) (Ours).
where d computes the discrepancy between its two inputs, T is the of all given time steps, I is the set of all pixel coordinates at the current resolution (see next section) and (x: t) are the given images. To capture information multiple scales employ image pyramid scheme. More details can be found in App.
EXPERIMENTS
We use two challenging physical models to experimentally evaluate our proposed approach. To analyze method and to compare previous work, we first consider synthetically created data. Afterwards, we show that our method achives promising results also on real-world data. For details about the ODEs deseribing the dynamics, additional implementation details, ablation study, as well as additional results we refer the reader to the Appendix.
Although several learning-based approaches that infer physical models from image data have been proposed (de Avila Belbute-Peres et al. 2013 Jaques et al.) 2020 Kandukuri et al. 2020 Zhong & Leonard 2020 Toth al.] (2020), existing approaches are particularly tailored towards settings with large training corpora. However, these methods typically suffer from a decreasing estimation accu- racy in scarce training data regimes,
of distribution generalization is required (cf. App. F). In contrast, our proposed approach is able to predict physical parameters from single short video clip. Due the lack of existing baselines tailored towards estimation from a single video, we adapt the recent work of Jaques et al. (2020) and Zhong & Leonard (2020) to act baseline methods.
4.1 TWo MASSES SPRING SYSTEM
We consider the example of two moving MNIST digits connected by an (invisible) spring on
CIFAR background, in similar spirit Jaques et al. (2020), see Fig. 3 Besides the initial positions and velocities, the spring constant & and the equilibrium distance of the connecting spring need to be identified for the dynamics model. For more detailed description of the model see App. B.
The approach of Jaques et al. (2020) uses a learnable encoder and velocity estimator to obtain po- sitions and initial velocities of a known number of objects from the video frames. After integrating the known parametric model, they use learnable coordinate- contatseeecnieeettt decoder in combination with learned object masks and colors to render frames from the integrated trajectories. Using a pho- tometric loss they require 5000 sequences of different runs of the same two masses spring system to train the model and identify the parameters. In order compare their method work in the set-

---

Under review as a conference paper at ICLR 2022
Baseline Ours
10 11
12
16
18
20
Figure 4: Prediction when training with the first N = 10 frames of sequence 1. Each image shows the prediction of the respective method in white, and the ground truth as green overlay. For both methods, the prediction of images seen during training (frames 1,4,7.10) works well. For unseen data (frames 11.12,16,11 18,20) our method (top) leads more reliable predictions, meaning that our physical parameter estimation is more accurate.
ting of parameter estimation from single video, in addition to their model trained on the full dataset (:B: Full"), we also consider their model trained for an individual sequences of the test dataset ('B: Overfit').
We fit our model to sequences from the test dataset, where we use two local representations and parametrize the spatial transformation shown in App. B.I By using the maximum of both masks as foreground mask, we enable the model to identify the object layering. We find that for training it is necessary to gradually build up the sequence of frames over training. We start with only two frames and add the respective next frame after 60 epochs. Also, the model appears to have a scale freedom in terms of the equilibrium length and the points where the spring is attached to the digits We therefore add an additional loss to keep the spring attachment close to the center of the bounding box of the digits in the first frame. We observe similar effects when overfitting the model of Jaques) et al. (2220) to a single sequence. When training on the full dataset, the effect seems to be averaged and is not observed.
Fig. 3 ssows a qualitative comparison of our results to the baseline Jaques et al. 20200) where the latter is trained in the two settings explained above. We observe, that for this sequence all approaches
yield reasonable results for the reconstruction of the training frames. However, for prediction the overfitted model of Jaques et al.) (2020) performs significantly worse, indicating that the physical model is poorly identified from a single video. The baseline trained on the full dataset yields results that are slightly worse than our results. We see that in both cases the parameters are identified correctly. The fact that achieve comparable results while using significantly less data highlights the advantage of combining the explicit dynamics model with the implicit representation for the objects. Note that chose sequence 6 since yielded the best results for the baseline. More results can be found in app.
4.2 NONLINEAR DAMPED PENDULUM
We use synthetically created videos of nonlinear damped pendulum to compare our method to the previous work of Zhong& & Leonard (2020) and also show the ability of our approach handle high resolution videos. The equations describing the pendulum dynamics can be found in App. B.2
Comparison Lagrangian Variational Autoencoder. We use the dataset of Zhong & Leonard (220) containing several sequences of simple pendulum (each comprising 20 frames), which was created by the OpenAl Gym simulator (Brockman et al. 2016). The method by [hoog & Leonard (2020) uses a coordinate aware encoder to obtain the distribution of the initial state from object masks. After sampling, the initial state is integrated using a learnable Lagrangian function parametrizing the dynamics of the system and coordinate aware decoder is used to render frames from the trajectories. We train the model using only the first frames of a single sequence as the training data (with external control input), effectively overfitting the model to each sequence.
Intuitively, if the motion is only in one direction vary the equilibrium length and adjust the spring attachments without changing the observed motion. Similar effects present in 2D motion.

---

Under review as a conference paper at ICLR 2022
Ours
Ground Truth
training frame
unseen frame
unseen frame
Figure 5: Rendered frames for the high-resolution (1280 x 854 pixel) lake sequence when training on the first 10 frames out of a total of 25 frames. While the first image is part of the training set, the two remaining images are frame 20 and 25 of the full sequence and have not been seen during training. We see that our method produces realistic reconstructions of the scene even for physical states that are not during training. Best viewed screen with magnification.
Similar to the baseline, we assume no damping and a known pivot point A in the middle of the frame to train our model. Since this dataset does not include image data, we only use the loss on the object mask, and train our model in this modified setup using the same frames for the baseline.
To evaluate the performance of each method in identifying the underlying dynamics, we compare the prediction of the unseen frames of the same sequence. Qualitative results presented in Fig. 4 We can observe that both methods fit the given training data very well, however, in the baseline the pendulum motion significantly slows down for unseen time steps and thus it is unable to ob- tain accurate predictions for unseen data. We emphasize that this happens because the method requires significantly larger training datasets, that it performs poorly in the single-video setting considered in this paper. In contrast, our method shows significantly better performance, which highlights the strength of directly modelling physical phenomena constrain the learnable dynam- ies in an analysis- meallsi--by-synhhee by-stnthssis manner. Due to aliasing effects that arise from the low resolution of the frames, our method does not give perfect predictions, however, we use high- nigh-resolution images for our method we achieve nearly perfect reconstruction as we show in Fig. 5 and Fig. 6 For quantitative comparison and further experimental details see App. E.2
In contrast to the baseline, our approach is able to handle high- resolution videos with complex background and pendulum shapes and textures. In this case, our approach accurately the parameters for the full pendulum model as we show in Fig.
High resolution videos.
For this experiment we created several videos by simulating a pendulum with known parameters and then rendering the pendulum on top of an image. Qualitative results of fitting our model to the lake scene can be seen in Fig. 5 We see that our model produces photorealistic renderings the scene, for the predicted frames. The renderings of other scenes are shown in App. E As show in Fig. 6 and Table T is necessary that the frames in the training set cover a sufficient portion of the motion enable correct estimation of the physical parameters.
4.3 REAL PENDULUM VIDEO
We now show that our approach is even able to infer physical parameters from real world data. We recorded the pendulum motion shown in Fig. [ The pendulum is mounted almost frictionless and due to its high weight we do not expect large air drag effects either. The video was recorded with a smartphone, which leads noticeable real-world noise such motion blur, however, the proposed method still manages to produce convincing results. The pseudo groundtruth segmentation masks generated semi-manually by using GrabCut (Rother al. 2004) and exhibit significant noise
8

---

Under review as a conference paper at ICLR 2022
Predicted Deflection Angle
Trajectory Error
10-1
Angle up [rad]
Angle MSE
10-4
Groundirahh
-0.5
truining frames
N 10 training frumes
10-7
2
5
10
15
Time
Training Frames N
Figure 6: Prediction evaluation on the lake scene. The estimated deflection angle is close to the groundtruth for training frames and virtually identical for N training frames even during extrapolation (left). We show the time span covered by training images with transparent boxes above the x axis. The trajectory error, measured by the MSE between the groundtruth deflection angle and predicted deflection angle for E [0, 5). improves with the number of training frames N (right). This shows that sufficient number frames necessary to constrain the model, which can then predict nearly perfect trajectory.
that is also handled well by the proposed model. We extract every third frame from the video s.t. there are 10 extracted frames per second and use the first 10 frames for training.
We use the the full damped pendulum model to estimate the physical parameters of the pendulum motion. The damping is estimated as 4.7 10-13 which matches our expectation for this low friction setting. For the pendulum length we note from Eq. (6) that the estimated length = 27.7 cm a real world quantity without scale ambiguity. Therefore, we can compare it to 4mensured = 27.1 cm which we obtained by measuring the length from the pivot point to the estimated center of gravity of the pendulum using a ruler. We would like to emphazise, that the very good correspondence shows, that we able to estimate scale in monocular video from pendulum motion.
CONCLUSION
In this work presented solution for learning physical model from an image sequence that depicts some physical phenomenon. To this end, we proposed to combine neural implicit repre- sentations and neural ordinary differential equations in analysis-by-synthes, fashion. Unlike existing learning-bar based approaches that require large training corpora, a single short video clip is sufficient for our approach. In contrast to prior works that use encoder-decoder architectures specif- ically tailored to 2D images, we built upon neural implicit representations that have been shown give impressive results for 3D scene reconstruction. Therefore, the extension of the proposed method to 3D is promising direction for future work.
We present diverse experiments in which the ODE parametrizes rigid-body transformation be- tween the background and the foreground objects, such as the pendulum motion. We emphasize that conceptually our model is not limited to rigid-body motions, and that it can directly be extended to other cases, for example to nonlinear transformations for modelling soft-body dynamics. The focus of this work is on learning physical model phenomenon from a short video. Yet, the high fidelity of our model's 's renderings, together with the easy modifiability of the physical parameters, enables various computer graphics applications such as the artistic re- re-rendering of scenes, which we briefly demonstrate in the supplementary video. Overall. per-scene model combines unique set of favorable properties, including the interpretability of physical parameters, the ability to perform long-term predictions, and the synthesis of high-resolution images. We believe that work may serve as inspiration for follow-up works on physics-based machine learning using neural implicit representations.

---

Under review as a conference paper at ICLR 2022
Ethics statement. This work attempts to learn interpretable physical models from video clips of physical phenomena. Our contribution is largely theoretical and we show experiments on synthetic data and limited real-world data. Nevertheless, like understanding of the real, physical world, it is paramount to ensure that they are deployed safely and according to strict ethical guidelines. While think that the current state of our work will not disadvantage or advantage specific groups of people, we recommend a careful ethical evaluation of derivative works that aim to close the gap to human physical reasoning. A potential positive impact of this work is that it can be beneficial to people with lower financial resources, as it overcomes the need for expensive experimental gear infer physical parameters, e.g. in the context of physics education.
machine learning models achieve more human-
To ensure the reproducibility of this work we give architecture and training details in App-[A [pp[AAand and Furthermore, we will release our code upon acceptance, so that all experiments and figures shown this paper can be reproduced.
Reproducibility statement.
REFERENCES
Matan Atzmon and Yaron Lipman. SAL: sign agnostic learning of shapes from raw data. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Dejan Azinovic, Ricardo Martin-Brualla, Dan B. Goldman, Matthias Nießner, and Justus Thies. Neural RGB-D surface reconstruction. CoRR, abs/2104.04532, 2021.
Daniel M. Bear, Elias Wang, Damian Mrowca, Felix J. Binder, Hsiau-Y Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan- Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. Physion: Evaluating physical prediction from vision in humans and machines. CoRR, abs/2106.082611 2021.
Mihail Bogojeski. Leslie Vogt- Mogt-Maranto, Mark Tuckerman, Klaus-Rober Müller, and Kieron Burke. Quantum chemical accuracy from density functional approximations via machine learn- ing. Nature communications, 11(1), 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR,
2016.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Conference on Neural Information Processing Systems (NeurIPS), 2018
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Yee Kenneth Wong, Joshua B. Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reason- ing. In International Conference on Learning Representations (ICLR), 2021.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Miles D. Cranmer, Sam Greydanus, Stephan Hoyer, Peter W. Battaglia, David N. Spergel, and Shirley Ho. Lagrangian neural networks. CoRR, abs/2003.046300 2020.
Filipe de Avila Selbuute-Peres, Kevin A. Smith, Kelsey R. Allen, Josh Tenenbaum, and J. Zico Kolter. End-to-end differentiable physics for learning and control. In Conference on Neural Information Processing Systems (NeurIPS). 2018
Yilun Du, Yinan Zhang, Hong-Xin Xing Yu, Joshua B Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In IEEE International Conference on Computer Vision (ICCV), 2021.
Chad R Galley, Classical mechanics of nonconservative systems. Physical review letters, 110(17), 2013.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Confer- on Neural Information Processing Systems (NeurIPS), 2019.
10

---

Under review as a conference paper at ICLR 2022
Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regu- larization for learning shapes. In International Conference on Machine Learning (ICML), 2020.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Miguel Jaques, Michael Burke, and Timothy M. Hospedales. Physics-as aoysice-as-invvrrsee graephss:: Unsu- pervised physical parameter estimation from video. In International Conference on Learning Represeniations (ICLR), 2020.
Rama Krishna Kandukuri, Jan Achterhold, Michael Möller, and Jörg Stückler. Learning to identify physical parameters from video using differentiable physics. In German Conference on Patter Recognition (GCPR), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Pierre Leclerc, Cédric Ray, Laurent Maahiuu-Wiliaame Laure Alston, Carole Frindel, Pierre-Francois Brevet, David Meyronet, Jacques Guyotat, Bruno Montcel, and David Rousseau. Machine learning-based prediction of glioma margin from 5-ala induced ppix fluorescence spectroscopy. Scientific reports, 10(1), 2020.
Nir Levine, Yinlam Chow, Rui Shu, Ang Li, Mohammad Ghavamzadeh, and Hung Bui. Predic- tion, consistency, curvature: Representation learning for locally-linear control. In International Conference on Learning Representations (ICLR), 2020.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics model prior for deep learning. In International Conference on Learning Representations (ICLR), 2019.
Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sehastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In IEEE International Conference on Computer Vision (ICCV), 2019.
Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable vol- umetric rendering: Learning implicit 3d representations without 3d supervision. IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2020.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Bruuall. Nerfies: Deformable neural radiance fields. In IEEE Interna- tional Conference on Computer Vision (ICCV), 2021.
Songyou Peng. Michael Niemeyer, Lars M. Mescheder, Marc Pollefeys, and Andreas Geiger. Con- volutional occupancy networks, In European Conference Computer Vision (ECCV), 2020.
11

---

Under review as a conference paper at ICLR 2022
Albert Pumarola, Enric Corona, Gerard Pons- and Francese Moreno-Noguer. D-nerf: Neu- ral radiance fields for dynamic scenes. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabeut"; interactive foreground traction using iterated graph cuts. ACM Transactions on Graphics (TOG), 23(3), 2004.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein, Scene representation networks: Con- tinuous 3d-structure-awan neural scene representations. In Conference on Neural Information Processing Systems (NeurIPS), 2019.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In Conference on Neural Information Processing Systems (NeurlPS), 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let net- works learn high frequency functions in low dimensional domains. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Alcksandar Botev, and Irina Higgins. Hamiltonian generative networks. In International Conference on Learning Represen- tations (ICLR), 2020.
Sebastian Weiss, Robert Maier, Daniel Cremers, Rüdiger Westermann, and Nils Thuerey. Correspondence-free material reconstruction using sparse surface constraints. In IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2020.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Yaofeng Desmond Zhong and Naomi Ehrich Leonard. Unsupervised learning of lagrangian dy- namics from images for prediction and control. In Conference on Neural Information Processing Systems (NeurIPS). 2020.
Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ode-net: Learning hamiltonian dynamics with control. In International Conference on Learning Representations (ICLR), 2020.
12

---

Under review as a conference paper at ICLR 2022
2
Nakip
FF
c(x), o(x)
Y(X
FC
FC
FC
FC
FC
FC
FC
FC
FF
Skip connection
Figure 7: Overview of our architecture for the implicit shape and appearance representations. The input vector is passed through layer of Nourierr Fourier features (FF) obtain the encoding 7(x).
The following neural network is constructed from Nrc fully connected layers (FC) of width Wic- We use ReLU activations between the layers. skip connection is used to feed the encoding 7(x) to the -ahip-th fully connected layer, where it is concatenated with the output of the previous layer. We feed the output of the last layer through sigmoid function, achieve values for the color and the opacity (only for the local representation) in the range [0, 1].
A MODEL ARCHITECTURE
We adopt the architecture used in Mildenhall et al. (2020) for the implicit representations, see Fig, for the basic structure. the Fourier features we use logarithmic scaling. The i-th of the Nourierr Fourier features is obtained as
7 (x) (sin (2)x), (2'x)))))
0, Nouurirr 1,
where s2n(2'x) for x E R2 means the element wise application of the sine function. We also include the original in the encoding 7(x).
B MODELS FOR THE DYNAMICS
B. TWO MASSES SPRING SYSTEM
The system is modeled as two-body system where the dynamic of each object is described by New- ton's second law of motion, i.e. F m²ë where F is the force. Since only the ratio between force and mass can be identified without additional measurement, we fix m = 1, analogously to the work of Jaques al. 2022)). Using Hooke's 's law, we write the force applied object by object as
Fig = -k (p:: (p:-P)) P2) (p:-
((::
P
21
(5)
[p: p,1
Using the position (t; k 1) of the objects to parametrize the trajectory the local coordinate systems, we can write the time-dependent 2D spatial transformation to the local coordinate system T: (z) (t(t: k, l), where and & are learnable parameters.
B.2 NONLINEAR DAMPED PENDULUM
A pendulum that is damped by air drag can be modelled as
- sin (4) - cw |سس
(6)
where E R is the deflection angle, w E R is the angular velocity, is the (known) gravitational acceleration, 0 is the (physical) length of the pendulum, and >> is the damping constant.
13

---

Under review as a conference paper at ICLR 2022
We use the solution curve up (t:l, to parameterize the time-dependent 2D spatial transformation A, where R SO (2) is a rotation matrix and E R2 is the pivot point of the pendulum. For the full model, the parameters 1 and are learnable. For the sake of simplicity we assume that the gravitational acceleration always points downwards in the global image coordinate system.
as Te (z) = R (p (t;
C TRAINING DETAILS
In the following we provide additional training details.
C.I DISCREPANCY MEASURE FOR THE LOSS TERM
Unless stated otherwise, for our experiments we use C = image channels, where the three first channels correspond to the RGB channels, and the last channel represents mask of the foreground object. For the real world data, we obtained the objects masks using a semi-manual approach as deseribed in Sec. 4.3 For the experiments the synthetically created high resolution videos used the masks constructed for the video creation for the experiments directly.
For the first three channels define the discrepancy measure in terms of the mean square error as (agh(ar, y) || vll²,
and for the mask in the last channel we consider the binary cross entropy loss, i.e. dwe( (zr y) = [x log (3) + (1 - x)) log (1 y)] With that, the overall discrepancy measure is given as d(x. y) (1gh M1:3) + deeg 84).
(8)
(9)
C.2 OPTIMIZATION
We train our model using the Adam optimizer (Kingma & Ba 2015) with exponential learning rate decay, which reads
r(e) ro ge/ Rawsy
(10)
where r(e) is the learning rate depending the epoch e, ro is the initial learning rate, 3 is the decay rate and Meaaa is the decay step size.
One important aspect of the training is to use different learning rates for the parameters Ong and Oeaj of the implicit representations on the one hand and the physical parameters Oodes 2g and 0. on the other hand.
In order to estimate the initial parameters the ODE and the transformation for the pendulum we employ heuristic that uses the information contained in the mask. To obtain initial estimate for the pivot point A we average all masks and use the the pixel with the highest value. To obtain an es- timate for the initial angle, perform principal component analysis (PCA) the pixel locations covered by the mask and use the angle between the first component and the vertical direction. The velocity is always initialized O. We initialize the damping
and the pendulum length as I = 2 m for the synthetic experiments and / = 0.4m m for the real world experiment.
C.3 IMAGE PYRAMID
To capture information on multiple scales we employ an image pyramid scheme. Due memory limitations, for large images we cannot evaluate all pixel values in one batch, and thus the classi- cal approach that considers all stages of the image pyramid at once not feasible in our setting- Therefore, during training, we sequentially traverse the image pyramid from the low-resolution lev- els towards the original high-resolution level. The idea is that the low resolution stages reveal global information about the movement of the object, whereas the later high-resolution stages allow to use finer details that improve the coarse estimates from the previous stages. To this end, we use Bino- mial kernel of size x 5 with stride two, which repeatedly apply Npyr times reduce the original resolution of the image. We start the training using the coarsest level, and then switch to the next finer level every upgr steps.
14

---

Under review as a conference paper at ICLR 2022
#Frames
270
Visual reconstruction
Full Loss
5
1.06e-03
1.60-033
7.23e-02
4.87e-02
Full Loss
1.71e-03
5.54e-04
1.07e-03
2.12e-03
Lake
Only digb
10
4.82e-03
3.35e-02
9.65e-03
1.71e-02
Only dses
10
3.11c-04
4.12e-03
1.85e-03
2.17e-03
X
Full Loss
5.87e-01
1.00e+00
1.81e-01
6,94e-01
Full Loss
10
1.90e-03
3.99e-03
4.73e-03
1.09c-02
City
Only digh
10
9.20e-04
1.49e-02
5.70c-04
4.18c-03
Only dseg
5.71e-04
9.00e-03
2.86e-03
7.44c-03
Table I: Relative error of the estimated parameters for the lake and city sequences. With N = 10 frames our model is able to accurately infer the physical parameters of the underlying physical phenomenon, while N 5 frames not sufficient to constrain all parameters. While the color as well as the segmentation loss alone give similar results to the combined loss, both losses are needed for the full model. The color loss enables the proposed approach to generate photorealistic renderings on top of physical parameter inference. While using only the color loss gives good accuracy for synthetic data, observed that the segmentation loss, especially in combination with using image pyramids, greatly improves the convergence behaviour for initial conditions that are farther away from the groundtruth parameters. Overall, both losses are necessary obtain accurate physical parameter estimates and photorealistic renderings in robust manner.
D ABLATION STUDY
To motivate the chosen loss functions, report the results for the parameter estimation with dif- ferent loss function configurations in Table Beyond the influence on the quality the parameter estimation, another motivation use the color loss is that it enables to learn the representation of the appearance of the background and the object in the implicit representation. This allows for photo-realistic rendering of unseen predictions, well the re-rendering of scenes with modified physical parameters, effectively allowing physical scene editing. For the mask loss, on the other hand, have found that it makes the estimation process more robust suboptimal initializations of the physical parameters.
E FURTHER EXPERIMENTAL DETAILS AND RESULTS
In the following we consider specific details for the different experiments.
E. TWO MASSES SPRING SYSTEM
Experimental details. We use both loss terms and Aseg 0.01 to balance them. Additionally, we use an MSE loss to keep the center of the bounding boxes of the digits close to the origin of the local representations in the first frame. This fixes the scale problem related to the equilibrium length described in the main text. Moreover, we use another MSE loss term to keep the opacity value close to zero outside (but close to) visible area. found this to be necessary, since otherwise artefacts might appear in the extrapolation when previously unseen parts of the mask appear in the visible area.
For the background we use implicit representation with NFourier Fourier features, NNc fully connected layers width WFC = 128 and an input skip to layer number Nakip = 4. For the local object representation
Nrourier 8 Fourier features, Nrc fully connected layers of width Wic = 128 and an input skip to layer number Nasip
We initial learning rate of F'LP o 0.001 for the parameters of the implicit representations and r paaamm, = 0.01 for the physical parameters. We set SMLP = 0.99954, Meeeaa 50, 3param 0.95 and Peeecaypammm 100.
15

---

Under review as a conference paper at ICLR 2022
B: Overfi
B: Full
Ours
GT
4
7
10
13
16
19
22
25
Figure 8: Two masses spring system, where MNIST digits connected by (invisible) spring. Reconstruction and prediction for test sequence O. The arrow indicates where the prediciton starts. For the spring constant and equilibrium distance (k, the different methods achieve the fol- lowing relative errors respectively: (20.4%, 57.6%) (B: Overfit), (3.7%. 1.8%) (B: Full), and (0.35%, 6.7%) (Ours).
For the image pyramid we use Nyr = train for 1500 epochs, where epoch is completed, when all the pixels in the current resolution have been considered.
stage and step up the pyramid every 7pyr = 200 epochs. We
Additional results. In Fig. and Fig. 9 we present additional results for sequence 0 and sequence of the test dataset. We see, that for both sequences, overfitting the baseline not able to produce reasonable extrapolation of the data and even produces artifacts for the reconstruction part of the sequence. One reason for this is that the model is unable to identify the physical parameters correctly as can be seen by the large relative errors. Our model, the other hand, able estimate the parameters with high accuracy that is even slightly better than the baseline trained on the full training dataset, which again shows the strength of our approach, considering, that we use a single video as input.
E.2 COMPARISON WITH THE LAGRANGIAN VARIATIONAL AUTOENCODER
Experimental details. The data used in this experiment does not include image data, therefore we do not use digh and set Aseg = 1. Since the predicted masks are obtained only from the local representation, we do not use an implicit representation for the background in this example. For the Fourier features, NFC 6 fully connected layers of width
local representation WFC 64 and an input skip to layer number Nakip 3.
NFouiier
We use an initial learning rate of TMLP 0.001 for the parameters of the implicit representations and Tparam, = 0.01 for the physical parameters. We set SMLP = 0.9954, 7LC = 10, 8param 0.995 and Midcaay,paaa param 50.
For the image pyramid we use Npy = stages and step up the pyramid every npyr = 75 epochs. We train for 1500 epochs, where epoch is completed, when all the pixels in the current resolution have been considered.
To quantitatively compare the temporal prediction ability of our approach with the baseline, we follow the procedure by Zhong & Leonard (2020) and report the average squared error (MSE) between the predicted and the ground truth mask for the frames
Quantitative comparison.
16

---

Under review as a conference paper at ICLR 2022
B: Overfi
B: Full
Ours
GT
4
7
10
13
16
19
22
25
Figure 9: Two masses spring system, where MNIST digits connected by (invisible) spring. Reconstruction and prediction for test sequence 1. The arrow indicates where the prediciton starts. For the spring constant and equilibrium distance (k, the different methods achieve the fol- lowing relative errors respectively: (13.6%, 90.9%) (B: Overfit), (3.7%, 1.8%) (B: Full), and (0.1%, 0.3%) (Ours).
Temporal Prediction Ability
10-2
Pixel MSE
Seq. I: Ours
Seq. 1: Baseline
45:
Ours
Seq. 45: Baseline
Seq.
163: Ours
Seq. 163: Baseline
10
7
10
11
12
13
14
15
First N frames used for training
Figure 10: Temporal prediction ability our approach and the approach of Zhong Leonard 2020) overfitted single sequence (baseline). We report the average MSE (Pixel MSE) of the predicted masks for the entire sequence. The horizontal axis indicates the number of frames used for training, and the vertical axis shows the resulting error.
of the full sequence, which we denote as pixel MSE for consistency with the previous work. The results for randomly chosen sequences of the dataset are presented in Fig. [0] can observe that the predictive power for both methods is limited when only few frames are available to infer the underlying dynamics. However, with an increasing number of frames, our method becomes able to reconstruct the physics more consistently, while the baseline does not noticeably benefit from more training frames. We believe that this is because the baseline method overfits to the given frames, whereas our method infers actual physical parameters.
17

---

Under review as a conference paper at ICLR 2022
E.3 EXPERIMENTS WITH HIGH RESOLUTION VIDEOS AND THE REAL WORLD VIDEO
Experimental details. We use the same architecture for the high resolution synthetic and the real-world video sequences. We both loss terms and set Aue 0.03 balance them. For the background we use an implicit representation with Fourier = 10 Fourier features, NFC = fully connected layers of width WFc 128 and an input skip layer number Nakip 4. For the local object representation we use Fourier = 8 Fourier features, NFC = 8 fully connected layers of width Wre 128 and an input skip layer number Nakip 4.
We use an initial learning rate of O 0.001 for the parameters of the implicit representations and "param. 0.05 for the physical parameters. We set SMLL 0.99954, Rdecay.MLP 10, Sparmm 0.95 and pade
100.
For the image pyramid we use Npy stages and step up the pyramid every "pyr 200 epochs. We train for 1500 epochs, where one epoch is completed, when all the pixels in the current resolution have been considered.
Additional results. sults. Fig. [ and Fig. 12 show additional reconstruction and prediction re- sults for additional synthetic high resolution scenes. To create the synthetic
In the following
present additional rendering
scenes we took the background images from https: http:://ppxxbaay com
caompppotos// (Lake). https: //ppp//a//pxaay/..// (City) and
https: xwwax.ba....oo.o wtt. httpa/ywwcc.o/.ooooo/appppeeeeee
hatpppe-rddapppeeeeeeeee (eped-apppes-603559997 60735997
(Apple). Fig. [3 allows for more detailed comparison for the results of the real pendulum video.
We show the images and masks used for training on the real pendulum video in Fig. [4]
Please also
see our supplementary video for additional results on this data.
F GENERALIZATION OF THE LAGRANGIAN VARATIONAL AUTOENCODER
One drawback of learning-baic based approaches for visual estimation of physical models is the poor generalization to data that deviates from the training data distribution. We confirm this for the fully (pre-)trained model of Zhong & Leonard (2020). While the Pixel MSE averaged over the full test set is 1.83 10-3, the error increases 1.22 10-2 when we shift the frames of the test data set by as much as 1 pixel in each direction. This corresponds to the case of input videos, where the pivot point of the pendulum is not in the center of the image, which is different from the training data. This effect is visualized in Fig. 15 which shows the output of the model for sequence 2 of the test data set with zero control input, both in the original version and in the shifted version. We observe that the small shift of only one pixel in each direction leads results that significantly off, and not even the first frame is predicted correctly. While Zhong & Leonard (2020) propose to coordinate-aware encoder based spatial transformers, this introduces additional complexity to the model. In contrast, our approach does not suffer from such issues.
18

---

Under review as a conference paper at ICLR 2022
Ours
Ground Truth
training frame
unseen frame
unseen frame
Figure 11: Reconstruction and prediction results for the city scene. The first frame in the training of 10 frames, while the two frames the right are frame 20 and 25 of the sequence.
Ours
Ground Truth
training frame
unseen frame
unseen frame
Reconstruction and prediction results for the apple scene. The first frame is in the training set of 10 frames, while the two frames on the right are frame 20 and 25 of the sequence.
Figure 12:
19

---

Under review as a conference paper at ICLR 2022
Ours
Figure 13: Comparison of the visual quality of the reconstruction for the real pendulum video trained on the sequence of 10 frames. The numbers indicate which frame of the sequence is shown. Best viewed on screen with magnification. Please also supplementary video.
Figure 14: Training frames and segmentation masks for the real world video. Best viewed on a digi- tal screen with magnification. Upon closer inspection the motion blur as well segmentation mask emror can be seen. The proposed approach can handle this real-world noise and produce compelling reconstructions and predictions shown in Fig. 1 and the supplementary video.
20

---

Under review as a conference paper at ICLR 2022
Shifted Original
Figure 15: Prediction of the fully trained model of Zhong Leonard (2020) for sequence 2 of the test dataset (with zero control). While the prediction for the original data is perfect, the prediction for shifting the frames by one pixel each direction significantly worse. This shows, that the model does not generalize well to input frames where the pivot point of the pendulum is not in the center the frame.
21


Reviews:
[{"review_id": "uRdiK7C5PYA", "reviewer": "Reviewer_s2K9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies the interesting problem of inferring the underlying physics of a scene given a video.\n\nThe paper is well written and the details clear to follow.\n\nHowever, I have several different major concerns about the approach both in terms of novelty and evaluated empirical results, .\n\nThe underlying setting specified in this paper -- that of learning physics from a single video is more of a limitation in my opinion than a strength. To really learn the underlying physics of objects, more than a single video is needed. For example, if I were only to see a block fall once, there is only so much physics information I can gain. Instead, I think the much more interesting question is how we may learn physics of objects more generally from a dataset of different videos.\n\nThe underlying novelty of the project is also rather limited. There are variety of different works in NERF that have been fit to different dynamic scenes and video. This paper appears to build largely on top of [1], but swaps a neuralODE to represent the propogation of dynamics, which has also been done before [3].\n\nThe synthetic pendulum results seem very toy, with the pendulum rendered on a fake background. In general, the evaluation is the paper is insufficient -- only pendulum scenes are consider, and in that only a single video of the real world.\n\nThe underlying approach is also very domain specific. The underlying ground truth ODE equations for the pendulum are given.Furthermore, utilizing a NeuralODE seems limiting when modeling physical scenes. Many physical scenes do not conserve physics.\n\nThe paper does not show any downstream applications of the physics that is learned by the approach. Nor is the approach shown to generalize in any way to a new dataset\n\nI would like to see a comparison where a non continuous neural network is used to predict the underlying dynamics of the approach.\n\nThe paper is also missing references several additional implicit representations for modeling dynamic scenes which are relevant [2-4]. [3] for example, also utilizes an NeuralODE to parameterize the underlying dynamics of a scene.\n\n[1] Ost et. al, Neural scene graphs for dynamic scenes. CVPR 2021.   \n[2] Xian et. al, Space-time  Neural  Irradiance  Fields  for  Free-Viewpoint Video, CVPR 2021.  \n[3] Du et. al, Neural Radiance Flow for 4D View Synthesis and Video Processing, ICCV 2021.  \n[4] Li et. al, Neural Scene Flow Fields for Space-Time ViewSynthesis of Dynamic Scenes, CVPR 2021. ", "summary_of_the_paper": "This paper proposes to infer from a single video, the underlying physical parameters of a system. To do this, NeuralODE is combined with a neural radiance field to fit the desired video. The physical parameters learned by a NeuralODE may then be reutilized for simulation of the video under novel dynamics", "summary_of_the_review": "The paper has a variety of issues which do not (in my opinion) make it fit for presentation at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "lmsma1fJoHQ", "reviewer": "Reviewer_fvJG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The proposed approach is novel. Although there are methods that represents a dynamic object as a canonical object and object motions, this work innovatively models object motions as an ordinary differential equation, which can produce interpretable physical parameters.\n2. The paper is well-written. I can understand the approach easily.\n3. The technical details are complete.\n\nWeaknesses\n1. The validation experiments are not sufficient. The paper says that it can learn physical models from videos. However, there are only experiments on a nonlinear damped pendulum model. It is not sure that the proposed approach can work on other physical models.\n2. The comparison experiments only compare the proposed approach with one baseline. I am not sure if there are other methods in the setting, but it would be better to construct some baselines to validate the effectiveness of the proposed components.\n3. The ablation studies are not sufficient. There are some ablation studies that can be conducted, such as the iamge pyramid scheme, validating the decomposition of foreground and background, and the accuracies of the approach for captured objects in different motion states.", "summary_of_the_paper": "Summary\n\nThis paper represents a video as a canonical implicit representation and a set of object motions that are described with an ordinary differential equation. It renders the representation into images and optimize its parameters by minimizing the difference between rendered images and target images. This work utilize an image pyramid for efficient training. It performs experimentes on a nonlinear damped pendulum in both synthetic and real-world settings.\n\nContributions\n1. This paper represents nonlinear object motions with an ordinary differential equation, which enables them to recover interpretable physical parameters from videos.\n2. It performs several experiments to validate the proposed approach.", "summary_of_the_review": "The proposed approach is novel and interesting. But there need to be more experiments to validate the approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dvpWVrkhAkH", "reviewer": "Reviewer_PZv8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The proposed method deals with video prediction in physics-aware fashion by incorporating implicit physics models, which seems to be novel and innovative. The direction that this paper is heading towards is interesting.\n- The presentation of the paper is clear and easy to follow.\n\nWeakness:\n\n- The whole framework is built upon an assumption that the video can be (near) perfectly decomposed into foreground objects and background, which is a very toy assumption and cannot be used in any complicated real video data.\n- This paper assumes knowing the underlying physics dynamics (in this case pendulum), which is an unreasonable assumption. Other dynamics, if it exists in the video, will not be able to be modeled. \n- The experiments are very weak. 1) only one physics dynamics model of the pendulum is shown; 2) For the pendulum, only one real video data is evaluated; 3) the other experiments are done on synthetically generated data, which are also very weak.\n- How about there is more than one pendulum in the video? How about viewing the pendulum from another viewpoint such that the motion pattern is not a perfect \"swing\"? These are not shown in the paper at all.\n\n", "summary_of_the_paper": "This paper proposed a model for learning physical parameters from video. The model takes in a video with a foreground object and background, and predicts the motion of the object in the video. The experiments are done on a set of videos of damped pendulums. ", "summary_of_the_review": "Though the direction that this paper is heading towards is interesting, the framework only works on a toy case. The experiments are very weak. This paper does not reach of the bar of the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MQrx20EUBM2", "reviewer": "Reviewer_LTRa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not especially familiar with literature or work in physical parameter inference and the state-of-the-art and practices in the field. Thus my comments focus more on the usage of implicit neural representations, or coordinate-based networks, in this application. \n\nIn my opinion, the strengths of the paper are as follows:\n1. It seems like the qualitative results significantly outperform the baseline method. Figure 3 shows that the proposed method generalizes to unseen frames significantly better than the baseline method. Figure 4 shows that this is especially the case when more training frames can be used to better infer physical parameters. \n2. The paper is written very clearly (minus a few pieces of information I could not find, see weaknesses), and is overall enjoyable to read. I feel like I learned something about inferring physical parameters from only visual observations. I feel like this is an important problem, and especially could enable synthesis of realistic new visual content.\n\nIn my opinion, the weaknesses of the paper are as follows:\n1. It seems like the results are very limited in terms of the types of situations they can be applied to. Only results with a pendulum overlaid over a synthetic background are shown, with one single real pendulum example? Is this formulation easy to extend to other physical models? If so, showing this would make the paper appear much stronger as this could be a more general tool for learning dynamics from only visual data.\n2. Some clarity of explanation that seems to be missing\n    - I don’t see any explanation of the “baseline method” which is compared to. I think much more detail needs to be given to this in the paper to understand if the method is providing a meaningful contribution or not. Similarly, why are no other inference methods that are mentioned in the related work compared to? It is hard to tell the magnitude of the contribution without this.\n    - I think that the methods section on “representation of dynamic objects” could contain more information which makes it more clear how the method from Ost et al. (2021) is adapted to this application. What does the dynamic object coordinate-based network model, pixel colors on a coordinate domain? I understand they are warped in the overall scene conditioned on the physical parameters and initial conditions, but it’s not immediately clear how this is translated into an image which can then be composited on to the background.\n3. I think it’s not immediately clear why a coordinate-based representation of the scene is really the useful tool here. Is it because rendering from the coordinate-based network is differentiable, allowing gradients to propagate back through the ODE solver and physical parameters? Because if not, I could see a potential baseline foregoing the neural representation of the video as a whole and optimizing physical parameters based on some deterministic rendering algorithm which warps pixels. For example, the background representation network seems unnecessary since it is completely static, why not just have a static image instead of a network to represent it? It would help the paper to have some explanation on this, and potentially ablate some of these contributions.\n", "summary_of_the_paper": "The paper proposes a method which is capable of obtaining physical models of motion using only visual observations of the motion from a single video (no training data). This is done by combining a differentiable ODE solver with a coordinate-based network which reconstructs the video frames based on the ODE solution. Since this system is fully differentiable, the synthesized video loss can be used to infer the physical parameters and initial condition of the physical model along with the coordinate-based network parameters which represent the video. The work demonstrates that the physical parameters learned using this method are more accurate than those of baseline methods, and can be used to make long-term predictions of future frames of the video more accurately.", "summary_of_the_review": "Overall, the paper describes a method which uses a coordinate-based network and ODE solver in conjunction to learn physical parameters (and their initial conditions) from a single video. This method is shown to outperform the “baseline” significantly, and seems to generate good results in terms of physical parameter estimation and video synthesis. Thus, I believe that this contribution is impactful, as I haven’t seen coordinate-based networks used in this way before. However, the amount of evaluation seems limited (only on the case of a pendulum), the comparisons to competing methods seem limited, and it’s not immediately clear to me why the coordinate-based network needs to be used for learning from a single video. With some clarification of this, I would be happy to raise my score, but based on my limited understanding I believe that the paper is borderline or slightly above.\n\n**Update after Author Response**:\nSee response comment, I have chosen to retain my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4WSERryawXB", "reviewer": "Reviewer_SZKC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper fails to provide sufficient empirical support for the advantages of their approach. In particular, I feel like critical baselines are missing (e.g. Jaques et al. 2020) and pendulums are the only dynamics tested. ", "summary_of_the_paper": "This paper proposes a novel method for inferring a physical model from a single video. Their approach estimates the physical parameters and initial conditions of an ODE that describe the dynamics of the object, using neural implicit representations to render a video sequence based on the physical parameters. The authors perform experiments using simulate and real-world pendulum videos and compare to a Lagrangian VAE baseline. ", "summary_of_the_review": "As it stands currently, my feeling is that the evidence for the contribution and improvement of the proposed method over prior work is lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for learning the physical parameters of a physical system’s dynamics from a video. The model combines a differentiable neural ODE solver (NODE) with neural implicit representations through a local coordinate-based network which reconstruct the frames based on the ODE solution. Both the static background and the moving objects are modeled via implicit representations. The system being differentiable, it can be trained to recover the physical parameters and the initial conditions of the ODE. Experiments are performed on two toy problems (pendulum and masses that are connected by a spring).

The reviewers agree on the originality of the approach. They however all consider that the paper falls short to demonstrate the potential of the proposed approach because of limited experiments, limited ablation analyses and comparison with baselines. The authors added a new experiment during the rebuttal, but this was not found sufficient to change the reviewers’ opinion.
Decision: Reject