Question: Can the proposed approach work on other physical models?
Evidence: We jointly optimize for the parameters of the neural implicit representations θ bg and θ obj and estimate the physical parameters θ ode , z 0 and θ + of the dynamics and the transformation. To this end, we use a simple photometric loss defined over all the pixel values, which reads where d computes the discrepancy between its two inputs, T is the set of all given time steps, I is the set of all pixel coordinates at the current resolution (see next section) and I (x, t) are the given images. To capture information on multiple scales we employ an image pyramid scheme. More details can be found in App. C.
In this work we presented a solution for learning a physical model from an image sequence that depicts some physical phenomenon. To this end, we proposed to combine neural implicit representations and neural ordinary differential equations in an analysis-by-synthesis fashion. Unlike existing learning-based approaches that require large training corpora, a single short video clip is sufficient for our approach. In contrast to prior works that use encoder-decoder architectures specifically tailored to 2D images, we built upon neural implicit representations that have been shown to give impressive results for 3D scene reconstruction. Therefore, the extension of the proposed method to 3D is a promising direction for future work.
We present diverse experiments in which the ODE parametrizes a rigid-body transformation between the background and the foreground objects, such as the pendulum motion. We emphasize that conceptually our model is not limited to rigid-body motions, and that it can directly be extended to other cases, for example to nonlinear transformations for modelling soft-body dynamics. The focus of this work is on learning a physical model of a phenomenon from a short video. Yet, the high fidelity of our model's renderings, together with the easy modifiability of the physical parameters, enables various computer graphics applications such as the artistic re-rendering of scenes, which we briefly demonstrate in the supplementary video. Overall, our per-scene model combines a unique set of favorable properties, including the interpretability of physical parameters, the ability to perform long-term predictions, and the synthesis of high-resolution images. We believe that our work may serve as inspiration for follow-up works on physics-based machine learning using neural implicit representations.
Two masses spring system in which MNIST digits are connected by an (invisible) spring.****We emphasize that conceptually our model is not limited to rigid-body motions, and that it can directly be extended to other cases, for example to nonlinear transformations for modelling soft-body dynamics. The focus of this work is on learning a physical model of a phenomenon from a short video. Yet, the high fidelity of our model’s renderings, together with the easy modifiability of the physical parameters, enables various computer graphics applications such as the artistic re-rendering of scenes, which we briefly demonstrate in the supplementary video. Overall, our per-scene model combines a unique set of favorable properties, including the interpretability of physical parameters, the ability to perform long-term predictions, and the synthesis of high-resolution images.
Answer: We show that our proposed method works for other physical systems, in this specific case two masses that are connected by a spring.