Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper propose a method to learn dynamics and make predictions from sequences of images of 3D rigid bodies. The method incorporates physics priors by using SO(3) as the latent space, and by translating the problem of learning the dynamics to that of learning a (reduced) Hamiltonian. Predictions are then made by and integrating the kinematic equations on SO(3), and Lie-Poisson equations on the Lie coalgebra of SO(3).

They perform experiments with a synthetic dataset consisting of sequences of rotating rigid bodies with different shapes and different, diagonal and non-diagonal, inertia tensors.
strength_and_weaknesses: The methods makes good use of well-known classical mechanics results about freely rotating rigid bodies giving good theoretical reasons for why the method should work. Interpretability of the learned dynamics is of utmost importance when using machine learning to model dynamical systems and the proposed algorithm gives very interpretable results constraining the dynamics on SO(3) and leaving all the unknown and irrelevant characteristics, such as color and shape of the modelled object, to the autoencoder.

Unfortunately, as stated in the paper, there aren't many datasets to assess performances for the particular problem tackled in this paper. 
The dataset could also represent a good benchmark for future works if made available, and therefore is a notable contribution.
The comparison with LSTMs and Neural ODEs is really meaningful as these methods are not as domain-specific as the proposed algorithms but certainly show how inductive biases like the one proposed in this paper are the way to go. Is there any 

Apart from the dynamics, the SO(3) latent space seems to facilitate the reconstruction: perhaps a pre-training of just the encoder-decoder architecture, which is not unrealistic for real-world application, would boost performances. Or perhaps you already do that.


clarity,_quality,_novelty_and_reproducibility: The background section is very concise but sufficiently clear for readers that are familiar with Hamiltonian formulation of rigid bodies dynamics.
As claimed by the authors, and to the best of my knowledge it is the first physics-informed deep learning algorithm for predicting image sequences of 3D rigid bodies.

A few questions:

How figure 4 should be interpreted? I struggle to understand why the predicted $S^{2}\times S^{2}$ parameterisation of SO(3) is so different from the ground truth. In this setting there is a lot of prior knowledge used in the algorithm, namely we know that the system is a freely rotating rigid body. Therefore, as you emphasise, "learning dynamics" boils down to making good predictions. In figure 3 I observe seemingly good predictions that I can't really conciliate with figure 4, where trajectories clearly differ. Section 6.2 does not provide sufficient explanation for why this is the case.

Because of the freedom os assigning an inertial frame Eq. (4) (and the corresponding Lie-Poisson) and (2) are independent. It is not very clear why you need the latter system of equations? Technically you could perform predictions using only the quaternion equations. Is it only because you want to enforce energy conservation? If so it would be interesting to know how the algorithm performs without this penalty in the loss.
summary_of_the_review: The paper propose a method to learn dynamics of freely rotating rigid bodies based on the simple idea that in the Hamiltonian framework, rigid bodies dynamics happens on $T^{*}SO(3)$. This makes learned dynamics interpretable and is a concrete step towards more trustworthy, less black-box machine learning models. Despite I asked for a few clarifications, I think the paper is overall clear.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a method for estimating and predicting 3D rotation dynamics from image sequences of freely rotating 3D rigid bodies such as satellites. The proposed method maps individual images to latent representations isomorphic to SO(3) using auto encoder type neural networks, calculates angular velocities from the latent pairs, and then leams the moment of inertia tensor J in the Hamiltonian. Based on this prediction of future latent states, a predictive image is constructed using the auto encoder. After constructing a new rotating rigid body dataset consisting of a sequence of rotating cubes and rectangles, the authors show that the proposed method gives higher performance than LSTM and NeuralODE.
strength_and_weaknesses: Strengths

This research is impressive because it is a challenging task to realize a neural network method that learns a physical model capable of predicting video images. The fact that they have succeeded in predicting and reconstructing video images to some extent by incorporating a physical inductive bias specific to rotational motion into the model is also highly novel.



Weaknesses

The following two points are considered weaknesses


The advantage over previous research is unclear:

Hamiltonian Generative Networks (HGN) [1] has been shown to be able to learn up to 400 steps in the future for a two-body problem [2].
HGN is also more general-purpose than the proposed method.
The authors need to show that the proposed method has better prediction performance for videos than HGN.

[1] Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, Irina Higgins, "Hamiltonian Generative Networks," ICLR 2020, 2019.

[2] Irina Higgins, Peter Wirnsberger, Andrew Jaegle, Aleksandar Botev, "SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision," NeurIPS 2021, 2021.



The purpose of the method is unclear:.

The author writes that the application of the study is to estimate the orientation of satellites, but the training data in this study assigns a different color to each side of the object. I think it does not match the configuration of the satellites. If the differences between each side of a satellite can be determined, the orientation and motion of a satellite with a known geometry can be estimated analytically. Therefore, the proposed method is not considered necessary to achieve the objectives of the study.

Also, the work [Duong and Atanasov (2021)] cited by the authors in section "2.3 LEARNING DYNAMICS FOR RIGID BODIES" to achieve more general learning of rigid body motions by estimating inverse mass properties could be used for satellite orientation estimation, but no comparative experiments have been conducted. 

These factors made me think that the relationship between the proposed method and the purpose of the study did not seem appropriate.
clarity,_quality,_novelty_and_reproducibility: quality

Some variables with hat are not defined.


clarity

Symbols and variables in chapter 3 were defined in chapter 4, which made the paper difficult for me to understand.


originality

Since there are already several methods for learning Hamiltonian systems from video images[1,2], it is difficult to say that the proposed method is novel unless its performance is sufficiently high.
summary_of_the_review: The importance of the research is high because it is a challenging task to realize a neural network method that learns a physical model capable of predicting video images. However, the performance of the method has not been compared to similar more general-purpose methods, which makes its effectiveness questionable. Therefore, I believe this concern needs to be resolved to accept this manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presented a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. The target is achieved by using a multi-stage prediction pipeline that maps individual images to a latent representation homeomorphic to SO(3), computes angular velocities from latent pairs, and predicts future latent states using the Hamiltonian equations of motion with a learned representation of the Hamiltonian.
strength_and_weaknesses: Strength: 

+ A physics-informed deep learning framework for predicting image sequences of 3D rigid-bodies by embedding the images as measurements in the configuration space SO(3) and propagating the Hamiltonian dynamics forward in time

+ A new rotating rigid-body dataset with sequences of rotating cubes and rectangular prisms with uniform and non-uniform density.

Weaknesses:

- As the paper acknowledged, the paper only evaluated the proposed method in a rather idealized setting. To conclude the current experimental evaluation and ablation studies are insufficient to justify the paper.

- The proposed approach currently relies on the ability to train the model for each system being examined.

- The secod paragraph of the Summary section seems to be overclaiming its contributions without sufficient justification.
clarity,_quality,_novelty_and_reproducibility: Clarity: The introduction section should be improved with details of the proposed method. In Fig. 1, what are (a), (b) and (c) actually not clear.

Quality & Novelty: As the paper acknowledged, the paper only evaluated the proposed method in a rather idealized setting. To conclude the current experimental evaluation and ablation studies are insufficient to justify the paper.


Reproducibility: Currently, more details are needed to achieve reproduction of the proposed method.
summary_of_the_review: The paper presented a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. However, the current experimental evaluation and ablation studies are insufficient to justify the paper. The novelty/contribution of the proposed method over existing work has not been fully justified.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method to estimate 3D rotations of an object from images using classical physics. The paper is focused on predicting  satellite rotations. It uses Hamiltonian representation of rotational motion. With this representation, motion trajectory is differentiable everywhere. This is not true for rotation parametrisation using exponential or quaternion maps as the inverses may be discontinuous. The paper provides a 3D dataset of uniform and non-uniform cube and prism to train the method.
The method uses a set of sequential images to estimate rotation and angular velocities. It encodes images to low-dimensional space representing rotations (learnt during training phase) which are used to estimate angular velocity and predicts next image using Hamiltonian dynamics.
strength_and_weaknesses: Strengths:
1. The use of hamiltonian dynamics is novel to the context of rotation estimation and trajectory prediction of satellites.
2. A dataset has been proposed in order to learn 3D dynamics through images.
3. The experiments show convincing results.

Weakness:
1. No evaluation on real data.
2. In figure 4, it seems that the rotation estimation starts well and drifts away quickly. The rotation trajectory prediction is very off while the image prediction is quite decent. Do the images in fugure 3 (upto t=50) represent a small portion of the trajectory show on figure 4. More details are needed.
clarity,_quality,_novelty_and_reproducibility: I think that the writing is sufficiently comprehensible except in few sections. The main concepts are made quite clear. The limitations and failure cases have been discussed and the future works have been correctly identified. The authors will release the code upon acceptance to make the method reproducible. The main contribution is novel in the context of satellite trajectory estimation.
summary_of_the_review: The paper is definitely interesting. My only concern is the lack of evaluation on real data and the inability of method to deal with non-uniform data. 

After reading the rebuttal and discussion with other reviewers, I have changed my opinion (a bit unfavorably). Here are the main concerns:

1. The experimentation is highly limited. The discussion on handling non-linear mass distribution is non-existent. The authors provide some explanation in the rebuttal but a lot questions remain. Given that the setup is difficult, perhaps the authors could explore non-linearity a bit more. For example, how does the method behave when the mass is non-uniform in a way that the centre of mass deviates strongly with respect to object centre. This discussion could have helped to understand the practical limitations of the method better. Plus, the authors must experiment with more data. Right now, the datasets are limited to a cube and a prism.

2. There were some gaps in writing. The figure 2 in initial draft, which shows the trajectory to drift apart drastically, was later on removed. It is unclear to the readers how the registration works if the estimation of trajectory is so wrong. It is not clear whether the authors are performing a global alignment to fit into the projection. Perhaps the authors should have clarified.

Overall, the paper has an important contribution over sota but the technical novelty is limited. Without a strong experimental evaluation, it was difficult to have an undoubted positive opinion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a very interesting idea, which conceptually I am very much in favor of. Specifically, the authors propose to learn dynamics from pixels (not given perfect state information), by positing as a learning problem in a latent manner. Particularly, they propose latent representations that are homeomorphic to SO(3).

The idea is great, however, there was significant scepticism about the results and the complexity of the experiments. A danger in this line of works is having published works that emphasize a lot on the theoretical aspects, however, they do not sufficiently quantify the success in practice. Now, this was a sufficiently hard case, just because the idea is so nice. At the end of the day, however, the experiments were on arguably data settings that are visually at least underwhelming. Having a rotating cube centered on the frame, with highly distinct colors does not make much of a case for learning dynamics directly from pixels, in other words the paper in its current form overclaims.
justification_for_why_not_higher_score: The empirical findings must become more convincing and generalizable
justification_for_why_not_lower_score: The key idea is great.
Decision: Reject