Learning Interpretable Dynamics from Images of a Freely Rotating 3D Rigid Body
Authors: Justice Mason
Christine Allen-Blanchette
Nicholas F Zolman
Elizabeth Davison
Naomi Leonard
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep Learning
Dynamics
3D
Rigid Body
Images
Physics informed

Abstract: In many real-world settings, image observations of freely rotating 3D rigid bodies, such as satellites, may be available when low-dimensional measurements are not. However, the high-dimensionality of image data precludes the use of classical estimation techniques to learn the dynamics and a lack of interpretability reduces the usefulness of standard deep learning methods. In this work, we present a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. We achieve this using a multi-stage prediction pipeline that maps individual images to a latent representation homeomorphic to $\mathbf{SO}(3)$, computes angular velocities from latent pairs, and predicts future latent states using the Hamiltonian equations of motion with a learned representation of the Hamiltonian. We demonstrate the efficacy of our approach on a new rotating rigid-body dataset with sequences of rotating cubes and rectangular prisms with uniform and non-uniform density.

Under review as a conference paper at ICLR 2023
LEARNING INTERPRETABLE DYNAMICS FROM IMAGES OF A FREELY ROTATING 3D RIGID BODY
Anonymous authors Paper under double-bling review
ABSTRACT
In many real-world settings, image observations of freely rotating 3D rigid bodies, such as satellites, may be available when low-dimensional measurements are not. However, the high- digh-dimensionality of image data precludes the use of classical estimation techniques to learn the dynamics and a lack of interpretability reduces the usefulness of standard deep learning methods. In this work, we present a thysics-inffrmed neural network model to estimate and predict 3D rotational dynamics from image sequences. We achieve this using multi-stage prediction pipeline that maps individual images to a latent representation homeomorphic to (3),,, computes angular velocities from latent pairs, and predicts future latent states using the Hamiltonian equations of motion with a learned representation of the Hamiltonian. We demonstrate the efficacy of approach a new rotating rigid-body dataset with sequences of rotating cubes and rectangular prisms with uniform and non-uniform density.
INTRODUCTION
Images of 3D rigid bodies in motion available across range of application areas and can give insight into system dynamics. Learning dynamics from images has applications to planning, navigation, prediction, and control of robotic systems. Resident space objects (RSOs) are natural or man-made objects that orbit a planet or moon and are examples of commonly studied, free-rotating rigid bodies. When planning proximity operation missions with 550sscoolecting samples from an asteroid (Williams et al. 201 8), servicing malfunctioning satellite (Flores-Abad et al. 2014), or active space debris removal (Mark and Kamath 2019) it is critical to correctly estimate the RSO dynamics in order to avoid mission failure. Space robotic systems typically have access to onboard cameras, which makes learning dynamics from images a compelling approach for vision-based navigation and control.
Previous work BlancheBl et al. 2020) Zhong and Leonard (2020) Toth et al. (2020) has made significant progress in learning dynamics from images of planar rigid bodies. Learning dynamics of 3D rigid-body motion has also been explored with a variety of types of input data Duong and Atanasov () Byravan and Fox (2017); Peretroukhin al. 2020). Duong and Atanasov (2021) uses state measurement data (i.e. rotation matrix and angular momenta), while PPeretrokhhnn et al.
(2020) learn the underlying dynamics in overparameterized black-box model. The combination of deep learning with physics-based models allows models to learn dynamics from high-dimensional
data such as images JAllen-Blanchele
al. 2020; Zhong and Leonard 2020 Toth al. 2020).
However, as far as we know, our method is the first to use the Hamiltonian formalism to learn 3D rigid-bodk dynamics from images
Kinematics and dynamics of 3D rigid body rotation are both fundamental to accomplishing the goals of this paper. The kinematics describe the rate of change of rigid body orientation as function of the orientation and the angular velocity. Our method integrates the kinematic equations to compute the orientation trajectory in latent space using the latent angular velocity. The dynamics describe the rate of change of the angular velocity as a function of the angular velocity and the moment-offinertia matrix J, which depends on the distribution of mass over the rigid body volume. J is unknown and cannot be computed from knowledge of the external geometry of the rigid body, except in the special case in which the mass is known and the mass is uniformly distributed over the rigid body volume. In our framework, we learn the dynamics from the motion of the rigid body, not from the external

---

Under review as a conference paper at ICLR 2023
n! frame
9 g S0(3)
E
x
SO(3)
Sequence
Input Sequence
Velocity Estimator
/Dynamics Rollout
S S SO(3)
X
O
s
O
SO(3)
SO(3)
Reconstructed Sequence
Consecurtive Orientatoon Matrices
Estimation Algorttim
Encoded Angular Mornentum Estimate
Predicted Angular Momentum
Tk, Ryg yRa
[u),
Angular
MBC
pee
antirray choice e- (ootukkanda)) =
Dynamics
R
u colR
Q
Encoded Rotation Matrix
Predicted Rotation Matrices
nomaliaaaa
(R. (18/Ojj
ess:
- con't
znju)
return
Update
Z
Figure 1: Schematic of model architecture (top). The architecture combines an auto-encoding neural network with a Hamiltonian dynamics model for 3D rigid bodies (bottom-right). The encoder maps sequence of images to sequence of latent states in (3)3). We estimate angular velocity and momentum, then predict future orientation and momentum using the learned Hamiltonian (bottom- left). Each future latent state is decoded into an image using only the predicted orientation.
geometry since the special case is not practical. Importantly, the difference between the dynamics of a uniformly distributed mass and non-uniformly distributed mass inside the same external geometry is significant. We show that we can learn these very different dynamics even when the external geometry is the same. By integrating the learned dynamics, we can predict future latent angular velocity. Works such as [1,2] estimate the transformation between image pairs, which can then be used to generate image trajectories. However, this approach implicitly assumes the object evolves with constant velocity, assumption that is not truc in the general case. Our work addresses this limitation by estimating the angular acceleration and mass distribution for the object in addition to its angular velocity.
In this paper we introduce a model, with architecture depicted in Figure I that is capable (1) learning 3D rigid-body dynamics from images, predicting future image sequences in time, and (3) providing a low-dimensional, interpretable representation of the latent state. Our model incorporates the Hamiltonian formulation of the dynamics as an inductive bias facilitate learning the moment of inertia tensor J E 33x3 and an auto ancoden map between images and SO(3). The efficacy of our approach is demonstrated through long-term image prediction and its additional latent space interpretability.
RELATED WORK
2.1 HAMILTONIAN AND LAGRANGIAN PRIORS IN LEARNING DYNAMICS
poysics-guided machine learning approaches with Lagrangian and Hamiltonian priors often
state-control (c.g. ID position and control) trajectories (Ahmadi and Khadir 2020) Chen et al.
2020 Cranmer et al. 2020a Duong and Atanasov 2021 Finzi et al. 2020 Greydanus al. 2019
2

---

Under review as a conference paper at ICLR 2023
Gupta et al. 2019 Lutter al. 2018 Zhong al.
2020b a train their models. Lagrangian- -bsed
methods (Lutter et al. 2018; Gupta et al. 2019; Cranmer et al.) 2020a Finzi et al.) 2020: Allen
Blanchette et al. 2020; Zhong and Leonard (020) parameterize the Lagrangian function [gg 9)
using neural networks to derive the Fuler- Huler-Lagrang Lagrange dynamical equations (EI.Es). These methods
have been used to learn inverse dynamics models to derive control laws from data (Lutter et al. 2018
and approximate the ELEs with neural networks for forward prediction (Cranmer et al.) 2020a Finzi
ct al. 2020). Greydanus et al. (2019) create Hamiltonian- baaddd priors by introducing Hamiltonian Neural Networks (HNNs), method for parameterizing the Hamiltonian, H, as neural network and calculating the symplectic gradients to construct Hamilton's 's equations. Zhong et al. (2020b) builds upon the ideas of (Greydanus et al. 2019) by further decomposing the Hamiltonian into different coordinate-dependent functions, such as the system's 's potential energy, inverse mass matrix, and input matrix-providing additional structure. This method was extended to more general systems by /Zhong] ct al. (220a)), where they introduce dissipative forces into their formulation. These models focus on using a dataset of state-control trajectories whereas model uses image sequences. Moreover, while (Zhong et al.] 2020b]a a have elements of their work that address rigid-body systems, these models focus on planar systems and do not use images.
2.2 USING IMAGES FOR LEARNING DYNAMICS
Image-based learning models focus on using a set of images that represent the state trajectories for systems as data train their models for a variety of tasks such as dimensionality reduction and learning dynamics, Approaches that model the dynamics of the system (Li et al., 2020 Zhong and Leonard 2020 AllennBBanchhttt et al. 2020 Toth et al. 2020) learn dynamics from image-state data but only for either 2D planar systems or systems with dynamics in R", using pixel images. Zhong and Leonard (2020) use Lagrangian prior and a coordinate- -waae aware encoder and decoder in their variational auto-encoding neural network to estimate the underlying dynamics and do video prediction with control. Allen-Blanchette et al. (2020) focus on learning dynamies from images for similar systems but using a non-variational approach, encouraging an interpretable latent space for long-term vidco prediction. Most similar to our work, Toth et al. (2020) use a Hamiltonian based approach to learning dynamics from images. The work of Toth et al. differs from ours in that do not use a variational auto-encoding neural network, our model is used for systems with configuration space on (3), and our work focuses learning a physically interpretable latent space while Toth et al. have a much higher dimension latent space than the physical configuration space.
2.3 LEARNING DYNAMICS FOR RIGID BODIES
Duong and Atanasov 2021 investigate learning rigid-body dynamics from state-control trajectory data using& a Hamiltonian approach. In their work, Duong and Atanasov focus on systems with configuration spaces on the special orthogonal groups and special Euclidean groups (e.g. SO(2). S0( and SE(3)). They parameterize the inverse mass properties, potential energy, and control input functions, which are estimated with neural networks similar to our method. Duong and Atanasov go on to use the estimated pose dynamics to develop control methodologies for stabilization and tracking using the Iearned model of a quad-rotor system. In contrast, our work focuses on learning the dynamics of free-rotating rigid body with configuration space on SO(3) using images.
3 BACKGROUND
3.1 THE S2 x 52 PARAMETERIZATION OF 3D ROTATION GROUP
The S² S2 parameterization of the 3D rotation group is a surjective and differentiable mapping with a continuous right inverse Falorsi et al. (0118). Define the n--phere: S" {v (m(+1) of + vz ... 2 = 1). and the 3D rotation group: SO(3) = {R E R3x3 R I,, det (R) +1}. where 13 denotes the 3 x 3 identity matrix. The S² x S parameterization of SO(3) is given by (u, v) + (WI, W2, w3) where wi u, W2 u v(u, v), W3 wi X W2, where w, are renormalized to have unit norm. Intuitively, this mapping constructs an orthonormal frame from the unit vectors and by Gram-Schmidt orthogonalization. The right inverse of the parameterization is given by (wi, W2, Wy) + (w,, W2). Other parameterizations of (O(3) such as the exponential map (so (3) + (3))) and the quaternion map (83 - SO(3)) do not have continuous inverses and
3

---

Under review as a conference paper at ICLR 2023
therefore more difficult use in deep manifold regression (Falorsi et al. 2018 Levinson et al. 2020 Brégier 2021).
3.2 3D ROTATING RIGID- BODY KINEMATICS
rotating 3D rigid body's orientation R(t) SO(3) changing over time can be computed from angular velocity (t(t) using the kinematic equations given by the time-rate-of iiee-rate-f-cchanee of R((t) as
dR(() = (t))
dt
where f2x is a 3 3-dimensional skew-symmetric matrix defined by (Ωx ) S x y. with R and is the vector cross-product, For computational purposes. 3D bigid-bdy rotational kinematics are commonly expressed in terms of the quaternion representation q(t) S of the rigid- written in terms of quaternions (Andrle and Crassidis)
orientation (t). The kinematics equation] I 2013) are
dq(t)
Ω2 52
Ω
dt = {()))))((),
Q( =
0
3.3 3D RIGID-BODY DYNAMICS IN HAMILTONIAN FORM
The canonical Hamiltonian formulation derives the equations of motion for mechanical system using only the symplectic form and a Hamiltonian function, which maps the state of the system to its total (kinetic plus potential) energy Goldstein al. (2002). This formulation has been used by several authors to learn unknown dynamics: the Hamiltonian structure (canonical symplectic form) used as a physics prior and the unknown dynamics are uncovered by learning the Hamiltonian (Greydanus et al. 2019 Zhong et al. 2020b] Toth et al. 2020). Consider a system with configuration space I" and a choice n generalizes coordinates that represent configuration. Let z(t) E 2 represent the vector of generalized coordinates and their conjugate momenta at time t. Define the Hamiltonian function H 2 -y IR such that (z) is the sum of the kinetic plus potential energy. Then the equations of motion derive as
dz
On In dt = Acan V2H(2), Acan = (-I,
O,,
where O, E R is the matrix of all zeros and Aran is the matrix representation of the canonical symplectic form Goldstein al. (2002). The equations of motion for a freely rotating rigid body evolve (3) and describe the evolution of R given by equation and the evolution of the angular momentum II given by Euler's equations Goldstein et al. (0022)
dI dt
-
(4)
where J denotes the moment-of offinrtta matrix and II = JΩ. Importantly, the dynamics equation
do not depend on R. This is due to the rotational symmetry it is arbitrary how we assign inertial frame. Due to this symmetry. the Hamiltonian formulation can be reduced using the Lie-Poisson Reduction Thcorem Marsden and Ratiu (1999) to describe the
freely rotating 3D rigid body, i.e.,
time evolution of II on I2 so (3), the Lie coalgebra of SO(3), independent of equation The reduced Hamiltonian h so (3) + for the freely rotating 3D rigid body is its kinetic energy:
I - II
J- II.
(5)
The reduced Hamiltonian formulation Marsden and Ratiu (199)) is
III
dl Aso (I)) Age (3) (II) IIx
(6)
which can be seen to be equivalent to equation 4 The equations equation 6 called Lie-Poisson equations, generalize the canonical Hamiltonian formulation. The generalization allows for different symplectic forms, i.c., (3)*(a) instead of Acan in this case, each of which is only related to the latent space and symmetry. Our physics prior is the generalized symplectic form and learning the unknown dynamics means learning the reduced Hamiltonian. This is generalization of the existing literature
4

---

Under review as a conference paper at ICLR 2023
where dynamics of canonical Hamiltonian systems are learned with the canonical symplectic form as the physics prior. (Greydanus et al.) 2019; Cranmer et al., 2020aa] Chen et al. 2020 Toth et al. 2020). Using the generalized Hamiltonian formulation extends the approach to much larger class of systems than those described by Hamilton's canonical equations, including rotating and translating 3D rigid bodies, rigid bodies in gravitational field, multi-hody systems, and more.
4 LEARNING HAMILTONIAN DYNAMICS ON SO(3)
In this section we outline our approach for learning and predicting rigid-body dynamics from image sequences. The multi-stage prediction pipeline maps individual images to an SO(3) latent space where angular velocities are computed from latent pairs. Future latent states are computed using the generalized Hamiltonian equations of motion and a learned representation of the reduced Hamiltonial Finally, the predicted latent representations are mapped to images giving a predicted image sequence.
4.1 NOTATION
N denotes the number of image sequences in the dataset and T'' the length of each image sequence. Image sequences are written 1 = .......:.... } with I E I, embedded sequences are written . net with
Z {=f--. =²} with = E 3, SO(3) latent sequences are written R2 = RØ E SO(3), and quaternion sequences are written qu = =g"..... ...,,))) with of E S().. Quantities generated with the learned dynamics denoted with hat (e.g., 9). For all sequences [I. N).
4.2 EMBEDDING TO AN SO(3) LATENT SPACE
We embed image observations of rotating rigid body to SO(3) latent space using the composition
of functions o π o Eg I + S0()). The encoding network Eg + R is learned during training, the projection
I + 2 x S² is defined as TT2) (u/llul|, // //[le]]), v E R where 2 = (u v), and the function f SS² x S + SO(3) denotes the surjective and differentiable S x S parameterization of SO(3) (see Section 3.1 which constrains embedded representations the (O manifold where we compute dynamics.
4.3 HAMII.TONIAN DYNAMICS ON SO(3)
We predict future SO (3) latent states using the equations of motion for a freely rotating 3D rigid
body (see equations 2 and 6 and the learned moment of inertial tensor Jg.. We construct the initial
state x = (gb: (T) using the pair of sequential SO(3) latent states (RRg R4) (see Section 4.2 The
quaternion 4 is computed using the implementation of a modified Shepperd's algorithm Markley 2008) proposed (Falorsi et al. 2018) and the angular momentum 11 is computed as 11 2
where the angular velocity 2 is approximated using the algorithm proposed (Barfoot 2017)
(see also Figure Angular Velocity Estimator). The kinematic equations equation 2 are integrated forward using a Runge-Kuul Kutta fourth-order solver (RK45) and a normalization step (Andrle and Crassidis 2013) to ensure the quaternions are valid.
We decode predicted image sequence from the predicted quaternion sequence qu in three steps. We first transform each of to Rt then apply the right inverse of the 8 x S2 parameterization of (3(3), and finally, decode from S x S with the decoding neural network D, which is learned during training.
4.4 LOss FUNCTIONS
In this section we describe each component our loss function: the auto-encoder reconstruction loss Lac, dynamics reconstruction loss Layn. and latent losses functions Liatent, R. Ltatent, IT. Lenergy. The function Cae ensures the embedding to SO(3) sufficiently expressive to represent the image state, and Layn ensures the dynamics consistency between decoded latent states images and their decoded predicted states. Both Limem. and Listent ensure the dynamics consistency of encoded images and their predicted states in the latent space. Lastly, Lenergy enforces the conservation of energy between
5

---

Under review as a conference paper at ICLR 2023
Figure 2: Example training sequences. (Top) Uniform density cube, (bottom) Uniform density prism.
encoded and predicted trajectories. For notational convenience we denote the embedding pipeline : I + S and the decoding pipeline D : S + I
4.4.1 RECONSTRUCTION LOSS
The auto-encoding reconstruction loss is the mean square enror (MSE) between the ground-truth image sequence:
ax NT
ΣΣ
(Do (1)
The dynamics reconstruction loss function is the MSE between the ground-truth image sequence and the predicted image sequence:
N
T
Layn
1
Σ
Σ
1
D ( 112
NT
4.4.2 LATENT LOSS
The latent state loss function is distance metric on SO(3) (Huynh 2009). defined as the MSE between the 3 x 3 identity matrix and right-difference between the encoded latent states and the latent states predict using the learned dynamics:
N
T
Liatent,
I
R (RA)
NT
R= 1=1
additional loss is computed over the angular momentum vectors estimated from the encoded latent states (see Figure and the predicted angular momentum vectors using the MSE loss:
Ltatent.
I enc.
NT
1=1
4.5 ENERGY-BASER LOSS
encourage conservation of energy in the latent representation using the energy conservation loss:
Lenergy
E E
N(T +
E
E
where, E = hIII& J is computed using equation equation
RIGID BODY DATASETS
The lack of exploration 3D learning tasks creates a deficit of datasets for models designed for 3D dynamics. Previous contributions to the study of learning dynamics from images (Greydanus et al.

---

Under review as a conference paper at ICLR 2023
Table 1: Average pixel mean square error over a 30 step unroll on the train and test data on four datasets. All values are multiplied by le+3. We evaluate our model and compare to three baseline models: (1) recurrent model (LSTM (LSTM Hochreiter and Schmidhuber (197)), (2) NeuralODE (Chen et al. (2018)), (3) HGN (Toth et al. (2020). Our model outperforms both baseline models in the
prediction task across the majority of the datasets. The number parameters for the dynamics models
of each baselines are given in the last row of the table.
Ours
LSTM
haseline
Ne
baseline
HON
Datasct
TRAIN
TEST
TRAIN
TEST
TRAIN
TEST
TRAIN
TEST
Uniform Prism
2,66
0.08
3.47
536
0.68
4.18 0.0
7.80
Uniform Cuhe
3.54
0.17
3.97 +: 0.16
21.55 + 1.98
21.64 + 2.12
9.48 + 1.19
9.43 + 1.20
1744 0.00
18.69 0.12 8.33 = 0.26
Non-uniforn Prism
4.27
0.18
6.61 0.88
4.50 1,31
4.52 4: 1.34
4.67
1 0.58
4.75
0.59
6.16 = 0.08
Non uniform Cube
0.35
7,51
7,89
1.50
14.11 0.13
18.14 0.36
Number of Farameters
32400
11400
2019. Toth et al. 2020,
et al. 2020, Zhong and Leonard 2020) have evaluated their models on pixel image sequences of 2D planar dynamics. Their datasets include the pixel pendulum, Acrobot, cart-pole (Zhong and Leonard 2020). as well as 2-body and 3-body problems (Toth et al.
2020). These approaches, which primarily use 2D datasets, learning 3D rigid-body dynamics. Therefore, we created datasets that demonstrate the rich dynamics behaviors of 3D rotational dynamics through images, capable of being used for 3D dynamics learning tasks. We empirically test the performance of our model on the following datasets:
not applicable to problem of
Uniform density cube: Multi-colored cube of uniform mass density
Uniform density prism: Multi-colored rectangular prism with uniform mass density
Non-uniform density cube: Multi-colored cube with non-uniform mass density
Non-uniform density prism: Multi-colored prism with non-uniform mass density
The uniform mass density cube and prism datasets demonstrate baseline capabilities of our approach while the non-uniform datasets are used to demonstrate capabilities of our model pertinent to applications. The uniform mass density datasets can be learned by estimating a diagonal moment of inertia, since the principle axes of rotation align with our defined body axes. These axes are more intuitive to estimate from a visual perspective. For the uniform cube dataset, inspired by Falorsi et al. (2018), the angular momentum vector is constant. For the uniform prism dataset, there exists set of initial conditions that result in dynamics that are not locally bounded, and thus more difficult to learn. The non-uniform density datasets are used to demonstrate the capability of our model to learn dynamics when the principle axes of rotation do not align with the set of intuitive body axes of the object. The physical appearance of the objects match those of the uniform mass density datasets; however, the momentum of inertia in the body axes frame will be non-diagonal. This dataset will validate the model's 's capability to predict changes in mass distribution that may not visible for failure diagnostics due to broken or shifted internal components.
For each dataset N 1000 trajectories were created. Each trajectory consisted of an initial condition (R,, Io) pair that was integrated forward in time using a Python-basee based Runge-Kutte solver for Xg = 100 timesteps with spacing At = 10-3, Initial conditions were chosen such that (Ro, Io) ~ Uniform (SO(3) S²) with H. scaled to have ||ool = 50.
The orientations from the trajectories were passed to Blender Community (2018) to render 28x28 pixel images. Examples of the renderings for both the cube and the rectangular prism can found in Figure For training, each trajectory is shaped into trajectory windows of sequence length = 10, using a sliding window such that each iteration trains on a batch of image sequences of length т.
6 RESULTS
6.1 IMAGE PREDICTION
One key contribution of this work is image prediction for freely rotating rigid bodies using the learned dynamics model. The model is capable of high-accuracy future prediction across four datasets. Figure 3 shows the model's performance on the datasets for both short and long-term predictions. The model's performance on the two uniform density datasets (top) is indicative of its capabilities to
7

---

Under review as a conference paper at ICLR 2023
Tras Soq
Unilomn Mae Danuty
Prodict
Trae
Prodect
Tie
Nowindom Moto heaery
Poefict
45
= 50
Figure 3: Predicted sequences for uniform and non non-uniform mass density datasets given by the model. At prediction time, the model takes the first two images of a sequence, encodes them into our latent space to estimate the angular momentum. The model then predicts and decodes the predicted future states into images of rotating rigid bodies. The prediction results show that the model is qualitatively capable predicting into the future using images.
predict dynamics and map them to image space. The model's ability to accurately predict future image states for the non-uniform density datasets also interesting in that it demonstrates that the model is able to predict states accurately even when the density is visually ambiguous. This is particularly important because mass density (and thus rotational dynamics) is not something that can casily be inferred directly from images. The model is compared three baseline models: (1) an LSTM-bascline, (2) Neural ODE (Chen et al. 2018)-baseline, and (3) the HGN (Toth et al.) 2020) model. Recurrent neural networks like the LST M-baselineeeeeeee provide discrete dynamics model. Neural ODE can be combined with a multi-layer perceptron to model and predict continuous dynamics. HGN is a variational model with a Hamiltonian inductive bias. Architecture and training details for each baseline is given in the appendix B The prediction performance of our model and baselines is shown in Table Our model outperforms the baseline models on most of the datasets with a more interpretable latent space, continuous dynamics, and fewer model parameters -motivating principles for this work.
6.2 LATENT SPACE ANALYSTS AND INTERPRETABILITY
Another contribution of this work is the interpretability of the latent space generated by our model from the image datasets. Black-box models with high-dimensional latent states make very difficult to interpret and inspect the behavior of the latent space. Because our approach encodes all images into a latent space homcomorphic to SO(3), we have convenient ways to interpret this low-dimensional space.
The form our latent space provides us a way of inspecting the behavior of our model that previous works lack. In this sense, approach provides step towards producing new frameworks for interpreting deep learning models, analyzing failure modes, and using control for dynamic systems with significant structure from prior knowledge.
8

---

Under review as a conference paper at ICLR 2023
7 CONCLUSIONS
7.1 SUMMARY
In this work, we have presented the first physics-informed deep learning framework for predicting image sequences of 3D rigid-bodies by embedding the images as measurements in the configuration space (3(3) and propagating the Hamiltonian dynamics forward in time. We have evaluated our approach on new dataset of free-rotating 3D bodies with different inertial properties, and have demonstrated the ability to perform long-term image predictions.
By enforcing the representation of the latent space to be the correct manifold, this work provides the advantage interpretability over black-box physics-inforee informed approaches. The extra interpretability of our approach is step towards placing additional trust into sophisticated deep learning models. This work provides a natural path to investigating how to incorporate- and evaluate the effect cl-ccassica model-based control directly trajectories in the latent space. This interpretability is essential to deploying ML algorithms safety-criticas environments.
7.2 LIMITATIONS
While this approach has shown significant promise, it is important to highlight that this has only been tested in an idealized setting. Future work can examine the effect of dynamic scenes with variable backgrounds, lighting conditions, object geometries, and multiple bodies. Perhaps more limiting, this approach currently relies on the ability to train the model for each system being examined; however future efforts can explore using transfer/few-shur learning between different 3D rigid-body systems.
7.3 POTENTIAL NEGATIVE SOCIETAL IMPACTS
While do not believe this work directly facilitates injury living beings, the inconsistency between the predicted latent representation and ground truth data may lead to unexpected results if deployed in real world environments.
7.4 FUTURE WORK
Although our approach so far has been limited to embedding RGB-images of rotating rigid-bodies with configuration spaces SO(3), is important to note that there are natural extensions to a wider variety of problems. For instance, this framework can be extended to embed different high- dimensional sensor measurements- as point clouds-by only modifying the feature extraction layers of the autoencoder. Likewise, depending be chosen to reflect the appropriate configuration space, such as generic rigid-bodies in SE(3) or systems in more complicated spaces, such the n-jointed robotic arm on restricted subspace of 11 (3).
the rigid-hody system, the latent space can

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amir Ali Ahmadi and Bachir EI Khadir. Learning dynamics systems with side information. In Learning for Dynamics and Control, June 2020.
Christine Allen-Blanche Blanchette, Sushant Veer, Anirudha Majumdar, and Naomi Ehrich Leonard. LagN A Lagrangian neural network for video prediction. arXiv, 2020.
Michael S. Andrle and John L. Crassidis. Geometric integration of quaternions. AIAA, Journal
Guidance and Control, 36(06):1762-1772 2013. doi:
Kendall A. Atkinson. An introduction to numerical analysis. John Wiley & Sons, 1989, ISBN 9780471500230.
Carles Balsells Rodas, Oleguer Canal Anton, and Federico Taschin. [re] hamiltonian generative networks. ReScience C, #1, May 2021. doi: 10.5281/2eeodo..4337777 URL https
//doi. 1rtoiioorg/1..522 5281/ www.de/oooooooooooooooooooooo www.ii/ooe/eeooo.oooo 2tt. 4835278
Timothy Barfoot. Robotic State Estimation. Number 1107159393. Cambridge University Press, 1st edition, 2017.
Romain Brégier. Deep regression manifolds: 3d rotation study. In 2021 International Conference on 3D Vision (3DV), pages 166-174. IEEE, 2021.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932 3937, 3937, 2016. doi: 10. 30.1007/pmas.1517333 1517384113. URL httph:tts///w..
pnas. org doi / abs/ 1073/ pnas. 151 73841 113
Arunkumar Byravan and Fox. SE3-nets: Learning rigid body motion using deep neural networks. In International Conference on Robotics and Automation, 2017.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, and Leon Bottou. Symplectic recurrent neural
networks. In International Conference on Learning Representations, 2020.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 2015.
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. UtLpttp:://// //ww: blender....r....r....... org
M. Cranmer, Sam Greydanus, Stephan Hoyer, Peter W. Battaglia, David N. Spergel, and Shirley Ho. Lagrangian neural networks. In International Conference on Learning Representations, 2020a.
Miles Cranmer, Alvaro Sanzhez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases, 2020b. URL htttts::/axxi..orgg////o//o www.orgggg....oooe/oor
Thai Duong and Nikolay Atanasov, Hamiltonian-based neural ODE networks on the SE(3) manifold for dynamics learning and control. Proceedings of Robotics: Science and Systems, July 2021.
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco Cohen. Explorations homeomorphic variational auto-encoding. International Conference of Machine Learning Workshop on Theoretical Foundations and Application of Deep Generative Models, 2018
Mare Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying Hamiltonian and Lagrangian neural networks via explicit constraints. Conference on Neural Information Processing Systems, 2020.
10

---

Under review as a conference paper at ICLR 2023
Angel Flores-Abad, Ou Ma, Khanh Pham, and Steve Ulrich. A review of space robotics technologies for on-orbit servicing. Progress in Aerospace Sciences, 68:1 2014. ISSN 0376-0421. doi: com/
(Rtttts:: httpp////www.ssccceeeeee......e. sci aeslennce/artiicl//p1////0 //10///2001//0011100011111111130
Herbert Goldstein, Charles P. Poole, and John L. Safko. Classical Mechanics. Addison Wesley, 2002, 2002. ISBN 9780201657029.
Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Conference on Neural Information Processing Systems, abs/ 1ess/1906.1163, 2019.
Jayesh K. Gupta, Kunal Menda, Zhachary Manchester, and Mykel J. Kochenderfer. A general framework for structured learning of mechanical systems. arXiv, 2019.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo Jimenez Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. ArXiv, abs/1812.022300 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Du Q. Huynh. Metrics for 3d rotations: Comparison and analysis. Math Imaging Vis, 35:155-164,
2009. ISSN 0921-8890. doi: 10. 10.1007/11031-00090006111 ed htbp//ccgg/dynge///raaagg htwpaaiiiinn/eeeei....oo pdf
btRp::/s: cs cmu
C. Kane, Jerrold E. Marsden, Michael Ortiz, and Matthew West. Variational integrators and the newmark algorithm for conservative and dissipative mechanical systems. International Journal for Numerical Methods in Engineering, 49:1295-1325, 2000.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. 2014.
Tacyoung Lee. Computational geometric mechanics and control of rigid bodies. PhD thesis, University of Michigan, 2008
Taeyoung Lee, Melvin Leok, and N. Harris McClamroch. Global Formulations of Lagrangian and Hamiltonian Dynamics on Manifolds. 1860-6245. Springer, Cham, 2018.
Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh. and Ameesh Makadia. An analysis of svd for deep rotation estimation. Advances in Neural Information Processing Systems, 13:22554-22565 2020.
Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel Yamins, Jiajun Wu, Joshua Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International conference machine learning, pages 5927-5936. PMLR, 2020.
Michale Lutter, Christian Ritter, and Jan Petter. Deep Lagrangian networks: Using physics as model prior for deep learning. International Conference of Learning Representattons, 2018.
C. Priyant Mark and Surekha Kamath. Review of active space debris removal methods. Space Policy, 47:194-200 2019. ISSN 0265-9646. doi: https://bo..rr///..006///sscec 2018.12.005. URL ht tps:////mmi www.://ceencdireeet...c///eeeeee
6526555461888 3110001311000
Landis Markley. Unit quaternion from rotation matrix. AIAA, Journal Guidance and Control, 31 (02):440-442, 2008. doi:
Jerrold E. Marsden and Tudor S. Ratiu. Introduction to Mechanics and Symmetry. Texts in Applied Mathematics. Springer New York, NY, 1999. ISBN 978-0-387-98643-2.
Jerrold E. Marsden and Matthew West. Discrete mechanics and variational integrators. Acta Numerica, 10:357 514, 2001.
11

---

Under review as a conference paper at ICLR 2023
Jürgen Moser and Alexander P. Veselov. Discrete versions of some classical integrable systems and factorization of matrix polynomials. Communications in Mathematicall Physics. 139(2):217-2))) 1991. doi: 10. N0.1007/BF02352494 /tt/tttss///oo..r///// 1t00:////1...000//2220222222
Samuel E. Outo and Clarence W. Rowley. Linearly-recurrent autoencoder networks for learning dynamics. ArXiv, abs/1772.01378, 2019.
Valentin Peretroukhin, Matthew Giamou, David M. Rosen, W. Nicholas Greene, Nicholas Roy, and Jonathan Kelly. A smooth representation of belief over SO(3) for deep rotation learning with uncertainty. CoRR, abs/2006.01031, 2020.
2000 01031
Peter Toth, Danilo J. Rezende, Andrew Jaegle, S 'ebastien Racanière, Aleksandar Botev, and Irina Hig- gins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
Johan Verbeke and Ronald Cools. The newton-raphson method. International Journal of Mathemati- cal Education in Science and Technology, 26: 177-193, 1995.
Manuel Watter and Martin Riedmiller Jost Tobias Springenberg, Joschka Boedecker. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 27, 2015.
Jeffrey M. Wendlandt and Jemrld E. Marsden. Mechanical integrators derived from discrete variational principle. Physica D: Nonlinear Phenomena, 106:223-246, 1997.
Jared Willard, Xiaowei Jia, Shaoming Xu, Michael S. Steinbach, and Vipin Kumar. Integrating scientific knowledge with machine learning for engineering and environmental systems. 2020.
B. Williams, Antreasian, E. Carranza, C. Jackman, J. Leonard, D. Nelson, B. Page, D. Stanbridge, D. Wibben, K. Williams, M. Moreau, K. Berry, K. Getzandanner, A. Liounis, A. Mashiku, D. Highsmith, B. Sutter, and D. S. Lauretta. OSIRiS- REX Flight Dynamics and Navigation Design. Space Science Reviews, 214(4):69, April 2018. ISSN 1572 doi: 12100/01111-118111111 URL https:///a....cg//....0/////////
0501-x
Linhui Xiao, Jinge Wang, Xiaosong Qiu, Zheng Rong, and Xudong Zou. Dynamic-samm: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomous Systems, 117:1-16 2019. ISSN 0921-8890. doi: https://doi.org/10.101 1016/ 2.1obot.219.03.000 032011. URL https: www. www.w/wecweeeeeeeeccooo http://eee...ooocccceeee/erat.oa / p11/50921889018300
Yaofend Desmond Zhong and Naomi Ehrich Leonard. Unsupervised learning of Lagrangian dynamics from images for prediction and control. In Conference on Neural Information Processing Systems, 2020.
Yaofend Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Dissipative SymODEN: Encoding Hamiltonian dynamics with dissipation and control into deep learning. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations. 2020a.
Yaofend Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learning Hamiltonian dynamics with control. In International Conference Learning Representations, 2020b.
Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. 2019 IEEE/CVF Conference Computer Vision and Patter Recognition (CVPR), pages 5738-5744 5746, 2019.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 AUTO-ENCODER ARCHITECTURE
Tables and give the architecture of the encoder/decoder neural networks. The encoder and decoder combine convolutional and linear layers map from images the desired latent space. The nonlinear activation function used in the auto-encoding neural network is the exponential linear unit (ELU). This activation function chosen for continuity, well prevent vanishing and exploding gradients as shown in Otto and Rowley (2019).
Table 2: Encoder architecture.
Encoder Layers
Layer Number
Layer Name
Input Channel Size
Output Channel Size
Kernel/Stride Size
Conv2d
3
16
3
2
ELU
N/A
N/A
N/A
3
Conv2d
16
16
3
ELU
N/A
N/A
N/A
5
MaxPOOL
N/A
N/A
2/2
6
BatchNorm2d
16
N/A
N/A
7
Conv2d
3
16
3
8
ELU
N/A
N/A
N/A
9
Conv2d
16
16
3
10
ELU
N/A
N/A
N/A
11
MaxPOOL
N/A
N/A
2/2
12
BatchNorm2d
32
N/A
N/A
13
Flatten
N/A
N/A
N/A
14
Linear
32*4#4
120
N/A
15
ELU
N/A
N/A
N/A
16
BatchNorm22
120
N/A
N/A
17
Linear
120
N/A
18
ELU
N/A
N/A
N/A
19
Linear
84
N/A
Table 3: Decoder architecture.
Decoder Layers
Layer Number
Layer Name
Input Channel Size
Output Channel Size
Kernel/Stride Size
Linear
84
N/A
2
ELU
N/A
N/A
N/A
3
Linear
84
120
N/A
4
Narch
120
N/A
N/A
5
ELU
N/A
N/A
N/A
6
Linear
120
32#4#4
N/A
7
Unflatten (32, 4, 4)
N/A
N/A
N/A
8
BatchNorm2d
32
N/A
N/A
9
MaxUNPOOL
N/A
N/A
2/2
10
ELU
N/A
N/A
N/A
11
Conv Crnn/raanpose2d
16
12
ELU
N/A
N/A
N/A
13
Conv Transpose2d
16
3
14
BatchNorm2d
16
N/A
N/A
15
MaxUNPOOL
N/A
N/A
2/2
16
ELU
N/A
N/A
N/A
17
Conv Transpose2d
16
16
18
ELU
N/A
N/A
N/A
19
Cony ConyTransposed
16
3
13

---

Under review as a conference paper at ICLR 2023
B BASELINES
We compare the performance of our model against three haseline architectures: (1) an LSTM baseline, (2) Neural ODE (Chen al. 2018) baseline, and the Hamiltonian Generative Network (HGN) (Tohh et al. 2020). The first two architectures use the same encoder-decoder backbone as our model (see Tables and 3) the HGN is trained with the architecture described their work. The baselines differ from our approach in the way the dynamics are computed.
B.I LSTM BASELINE
The LSTM- -STM-laslli uses an LSTM network to predict the dynamics. The layer LSTM network with input dimension of 6 and a hidden dimension of 50. The hidden state and cell state are randomly initialized and the output of the network is mapped to a 6-dimensional latent vector by learned linear transformation. We train the LSTM-baseline to predict single step from 9 sequential latent vectors by minimizing the sum of the autoencoder and dynamics losses, Cac and Layn defined in equations 4.4.1 and and the MSE loss between the encoder generated latent vectors and the LSTM-baseline latent vectors. The baseline is trained using the hyper-pane parameters in Tables 8 and 6 At inference we use recursive strategy to predict farther into the future. The is given in figure and the quantitative performance terms of pixel mean-square error given Table The total number of parameters in the network is 52400.
is a three-
qualitative performance for the
True Seq.
Liiffrm Density
Truc Seq.
Predict
True Seq.
Non unifom Densty
Predict
True Seq.
1=2
Figure 4: Predicted sequences for cettform/noouuniffoo prism and cube datasets given by the LSTM- baseline. At prediction time, the model takes the first encoded latent states and predicts a sequence of 20 time steps recursively. The figure shows time steps 10-19, which the first 10 predictions of the model. The LSTM-baseline has poorer performance than the proposed approach on all datasets.
B.2 NEURAL ODE (CHEN ET AL. 2018 BASELINE
The Neural ODE-baseline uses Neural ODE (Cen et al. 2018) framework predict dynamical updates. The Neural ODE-baseline is three layer MLP with ELU et al. (2015) activations. The baseline has an input dimension of 6, a hidden dimension of 50, and an output dimension We train the Neural ODE-baseline to predict sequence of latent vectors from an single latent vector input by minimizing the sum of the autoencoder and dynamics losses, Ca and Layn as defined in equations I and 7 and the MSE loss between the encoder generated latent vectors and the Neural ODE-baseline latent vectors. We use the RK4-integrator to integrate the dynamical update and the hyper-parameters in Tables
and 6 The qualitative performance for the DeuralODE-haseliing is
14

---

Under review as a conference paper at ICLR 2023
given in figure and the quantitative performance in terms of pixel aecansquaree given in table The total number of parameters in the network 11406.
Sey.
Uniform Mass Density
Predict
Tru Soq.
True Seq.
Not- Cont-aaifom Mass Densiy
Predict
Trac Seq.
Prodict
r=0
7=1
t=2
1=3
7=4
r=5
T=6
T=7
T=8
Figure 5: Predicted sequences for uniform/non-miifon prism and cube datasets given by the Neural ODE-baseline. At prediction time, the model takes the first encoded latent states and predicts sequence of 10 steps recursively. The Neural ODE-baseline has poorer performance than the proposed approach on all datasets.
B.3 HAMILTONIAN GENERATIVE NETWORK (HGN)
HGN (Toth et al. 2020 uses combination of variational auto-encoding neural networks, transformer, and Hamiltonian dynamics do video prediction. For our comparison experiments, implementation Balsells Rodas et al. (202T). During training, the same training hyperparameters are used described in Toth et al. (2020) We train HGN (i.e. reconstruction and KI. divergence). We use the Leap-frog integrator as this provided their model with its best performance on their datasets. The qualitative performance for HGN on our datasets is given in figure ??, and the quantitative performance in terms of pixel mean-square error given in table
the
four datasets using their loss function
C ABLATION STUDY
In this ablative study we explore the impacts of the dynamics loss 7 and energy loss 4.5 on the performance of the proposed model. The ablated model is trained similarly the proposed model except the dynamics (or energy) loss not present the total loss. From table[aane figure it can be seen that removing the dynamics loss negatively affects the performance of our proposed model. Inspecting table it can be seen that removing energy loss has a minimal degradation effect on model performance. These results are further corroborated in the literature Watter and Jost Tobias Springenberg 2015).
al. 2020
D EXPERIMENT
D.1 UNIFORM MASS DENSITY CUBE
3. 0.
0.7
0.
0.
cuhe
00. 0. 3.
15

---

Under review as a conference paper at ICLR 2023
Table Average pixel MSE over 30 step unroll on the train and test data on four datasets for our ablative study. All values are multiplied by le+3. We evaluate our model and compare to version of our model without the dynamics loss 7 and without the energy-base loss. Our full model outperforms the ablated models in the prediction task across all datasets.
Dataset
Ours
Ours dynamics TRAIN
Ours
energy
TRAIN
TEST
TEST
TRAIN
TEST
Uniform Prism
2.66
0.10
2.71 + 0.08
6.41 2.02
6.40 1.98
3,03
1.26
3.05
+ 1.21
Uniform Cube
3.54
0.17
3,97 1 0.16
11.30 = 1.47
11:30 - 1.51
4.13
2.14
4.62 - 2.02
Non-uniform Prism
4.27
0.18
6.61
0.88
8.80 2.51
8.75 2.51
4.98
1.26
7.07
1.88
Non-uniform Cube
6.24
0.29
4.85 + 0.35
13.84
13.83 + 1.66
7.27
1.06
5.65 +
True Seç.
Unitorm Mass Density
Prodict
True Sog.
Prodict
True Seq.
Non-amffor Mast Deasity
Prodict
True Seq.
Predict
T=1
T=2
x=5
r=6 6
T=7
T=8
T=
Figure 6: Predicted sequences for uniform/non-unifoon prism and cube datasets given by the ablated version of our proposed model. At prediction time, the model takes the first encoded latent states and predicts sequence of 10 steps recursively. The ablated model has poorer performance than the proposed approach on all datasets.
The training parameters used to run this experiment with the train_der py file is given in table
Table 5: Hyper-parameters used train model for the uniform mass density cube experiment. Only values differing from default values are given in the table.
Uniform Cube Experiment
Parameter Name
Value
seed
17
single-gpu
True
test_split
0.2
val_split
0.1
n_epoch
1000
batch_size
256
learning rate ae
1E-3
learning rate rate_dyn
1E-3
seq_len
10
time_step
1E-3
loss loss_gamma
1. 1. 0.1 0.1
16

---

Under review as a conference paper at ICLR 2023
D.2 UNIFORM MASS DENSITY PRISM
2.4
0.
0.
0.
0.71
0.
prism
0.
0.
0.6
The training parameters used to run this experiment with the ccrain_dev .ev. file is given in table
Table 6: Hyper-paramettn used to train model for the uniform mass-density prism experiment. Only values differing from default values are given in table.
Uniform Prism Experiment
Parameter Name
Value
seed
17
single-gpu
True
test_split
0.2
val_split
0.1
e_epoch
1000
batch_sizz
256
learning_ratee rate_ae
1E-3
learning_rate_dyy
1E-3
seq_len
10
time_stop
1E-3
loss jas_gamma
1. 1. 1. 0.1 0.1
D.3 NON-UNIFORM DENSITY CUBE
4.52677669
-2.61906366
-0.436510611
nu-cube
2.61906366
1.34388683
-0.77601886
0.443651061
-0.77601886
-0.12933648
The training parameters used to run this experiment with the train_dev. file is given in table
7
Table 7: param-parsmeters used train model for the non-uniform mass density cube experiment. Only values differing from default values are given in table.
Non-uniform Cube Experiment
Parameter Name
Value
seed
17
single-gpu
True
test_split
0.2
val_split
0.1
n_epoch
1000
batch_size
256
learning rate ae
1E-3
learning rate rate_dyn dyn
1E-3
seq_len
10
time_step
1E-3
loss_gamma
1.
1. 0.1
D.4 NON-UNIFORM DENSITY PRISM
22.36999579 0.12136536
0.12136536
0.394437421
0.67762326
0.2022756
mu-prism
00.39443742
0.2022756
0.6573957
The training parameters used to run this experiment with the train_dewy py file is given in table
17

---

Under review as a conference paper at ICLR 2023
Table Hyper-parameters used to train model for the non-uniform mass density prism experiment.
Only values differing from default values are given in table.
Non-uniform Prism Experiment
Parameter Name
Value
seed
17
single-gpu
True
test_split
0.2
val_split
0.1
e_epoch
1000
batch_ _ize
256
learning_rate__ee
1E-3
learning_rate_dyy
1E-3
seq_len
10
time steettteppppp
1E-3
loss_gammas
1. 1. 0.1 0.1
E ANGULAR VELOCITY ESTIMATOR
The angular velocity is estimated using two sequential images frames, To and 71. The image frames encoded into two latent states, Ro and R1. These latent states are the orientation matrices in the body-fixed frame. The angular velocity between frame 0 and frame is calculated in two parts: (I) the unit angular velocity vector and (2) the angular velocity magnitude. The unit vector and magnitude are estimated as shown Figure
F COMPUTE RESOURCES & GPUs
The models are trained on a server with NVIDIA A100 SXM4 GPUs. The processor is an AMD EPYC 7763, with 64 cores, 128 threads, 2.45 GHz base, 3.50 GHz boost, 256 MB cache, PCIe 4.0.
G
CODE REPOSITORY & DATASET
For access the code repository, go to the Github link: https:// ctm/ ctt. 8msos// LearniiagS03sfrommmaaass For access to data used in this work please see link at https:// www.droobox.ccom http://wwwwo.....ooooooooooooce sh httamm/sh/meeeee 1 9mguul wh /ttp///oooostttttteeeeeeeeecco 7a7d111
18


Reviews:
[{"review_id": "nuUCV7SZTzH", "reviewer": "Reviewer_14VD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to estimate 3D rotations of an object from images using classical physics. The paper is focused on predicting  satellite rotations. It uses Hamiltonian representation of rotational motion. With this representation, motion trajectory is differentiable everywhere. This is not true for rotation parametrisation using exponential or quaternion maps as the inverses may be discontinuous. The paper provides a 3D dataset of uniform and non-uniform cube and prism to train the method.\nThe method uses a set of sequential images to estimate rotation and angular velocities. It encodes images to low-dimensional space representing rotations (learnt during training phase) which are used to estimate angular velocity and predicts next image using Hamiltonian dynamics.", "summary_of_the_review": "The paper is definitely interesting. My only concern is the lack of evaluation on real data and the inability of method to deal with non-uniform data. \n\nAfter reading the rebuttal and discussion with other reviewers, I have changed my opinion (a bit unfavorably). Here are the main concerns:\n\n1. The experimentation is highly limited. The discussion on handling non-linear mass distribution is non-existent. The authors provide some explanation in the rebuttal but a lot questions remain. Given that the setup is difficult, perhaps the authors could explore non-linearity a bit more. For example, how does the method behave when the mass is non-uniform in a way that the centre of mass deviates strongly with respect to object centre. This discussion could have helped to understand the practical limitations of the method better. Plus, the authors must experiment with more data. Right now, the datasets are limited to a cube and a prism.\n\n2. There were some gaps in writing. The figure 2 in initial draft, which shows the trajectory to drift apart drastically, was later on removed. It is unclear to the readers how the registration works if the estimation of trajectory is so wrong. It is not clear whether the authors are performing a global alignment to fit into the projection. Perhaps the authors should have clarified.\n\nOverall, the paper has an important contribution over sota but the technical novelty is limited. Without a strong experimental evaluation, it was difficult to have an undoubted positive opinion.", "strength_and_weaknesses": "Strengths:\n1. The use of hamiltonian dynamics is novel to the context of rotation estimation and trajectory prediction of satellites.\n2. A dataset has been proposed in order to learn 3D dynamics through images.\n3. The experiments show convincing results.\n\nWeakness:\n1. No evaluation on real data.\n2. In figure 4, it seems that the rotation estimation starts well and drifts away quickly. The rotation trajectory prediction is very off while the image prediction is quite decent. Do the images in fugure 3 (upto t=50) represent a small portion of the trajectory show on figure 4. More details are needed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_hJfAXkNesh", "reviewer": "Reviewer_RcCn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes a method for estimating and predicting 3D rotation dynamics from image sequences of freely rotating 3D rigid bodies such as satellites. The proposed method maps individual images to latent representations isomorphic to SO(3) using auto encoder type neural networks, calculates angular velocities from the latent pairs, and then leams the moment of inertia tensor J in the Hamiltonian. Based on this prediction of future latent states, a predictive image is constructed using the auto encoder. After constructing a new rotating rigid body dataset consisting of a sequence of rotating cubes and rectangles, the authors show that the proposed method gives higher performance than LSTM and NeuralODE.", "summary_of_the_review": "The importance of the research is high because it is a challenging task to realize a neural network method that learns a physical model capable of predicting video images. However, the performance of the method has not been compared to similar more general-purpose methods, which makes its effectiveness questionable. Therefore, I believe this concern needs to be resolved to accept this manuscript.", "strength_and_weaknesses": "Strengths\n\nThis research is impressive because it is a challenging task to realize a neural network method that learns a physical model capable of predicting video images. The fact that they have succeeded in predicting and reconstructing video images to some extent by incorporating a physical inductive bias specific to rotational motion into the model is also highly novel.\n\n\n\nWeaknesses\n\nThe following two points are considered weaknesses\n\n\nThe advantage over previous research is unclear:\n\nHamiltonian Generative Networks (HGN) [1] has been shown to be able to learn up to 400 steps in the future for a two-body problem [2].\nHGN is also more general-purpose than the proposed method.\nThe authors need to show that the proposed method has better prediction performance for videos than HGN.\n\n[1] Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, Irina Higgins, \"Hamiltonian Generative Networks,\" ICLR 2020, 2019.\n\n[2] Irina Higgins, Peter Wirnsberger, Andrew Jaegle, Aleksandar Botev, \"SyMetric: Measuring the Quality of Learnt Hamiltonian Dynamics Inferred from Vision,\" NeurIPS 2021, 2021.\n\n\n\nThe purpose of the method is unclear:.\n\nThe author writes that the application of the study is to estimate the orientation of satellites, but the training data in this study assigns a different color to each side of the object. I think it does not match the configuration of the satellites. If the differences between each side of a satellite can be determined, the orientation and motion of a satellite with a known geometry can be estimated analytically. Therefore, the proposed method is not considered necessary to achieve the objectives of the study.\n\nAlso, the work [Duong and Atanasov (2021)] cited by the authors in section \"2.3 LEARNING DYNAMICS FOR RIGID BODIES\" to achieve more general learning of rigid body motions by estimating inverse mass properties could be used for satellite orientation estimation, but no comparative experiments have been conducted. \n\nThese factors made me think that the relationship between the proposed method and the purpose of the study did not seem appropriate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OizowNN51lB", "reviewer": "Reviewer_7a5H", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presented a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. The target is achieved by using a multi-stage prediction pipeline that maps individual images to a latent representation homeomorphic to SO(3), computes angular velocities from latent pairs, and predicts future latent states using the Hamiltonian equations of motion with a learned representation of the Hamiltonian.", "summary_of_the_review": "The paper presented a physics-informed neural network model to estimate and predict 3D rotational dynamics from image sequences. However, the current experimental evaluation and ablation studies are insufficient to justify the paper. The novelty/contribution of the proposed method over existing work has not been fully justified.\n", "strength_and_weaknesses": "Strength: \n\n+ A physics-informed deep learning framework for predicting image sequences of 3D rigid-bodies by embedding the images as measurements in the configuration space SO(3) and propagating the Hamiltonian dynamics forward in time\n\n+ A new rotating rigid-body dataset with sequences of rotating cubes and rectangular prisms with uniform and non-uniform density.\n\nWeaknesses:\n\n- As the paper acknowledged, the paper only evaluated the proposed method in a rather idealized setting. To conclude the current experimental evaluation and ablation studies are insufficient to justify the paper.\n\n- The proposed approach currently relies on the ability to train the model for each system being examined.\n\n- The secod paragraph of the Summary section seems to be overclaiming its contributions without sufficient justification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6H7J8UOBag", "reviewer": "Reviewer_togy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a method to learn dynamics and make predictions from sequences of images of 3D rigid bodies. The method incorporates physics priors by using SO(3) as the latent space, and by translating the problem of learning the dynamics to that of learning a (reduced) Hamiltonian. Predictions are then made by and integrating the kinematic equations on SO(3), and Lie-Poisson equations on the Lie coalgebra of SO(3).\n\nThey perform experiments with a synthetic dataset consisting of sequences of rotating rigid bodies with different shapes and different, diagonal and non-diagonal, inertia tensors.", "summary_of_the_review": "The paper propose a method to learn dynamics of freely rotating rigid bodies based on the simple idea that in the Hamiltonian framework, rigid bodies dynamics happens on $T^{*}SO(3)$. This makes learned dynamics interpretable and is a concrete step towards more trustworthy, less black-box machine learning models. Despite I asked for a few clarifications, I think the paper is overall clear.", "strength_and_weaknesses": "The methods makes good use of well-known classical mechanics results about freely rotating rigid bodies giving good theoretical reasons for why the method should work. Interpretability of the learned dynamics is of utmost importance when using machine learning to model dynamical systems and the proposed algorithm gives very interpretable results constraining the dynamics on SO(3) and leaving all the unknown and irrelevant characteristics, such as color and shape of the modelled object, to the autoencoder.\n\nUnfortunately, as stated in the paper, there aren't many datasets to assess performances for the particular problem tackled in this paper. \nThe dataset could also represent a good benchmark for future works if made available, and therefore is a notable contribution.\nThe comparison with LSTMs and Neural ODEs is really meaningful as these methods are not as domain-specific as the proposed algorithms but certainly show how inductive biases like the one proposed in this paper are the way to go. Is there any \n\nApart from the dynamics, the SO(3) latent space seems to facilitate the reconstruction: perhaps a pre-training of just the encoder-decoder architecture, which is not unrealistic for real-world application, would boost performances. Or perhaps you already do that.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject