KeyCLD: Learning Constrained Lagrangian Dynamics in Keypoint Coordinates from Images
Authors: Rembert Daems
Jeroen Taets
Francis wyffels
Guillaume Crevecoeur
Venue: Submitted to ICLR 2023 (2023)
Keywords: Lagrangian
dynamics
keypoints
images
unsupervised

Abstract: We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoint representations derived from images are directly used as positional state vector for jointly learning constrained Lagrangian dynamics. KeyCLD is trained unsupervised end-to-end on sequences of images. Our method explicitly models the mass matrix, potential energy and the input matrix, thus allowing energy based control. We demonstrate learning of Lagrangian dynamics from images on the dm_control pendulum, cartpole and acrobot environments, wether they are unactuated, underactuated or fully actuated. Trained models are able to produce long-term video predictions, showing that the dynamics are accurately learned. Our method strongly outperforms recent works on learning Lagrangian or Hamiltonian dynamics from images. The benefits of including a Lagrangian prior and prior knowledge of a constraint function is further investigated and empirically evaluated.

KEYCLD: LEARNING CONSTRAINED LAGRANGIAN DYNAMICS IN KEYPOINT COORDINATES FROM IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoint representations derived from images directly used posi- tional state vector for jointly learning constrained Lagrangian dynamics. KeyCLD is trained unsupervised end-to-end sequences of images. Our method explicitly models the mass matrix, potential energy and the input matrix, thus allowing energy based control. We demonstrate learning of Lagrangian dynamics from images the din_cont rol pendulum, cartpole and acrobot environments, wether they are unactuated, underactuated fully actuated. Trained models are able to produce long-term video predictions, showing that the dynamics are accurately learned. Our method strongly outperforms recent works on learning Lagrangian or Hamiltonian dynamics from images. The benefits of including Lagrangian prior and prior knowledge of constraint function further investigated and empirically evaluated.
to
0
(b)
(c)
(d)
Figure 1: KeyCLD learns Lagrangian dynamics from images. (a) An observation of dynamical system is processed by a learned keypoint estimator model. (b) The model represents the positions of the keypoints with set of spatial probability heatmaps. (c) Cartesian coordinates
extracted using spatial softmax and used as positional state vector to learn Lagrangian dynamics. (d) The information in the keypoint coordinates bottleneck suffices for learned renderer model to reconstruct the original observation, including background, reflections and shadows. The keypoint estimator model, Lagrangian dynamics models and renderer model are jointly learned unsupervised on sequences of images.
INTRODUCTION AND RELATED WORK
Learning dynamical models from data is crucial aspect while striving towards intelligent agents interacting with the physical world. Understanding the dynamics and being able to predict future states is paramount for controlling autonomous systems or robots interacting with their environment. For many dynamical systems, the equations of motion can be derived from scalar functions such as the Lagrangian Hamiltonian. This strong physics prior enables more data-efficient learning and holds energy conserving properties. Greydanus et al. (2019) introduced Hamiltonian neural networks. By using Hamiltonian mechanics inductive bias, the model respects exact energy conservation laws. Lutter et al. (2018; 2019) pioncered the use Lagrangian mechanics as physics priors for general learning dynamical models from data. Cranmer et al. (2020) expanded this idea

---

setting. By modelling the Lagrangian itself with neural network instead of explicitly modelling mechanical kinetic energy, they can model physical systems beyond classical mechanics. Zhong et al. (2020) included external input forces and energy dissipation, and introduced energy-based control by leveraging the learned energy models. Finzi et al. (2020) introduced learning of Lagrangian or Hamiltonian dynamics in Cartesian coordinates, with explicit constraints. This enables more data efficient models, at the cost of providing extra knowledge about the system in the form of constraint function.
Learning Lagrangian dy dynamics from images It is often not possible to observe the full state of a system directly. Cameras provide a rich information source, containing the full state when properly positioned. However, the difficulty lies in interpreting the images and extracting the underlying state. As was recently argued by Lutter & Peters (2021), learning Lagrangian or Hamiltonian dynamics from realistic renderings remains an open challenge. The majority of related work (Greydanus et al., 2019; Toth et al., 2020; Saemundsson et al., 2020; Allen-Blanchette et al., 2020; Botev et al., 2021) use variational auto-encoder (VAE) framework to represent the state in a latent space embedding. The dynamics model is expressed in this latent space. Zhong & Leonard (2020) use interpretable coordinates, however need full knowledge of the kinematic chain, and the images are segmented per object. Table provides an overview of closely related work in literature.
Table 1: An overview of closely related Lagrangian or Hamiltonian models. Lag-caVAR (Zhong & Leonard, 2020) is capable of modelling external forces and learning from images, but individual moving bodies need to be segmented in the images, on a black background. additionally needs full knowledge of the kinematic chain, which is more prior information than the constraint function necessary for our method (see Section 2). HGN (Toth et al., 2020) needs no prior knowledge of the kinematic chain, but is unable to model external forees. CHNN (Finzi al., 2020) expresses Lagrangian or Hamiltonian dynamics in Cartesian coordinates, but can not be learned from images. Our method, KeyCLD, is capable of learning Lagrangian dynamics with external forces, from unsegmented images with shadows, reflections and backgrounds.
HGN
Lag-ca VAE
CHNN KeyCLD
External forces (control) Interpretable coordinates
Cartesian coordinates
Learns from images
Learns from unsegmented images
V
Needs kinematic chain prior
V
Needs constraint prior
Keypoints Instead of using VAE inspired latent embeddings, our method leverages fully convo- lutional keypoint estimator models to observe the state from images. Because the model is fully convolutional, it is also translation equivariant, this leads to higher data efficiency. Objects can be represented with one or more keypoints, fully capturing the position and orientation. Zhou et al. (2019) used keypoints for object detection, with great success. Keypoint detectors used for human pose estimation (Zheng et al., 2020). More closely related to this work, keypoints can be learned for control and robotic manipulation (Chen et al., 2021; Vecerik et al., 2021). Minderer et al. (2019) learn unsupervised keypoints from videos to represent objects and dynamics. Jaques et al. (2021) leverage keypoints for system identification and dynamic modelling. Jakab et al. (2018) learn a keypoint representation unsupervised by using it as an information bottleneck for reconstructing images. The keypoints represent semantic landmarks the images and generalise well to unseen data. It is the main inspiration for the use of keypoints in our work.
commonly
We introduce KeyCLD, a framework to learn constrained Lagrangian dynamics from images. We are the first to use learned keypoint representations from images to learn Lagrangian dynamics. We show that keypoint representations derived from images can directly be used as posi- tional state vector for learning constrained Lagrangian dynamics, expressed in Cartesian coordinates. (2) We show how to control constrained Lagrangian dynamics Cartesian coordinates with energy shaping, where the state is estimated from images. (3) We adapt the pendulum, cartpole and acrobot
Contributions (I)
2

---

systems from dm.control (Tunyasuvunakool et al., 2020) benchmarks for learning Lagrangian or Hamiltonian dynamics from images. (4) We show that KeyCLD can be learned on these systems, wether they unactuated, underactuated or fully actuated. We compare quantitatively with Lag- CAV,E, lag- VAE (Zhong & Leonard, 2020) and HGN (Toth et al., 2020), and investigate the benefit of the Lagrangian prior and the constraint function. KeyCLD performs best on all benchmarks.
CONSTRAINED LAGRANGIAN DYNAMICS
Lagrangian Dynamics For dynamical system with degrees of freedom, set of independent generalized coordinates q E IR' represents all possible kinematic configurations of the system. The time derivatives q E I are the velocities of the system. If the system is fully deterministic, its dynamics are described by the equations of motion, set of second order ordinary differential equations (ODE):
(I) where ul) are the external forces acting on the system. From a known initial value (q, q). we can integrate f through time to predict future states of the system. It is possible model f with neural network, and train the parameters with backpropagation through an ODE solver (Chen et al., 2018).
q = f(q(),, f, ((f),
However, by expressing the dynamics with a Lagrangian we introduce a strong physies prior (Lutter et al., 2019):
L( a) T'q a) (g))
(2)
T is the kinetic energy and V is the potential energy of the system. For any mechanical system the kinetic energy defined as:
1 M(q)á
T(q q)
2
where M(q) Rmxm is the positive semi-definite mass matrix. Ensuring that M(q) is positive semi-definite can be done by expressing M(q) = L L(q)L(q)T, where L(q) is lower triangular matrix. is now possible describe the dynamics with two neural networks, one for the mass matrix and one for the potential energy. Since both are only in function of and not q. and expressing the mass matrix and potential energy more straightforward than expressing the equations of motion, is generally much more simple to learn dynamics with this framework. In other words, adding more physics priors in the form of Lagrangian mechanics, makes learning the dynamics more robust and data-cfficient (Lutter et al., 2018; 2019; Cranmer et al., 2020; Lutter & Peters, 2021).
The EulernLagrange equations (4) allow transforming the Lagrangian into the equations motion by solving for q:
d
V - = V dt
W.W = =g(q)
(5)
where W is the external work done on the system, e.g. forces applied for control. The input matrix E m allows introducing external forces u E R for modelling any control-affine system. If the external forces and torques are aligned with the degrees of freedom q. g can be a diagonal matrix or even identity matrix. More generally, if no prior knowledge is present about the relationship between u and the generalized coordinates g(q) R.m + Rmxl is a function of q and can be modelled with third neural network (Zhong et al., 2020). If the system is fully actuated i m, if it. is underactuated I < m.
Cartesian coordinates Finzi et al. (2020) showed that expressing Lagrangian mechanics in Carte- sian coordinates x E RR* instead of independent generalized coordinates E Rm has several advantages. The mass matrix no longer changes in function of the state, and is thus static. This means that neural network is no longer required to model the mass matrix, simply the values in the matrix itself are optimized. The potential energy Vxx) and input matrix g(x) are now in function of Expressing the potential energy in Cartesian coordinates often be simpler than in generalized coordinates. E.g. for gravity, this is simply a linear function.
To use the Euler- Lagrange equations without constraint forces, it is required that the system is expressed in independent generalized coordinates, meaning that all possible values of q correspond to possible states of the system. Since we now expressing the system in Cartesian coordinates,
3

---

eyx1 - |x] x1
((x)
Figure Example of constraint function f(x) to express the caripole system with Cartesian coordinates. The cartpole system has 2 degrees of freedom, but is expressed in x E IR. Valid configurations of the system in RR4 are constrained on manifold defined by = (x). The first constraint only allows horizontal movement of X1, and the second constraint enforces constant distance between X and X2. Although unknown [ and l2 constants are present in (x), their values are irrelevant, since only the Jacobian of (xx) is used in our framework (sce equation (6)). See Appendix A.4 for more examples of constraint functions.
this requirement no longer holds. We need additionally a of holonomic constraint functions (x) : R + _ where is the number of constraints so that the degrees of freedom are correct: m n.. Deriving the equations of motion including the holonomic constraints yields (see Appendix A.1 for the full derivation and details):
f = V +u
(6)
3 DDM T DDMM (
with D being the Jacobian operator. Since time derivatives of functions modelled with neural networks are no longer present, equation (6) can be implemented in an autograd library which handles the calculation of gradients and Jacobians automatically. See Appendix A.2 for details and the implementation of equation (6) in JAX (Bradbury et al., 2018).
Note that equation (6) only the Jacobian (xx) is present. This means that there is no need to learn explicit constants in (x), such as lengths or distances between points. Rather that constant distances and lengths through time are enforced by (*) = We use this property to our advantage since this simplifies the learning process. See Fig. 2 for an example.
The constraint function (xx) adds extra prior information to our model. Altemnttiely,, we could use mapping function F(q). This leads directly to expression of the Lagrangian in Cartesian coordinates using x = (F(qq
C(q. 4)
DF(q) (MDF(q)4 V(F(g))
from which the equations of motion be derived using the Euler- LagranLag equations, similar to equation (6). In terms of explicit knowledge about the system, the mapping x = F(q) is equivalent to the kinematic chain as required for the method of Zhong & Leonard (2020). Using the constraint function is however more general. Some systems, such as systems with closed loop kinematics, can not be expressed in generalized coordinates q. and thus have mapping function (Betsch, 2005). We therefore argue that adopting the constraint function (x) is more general and requires less explicit knowledge injected in the model.
Relationship between Lagrangian and Hamiltonian Both Lagrangian and Hamiltonian me- chanics ultimately express the dynamics in terms of kinetic and potential energy. The Hamiltonian expresses the total energy of the system H (q. p) = T(q. p) + Vq)) (Greydanus et al., 2019; Toth al., 2020). It expressed in the position and the generalized momenta (q. p), instead of generalized velocities. Using the Legendre transformation it is possible to transform L into H or back. We focus our work on Lagrangian mechanics because is more general (Cranmer et al., 2020) and observing the momenta p is impossible from images. See also Botev et al. (2021) for short discussion on the differences.
LEARNING LAGRANGIAN DYNAMICS FROM IMAGES
Keypoints state representations We introduce the of keypoints to leam Lagrangian dynam- ics from images. KeyCLD is trained unsupervised on sequences of n images {z'}, i (1....,n) and constant input vector See Fig. 3 for a schematic overview.
4

---

keypoint
spanaal
X
s
z
keypoint estimator
spattal sottmax
Gaussian biobs
rendener model
z
keypaint
spelial
Gaussian
renderer
time
2
g
X
z
Ld
Figure 3: Schematic overview of training KeyCLD, A sequence of images {z'},i {I...., .., n2, is processed by the keypoint estimator model, returning heatmaps {s'} representing spatial probabilities of the keypoints. s' consists of m heatmaps s one for every keypoint x h E {i..... m). Spatial softmax is used to extract the Cartesian coordinates of the keypoints, and all keypoints are concate- transformed back spatial representation s using Gaussian the keypoint estimator model by binary cross-entropy loss Le
nated in the state vector x'. blobs. This prior is encouraged between s' and s". The renderer model reconstructs images z based on s"i with reconstruction loss I, The dynamics loss La is calculated model, renderer model and the dynamics models (mass matrix, potential energy and input matrix) are jointly trained with weighted sum of the losses L = L, L + ALa-
the sequence of state vectors x Keypoint estimator
All images z' in the sequence are processed by the keypoint estimator model, returning each set of heatmaps sé representing the spatial probabilities of keypoint positions. sf consists of m heatmaps si. one for every keypoint xL, k E {I,..., m.. The keypoint estimator model is a fully convolutional neural network, maintaining spatial representation from input output (see Fig. for the detailed architecture). This contrasts with a model ending in fully connected layers regressing the coordinates directly, where the spatial representation is lost (Zhong & Leonard, 2020). Because a fully convolutional model equivariant to translation, it can better generalize to unseen states that translations of states. Another advantage is the possibility of augmenting with e.g. random transformations of the D4 dihedral group to increase robustness and data efficiency, Because s can be transformed back with the inverse transformation, this augmentation is confined the keypoint estimator model and has no effect on the dynamics.
To distill keypoint coordinates from the heatmaps, we define Cartesian coordinate system in the image (see for example Fig. 1). Based on this definition, every pixel p corresponds to point Xp in the Cartesian space. The choice of the Cartesian coordinate system is arbitrary but is equal to the space of the dynamics x(*, u) and the constraint function ((x) (see Section 2). We use spatial softmax over pixels P to distill the coordinates of keypoint X from its probability heatmap:
EpEp xpp))(p)
X4 Eperes (P)
(8)
Spatial softmax differentiable, and the loss will backpropagate through the whole heatmap since Xş depends on all the pixels. Cartesian coordinates x of the different keypoints concatenated in vector x which serves as the positional state vector of the system. This compelling connection between image keypoints and Cartesian coordinates forms the basis of this work. The keypoint estimator model serves directly as state estimator to learn Lagrangian dynamics from images.
Similar to Jakab et al. (2018), acts information bottleneck, through which only the Cartesian coordinates of the keypoints flow to reconstruct the image with the renderer model. First, all X are transformed back spatial representations s/c using Gaussian blobs, parameterized by

---

hyperparameter
Ixp xx 2
5/2 exp
A binary cross-entropy loss Le is formulated over s and s' to encourage this Gaussian prior. The renderer model can more easily interpret the state in this spatial representation, it lies closer to its semantic meaning of keypoints as semantic landmarks in the reconstructed image. The renderer model learns a constant feature tensor (inspired by Nguyen-Phure with positional information, since the model itself is translation equivariant. See Fig. 4 for the detailed architecture.
al. (2019)), which provides
Z'
sood tensor
concatonate
groupnorm
mmxppoo
upsample
Figure 4: Visualization of the keypoint estimator (top) and renderer (bottom) model architectures. The keypoint estimator model and renderer model have similar architectures, utilizing down- and upsampling and skip connections wich help increasing the receptive field in Gu et al. (2019); Newell et al. (2016). The renderer model learns a constant feature tensor that is concatenated with the input s'. The feature tensor provides positional information since the fully-convolutionall model is translation equivariant.
Finally, reconstruction loss formulated over the reconstructed images {z"} and original images {z'):
L,
(10)
Dynamics loss function The sequence (x*), corresponding the sequence of given images {z*) and the constant input is used to calculate the dynamics loss. A fundamental aspect in learning dynamics from images is that velocities can not be directly observed. A single image only captures the position of a system, and contains no information about its velocities Other work uses sequences of images as input to model (Toth al., 2020) specific velocity estimator model trained to estimate velocities from a sequence of positions (Jaques et al., 2019). Zhong & Leonard (2020) demonstrate that for estimating velocities, finite differencing simply works better.
}.
We use central first order finite difference estimation, and project the estimated velocity on the constraints, so that the constraints are not violated:
(x' +Dx(x')
, {2, (2....... n -13
(11)
2h
where (.)+ signifies the Moore-Penrose pseudo-inverse and the timestep. We can now integrate future timesteps starting from initial values (xf, x²) using ODE solver. The equations of motion (6) are solved starting from all initial values in parallel, for timesteps This maximizes the learning signal obtained to learn the dynamics and leads overlapping subsequences of length
tlt)) i (2..... . v}
(12)
Neglecting side-effects such motion blur, which not very useful for this purpose.

---

Thus, X is obtained by integrating timesteps forward in time, starting from initial value x', which was derived by the keypoint estimator model. All t in all subsequences are compared with their corresponding keypoint states x in L2 loss:
L== -=
(13)
j
Total loss The total loss is the weighted sum of Lig Le and La, with weighing hyperparameter L = L, Le + XLd
To conclude, the keypoint estimator model, renderer model and dynamics models (mass matrix, potential energy and input matrix) are jointly trained end-to-end on sequences of images {z') and constant inputs u with stochastic gradient descent.
Rigid bodies as rigid sets of point masses By interpreting a set of keypoints as a set of point masses, can represent any rigid body and its corresponding kinetic and potential energy. Addi- tional constraints are added for the pairwise distances between keypoints representing a single rigid body (Finzi al., 2020). For 3D systems, least four keypoints required to represent any rigid body (Laus & Selig. 2020). We focus in our work on 2D systems in a plane parallel to the camera plane. 2D rigid bodies be expressed with set of point masses, which further be reduced depending on the constraints and connections between bodies. See Appendix A.3 for more details and proof.
4 EXPERIMENTS
We adapted the pendulum, cartpole and acrobot environments from dm.control (Tunyasuvunakool et al., 2020; Todorov et al., 2012) for our experiments. See Appendix A.4 for details about the environments, their constraint functions and the data generation procedure. The exact same model architectures, hyperparameters and control parameters were used across the environments. This further demonstrates the generality and robustness of our method. See Appendix A.5 for more details.
Since KeyCLD is trained directly on image observations, quantitative metrics can only be expressed in the image domain. The mean square error (MSE) in the image domain not good metric of long term prediction accuracy (Minderer et al., 2019; Zhong & Leonard, 2020). A model that trivially learns to predict static image, which is the average of the dataset, learns dynamics at all yet this model could report lower MSE than a model that did learn the dynamics but started drifting from the groundtruth after some time. Therefore, we use the valid prediction time (VPT) score (Botev et al., 2021; Jin et al., 2020) which measures how long the predicted images stay close to the groundtruth images of sequence.
VPT = argmin, (MSE(", x²' >
(14)
where z' are the groundtruth images, z" are the predicted images and is the error threshold. c is determined separately for the different environments because it depends on the relative size in pixels of moving parts. We define it the MSE of the averaged image of the respective validation dataset. Thus is the lower bound for a model that would simply predict a static image.
We present evaluations with the following ablations and baselines:
KeyCLD KeyLD
The full framework as described in Sections 2 and 3.
The constraint function is omitted.
KeyODE2
A second order neural ODE modelling the acceleration is used instead of the
Lagrangian prior. The keypoint estimator and renderer model are identical KeyCLD.
Lag-caVAE
The model presented by Zhong & Leonard (2020). We adapted the model to the higher resolution and color images.
Lag-VA VAE
The model presented by Zhong & Leonard (2020). We adapted the model to the higher resolution and color images.
HGN
Hamiltonian Generative Network presented by Toth et al. (2020).
7

---

Table Valid prediction time (higher is better) in number predicted frames (mean + std) for the different models evaluated on the 50 sequences in the validation set. Lag-ca VAE and Lag-Vi VAE are only reported on the pendulum environment, since they are unable model more than one moving body without segmented images. HGN is only reported on non-actuated systems, since it is incapable of modelling external forces and torques. KeyCLD achieves best results on all benchmarks.
actuation
KeyCLD
KryLD
KeyODE2
caVAE
VAB
HGN
Peadulum
(Hig.
43.1 9.7
+
19.1
0.2
(Fig.
39.3 9.8
14.9 7.9
12:0
0.0
8.0 10.2
(Fig. 10)
39.9 + 7.4
29.8 11.2
29.5
0.0 + 0.0
Carppule
(Pig.
8.7
28.0
24.4
(Fig. 12)
30.2 10.7
23.9 9.6
17.7 8.2
47.0 6.0
34.3
2.2 6.9
Acrobot
(Fig. 14)
46.8 1: 4.6
29.5 6.3
33.0 = 7.4
(Pig. 15)
47.0 3.5
39.1 9.9
30.8 9.3
Ground muth
KLyCLD
KeyLD
KeyODE2
Lap-VVA
Tag-VAE
Figure 5: Future frame predictions of the unactuated pendulum. These correspond to the first row in Table 2. 50 frames predicted based on the first three frames of the ground truth sequence to estimate the velocity. Every third frame of every sequence is shown. KeyCLD is capable making accurate long-term predictions with minimal drift of the dynamics. Without constraint function, KeyLD is not capable of making long-term predictions. Similarly, KeyODE2 is unable of making long-term predictions. Lag-caVAIB is fundamentally incapable of modelling data with background information, since the reconstructed images are explicitly rotated. Lag- VAE does not succeed in modelling moving parts the data, and simply learns to predict static images. HGN also does not capture the dynamics and only learns the background.
See Table for an overview of results, and Fig. for qualitative results on the unactuated pendulum. KeyCLD achieves the best results on all benchmarks. Lag-ca is unable model data with background (see also Fig. 5), Despite our best efforts for implementation and training, Lag- VAE and HGN perform very poorly. The models are not capable of handling the relatively more challenging visual structure of dmmcconte con environments. Removing the constraint function (KeyLD) has detrimental effect on the ability to make long-term predictions. Results are comparable to removing the Lagrangian prior altogether (KeyODE2). This suggests that modeling dynamics in Cartesian coordinates coupled with keypoint representations is itself a very strong prior, consistent with recent findings by Gruver et al. (2021). However, using Lagrangian formulation allows leveraging a constraint function, since a general neural ODE model can not make use of a constraint function. Thus, if a constraint function is available, the Lagrangian prior becomes much more powerful. See Appendix A.7 for more qualitative results and insights.
Interpretable energy models and control A major argument in favor of expressing dynamics in terms of matrix and potential energy is the straightforward control design via passivity based control and energy shaping. See Appendix A.6 for details and derivation of an energy shaping controller, and successful swing-up results of the pendulum, cartpole and acrobot system.
8

---

5 CONCLUSION AND FUTURE WORK
We introduce the use of keypoints to learn Lagrangian dynamics from images. Learned keypoint rep- resentations derived from images are directly used as positional state vector for learning constrained Lagrangian dynamics. The pendulum, cartpole and acrobot systems of con.contoll are adapted as benchmarks. Previous works in literature on learning Lagrangian or Hamiltonian dynamics from images were benchmarked on very simple renderings of flat sprites on blank backgrounds (Botev et al., 2021: Zhong & Leonard, 2020; Toth et al., 2020), whereas dmm, dimcontrrol is rendered with lighting effects, shadows, reflections and backgrounds. We believe that working towards more realistic datasets is crucial for applying Lagrangian models in the real world. The challenge of learning Lagrangian dynamics from more complex images should not be underestimated. Despite our best efforts in implementing and training Lag-ca VAF, Lag- VAE (Zhong & Leonard, 2020) and dm_cont rol benchmark. KeyCLD is
HGN (Toth al., 2020), they perform very poorly capable of making long-term predictions and learning accurate energy models, suitable for energy shaping control. When no constraint prior available, results are comparable general second order neural ODE. This signifies the benefit of using keypoint representations coupled with Cartesian coordinates to model the dynamics.
Our work focusses on 2D systems, where the plane of the system is parallel with the camera plane. Elevation to 3D, c.g. setups with multiple cameras, is an interesting future direction. Secondly, modelling contacts by using inequality constraints could be useful addition. Thirdly, our work focusses on energy-conserving systems. Modelling energy dissipation necessary for real-world applications. Several recent papers have proposed methods to incorporate energy dissipation in the Lagrangian dynamics models (Zhong et al., 2020; Greydanus & Sosanya, 2022). However, Gruver et al. (2021) argue that modelling the acceleration directly with second order differential equation and expressing the system in Cartesian coordinates, is a better approach. Further research into both approaches would clarify the benefit of Lagrangian and Hamiltonian priors real-world applications. Lastly, a major argument in using the Lagrangian prior is the availability of the mass matrix, potential energy and input matrix. This allows explainability and powerful control designs, such energy shaping. Model-based research for underactuated systems uses stochastic optimal control which often fails since a long prediction horizon is required. With our method, feedback controllers based on the mass matrix, potential energy and input matrix are possible which are more robust and do not require complicated optimal control.
BROADER IMPACT
tenacious divide exists between control engineering researchers and computer science researchers working control. Where the first would use known equations of motion for specific class of systems and investigate system identification, the latter would strive for the most general method with no prior knowledge. We believe this is a spectrum worth exploring, and such use strong physics priors as Lagrangian mechanics, but still model e.g. the input matrix and the potential energy with arbitrary neural networks. The broad field of model-based reinforcement learning could benefit from decades of theory and practice in classic control theory and system identification. We hope this paper could help bridge both worlds.
Using images as input is, in a broad sense, very powerful. Since camera sensors are consistently becoming cheaper and more powerful due advancements in technology and scaling opportunities. we can leverage these rich information sources for a deeper understanding of the world our intelligent agents are acting in. Image sensors can replace and enhance multiple other sensor modalities, at a lower cost. This work demonstrates the ability to efficiently model and control dynamical systems that are captured by cameras, with no supervision and minimal prior knowledge. We want to stress that we have shown it is possible to learn both the Lagrangian dynamics and state estimator model from images in one end-to-end process. The complex interplay between both, often makes them the most labour intensive parts in system identification. We believe this is gateway step in achieving reliable end-to-end learned control from pixels, especially since the availability of mass matrix, potential energy and input matrix models allows powerful control designs.

---

REPRODUCIBILITY STATEMENT
Please see the attached codebase to reproduce all experiments reported in this paper. The README, mc file contains detailed installation instructions and scripts for every experiment.
REFERENCES
Christine Allen-Blanchette, Sushant Veer, Anirudha Majumdar, and Naomi Ehrich Leonard. LagNetVip: A lagrangian neural network for video prediction. arXiv preprint arXiv:2010.1290 12932, 2020.
Jonathan I Barron. Continuously differentiable exponential linear units. arXiv preprint
1704.0048888 2017.
Peter Betsch. The discrete null space method for the energy consistent integration of constrained mechanical systems: Part I: Holonomic constraints. Computer Methods in Applied Mechanics and Engineering, (1150-52):5559-51999 2005.
Aleksandar Botev, Andrew Jaegle, Peter Wirnsberger, Daniel Hennes, and Irina Higgins. Which priors matter? benchmarking models for learning latent dynamics. arXiv e-prints, pp. arXiv-2111, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mihne and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http:/ /tit:: com/ google/ jax.
Boyuan Chen, Pieter Abbeel, and Deepak Pathak. Unsupervised learning of visual 3d keypoints for control. In International Conference on Machine Learning, pp. 1539-1549. PMLR, 2021.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 6572-6583, 2018.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying Hamiltonian and Lagrangian neural networks via explicit constraints. Advances in neural information processing systems, 33: 13880-13889, 2020.
Fabio Estern Romeo Ortega, Francisco R Rubio, and Javier Aracil. Stabilization of class of underactuated mechanical systems via total energy shaping. Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No. 01CH132288), volume 2, pp. 1137-1143. IEEE, 2001.
Sam Greydanus and Andrew Sosanya. Dissipative hamiltonian neural networks: Learning dissipative and conservative dynamics separately, arXiv preprint arXiv:2201.10080 2022.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. In Wallach, H Larochelle, A Beygelzimer, F d'Alché-Buc, Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: ptattp:://ppoceeedingg wwtp://wwwwdiinnn.rr.........ooe wttppppppcpppppe/22..12 .df..
Nate Gruver, Marc Anton Finzi, Samuel Don Stanton, and Andrew Gordon Wilson. Deconstructing the inductive biases of hamiltonian neural networks. In International Conference on Learning Representations, 2021.
Zaiwang Gu. Jun Cheng, Huazhu Fu, Kang Zhou, Huaying Hao, Yitian Zhao, Tianyang Zhang, Shenghua Gao, and Jiang Liu. Ce-net: Context encoder network for 2d medical image segmentation. IEEE transactions medical imaging, 38(10))2281-222) 122222, 2019.
10

---

Matteo Hessel, David Budden, Fabio Viola, Mihaela Rosca, Eren Sezener, and Tom Hennigan. Optax: composable gradient transformation and optimisation, in jax! 2020. URL http: /tt:://gitth./. com/ wwtppeede/pppooteeeeeedee optaxx
Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pp. 4020-4031, 2018.
Miguel Jaques, Michael Burke, and Timothy Hospedales. arsiics-as-invenne pensiic-a--ivvere--eee physical parameter estimation from video. In International Conference on Learning Representa- tions, 2019.
Unsupervised
Miguel Jaques, Martin Asenov, Michael Burke, and Timothy Hospedales. Vision-based system iden- tification and 3d keypoint discovery using dynamics constraints. arXiv preprint arXiv:2109.05928, 2021.
Pengzhan Jin, Zhen Zhang, Aiqing Zhu, Yifa Tang, and George Em Karniadakis. Sympnets: Intrinsic structure-preservingg symplectic networks for identifying hamiltonian systems. Neural Networks, 132:166-179, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Cornelius Lanezos. The variational principles of mechanics. The Variational Principles of Mechanics. University of Toronto press, 2020.
L. P. Laus and J. M. Selig. Rigid body dynamics using equimomental systems of point- masses. Acta Mechanica, 231(1):221-236, Jan 2020. ISSN 0001-5970, 1619-6937. doi:
1010077/600077-00944555555
M Lutter, K Listmann, and J Peters. Deep lagrangian networks for end-to-end learning of energy- based control for inder-actuated systems. In IEEE/RSJ International Conference Intelligent Robots and Systems (IROS 2019), pp. 7718-7725. IEEE, 2019.
Michael Lutter and Jan Peters. Combining physics and deep learning to learn continuous- time dynamics models. arXiv e-prints, pp. arXiv-2110, 2021.
Michael Lutter, Christian Ritter, and Jan Peters. Deep lagrangian networks: Using physics as model prior for deep learning. In International Conference on Learning Representations, 2018.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32, 2019.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pp. 483-499, Springer, 2016.
Thu Nguyen-Phrnn Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. HoloGAN: Unsupervised learning of 3d representations from natural images. Proceedings of the IEEE/CVF Computer Vision, pp. 7588-78 75987777,57 2019.
International Conference
Romeo Ortega, Arjan J Van Der Schaft, Iven Mareels, and Bernhard Maschke. Putting energy back in control. IEEE Control Systems Magazine, 21(2):18-33 2001.
Steindor Saemundsson, Alexander Terenin, Katja Hofmann, and Mare Deisenroth. Variational integrator networks for physically structured embeddings. In International Conference on Artificial Intelligence and Statistics, pp. 3078-3087. PMLR, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Peter Toth, Danilo J Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Hig- gins. Hamiltonian generative networks. In International Conference on Learning Representations, 2020.
11

---

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa, dm.control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.
org/10. 10mp:....6//ssmmmaaaaaaa 10000 100022. URL ht.ps:://///// //ww: www. sci sncendedi rect.com. wtt.:/ccc/csceeeeeceee science/ artticlleeel1/226666633000000
Mel Vecerik, -ean-Baptiste Regli, Oleg Sushkov, David Barker, Rugile Pevceviciute, Thomas Rothörl, Raia Hadsell, Lourdes Agapito, and Jonathan Scholz. S3k: Self-supervieed semantic keypoints for robotic manipulation via multi-view consistency. In Conference on Robot Learning, pp. 449-460. PMLR, 2021.
Ce Zheng, Kehtarnavaz, and Mubarak Shah. Deep learning- based human pose estimation: A survey, arXiv preprint arXiv:20/2. /3392, 2020.
Wenhan Wu, Taojiannan Yang, Sijie Zhu, Chen Chen, Ruixu Liu, Ju Shen, Nasser
Yaofeng Desmond Zhong and Naomi Leonard. Unsupervised learning of lagrangian dynamics from images for prediction and control. Advances in Neural Information Processing Systems, 33, 2020.
Yaofeng Desmond Zhong, Biswadip Dey. and Amit Chakraborty. Dissipative symoden: Encoding hamiltonian dynamics with dissipation and control into deep learning. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects points. arXiv preprint 1rX 2019.
12

---

A APPENDIX
A.1 DERIVATION OF CONSTRAINED
EQUATIONS
The Lagrangian of mechanical system described in Cartesian coordinates 12 is:
[(x, x) = MX
((x)
(15)
with M a static mass matrix, not depending on x, and V(x the potential energy. If the system has m degrees of freedom, additionally holonomic constraints necessary such that k These are described by constraint function (xx) I + R". Including the input matrix g(x) E R&xi and external inputs utt) R., the constrained Buler-Lagrange equations is expressed with a vector (t) E R containing Lagrange multipliers for the constraints (Finzi et al., 2020; Lanczos, 2020):
d Vx[(x,
[x x) g(x)u(() + D T(())
(16)
dt
Because the mass matrix is static
this is simplified to:
MX + V = g(x)u(t) + ( X(t)
(17)
X = M + - X(t), = v + g(x)u((t)
(18)
Calculating twice the time derivative of the constraint conditions yields:
(xx)
(xx)
(19)
D
( (
The Lagrange multipliers X(t) are solved by substituting & from equation (18) in equation (19):
(x) D (* DD ( A(1)
(20)
(()) = (- ( (D f DZ(())
use chain rule second time to get rid of the time derivative of (*b(
D$(x)* x)*
(21)
Substituting A(I) in (18) we arrive at equation (6).
A.2 IMPLEMENTATION CONSTRAINED
EQUATIONS IN JAX
It could seem daunting task to implement the derivation of the constrained Euler-Lagrange equa- tions (6) an autograd library. Therefore, we provide an implementation JAX (Bradbury et al., 2018). For more context, please the full code base in the supplementary materials (keycld/modee) www.d//moels....mmm py).
import jax
import numpy jnp
def constraint (n))): ing.array 1)
def matria
return
II other words, the centrifugal and Coriolis forces are zero hecause M 0 and V.M 0.
13

---

X1/C
W2
X2/c
X1
X
X
e
e,
X2
2
T
2 ml 12,12
- 1w
T
Σ m:; n111|x1112 2 i=1
1
2
Figure 6: Any 2D rigid body with mass and rotational inertia / is equivalent to set of two point masses X, and X2 with masses m and my. The kinetic energy of the rigid body, expressed translational part and rotational part, is equal the sum of the kinetic energies of the point masses.
def potential energy (parama, x)
function that
energy
zetuzn
ingut_matrix matrix (parans, x]
function that returns the input natrix
def ouler lagrange (parans, X.t, action)
jap www.llinnngiiiii... pinv
(parans,
jax. gead (potenttal energy, 1) (pacans, x) input mat.ix (parama, x) 8 action
Ophi jax. jac.biannnnnnn (constraint, En) (x)
BDphi jax. jaccbaaannnnn... jacobian (conatraint__e))
Lagzange multing miltppicatoos:
jap. linaly pinv (Ophá ini Ophi.t) (Dphi a_iev E DDphi x.t. K.E)
inv Ophi.
zetuzn at
A.3 RIGID BODIES AS SETS OF POINT MASSES
The position of rigid body in 2D is fully described by the position of its center of mass x and orientation 0 Potential energy only depends on the position, thus if want describe the potential energy with an equivalent rigid set of point masses, two points are sufficient to fully determine and 0. For the kinetic energy, we provide the following Theorem and proof:
Theorem For any 2D rigid body, described by its center of mass c, mass m and rotational inertia there exists equivalent rigid set of two point masses X1 and X2 with masses my and m2-
Proof. To find conditions such that the kinetic energy expressed in two point masses should be equal to the rigid body representation, we start by expressing general 3D-movement:
X.
Xi/e
E (1,2)
(22)
Where the vector x, are the coordinates the center mass and the vector X./a the position of the point mass relative the center of mass. Since this relative position X1/e has fixed length, only rotation is possible and hence the equation of the velocity is:
*; S..
Xi/o
i (1,2)
(23)
14

---

where is the rotational velocity of the body. Substituting this in the kinetic energy of the point masses, we get:
2
T
m:
X + w x X:/e|l2
2
i=1
(24)
1 2
[|x,||2 + m| xx/el1+ + 2x1/c (&c x w)
my
2
1=1
Where we calculated the square and used the circular shift property of the triple product on the last term.
For movement in the 2D-plane (i.e. w ē.w: and x; Ni. + EyXiy). this becomes:
1 2
T mi || 1| ||xs/e|| ww? + 2x//e" (Xe w)
1 2
m1 + m2) || (mi / m (m1x1/0 + m2x2/c) (&e x w)
(25)
Matching the kinetic energy of the 2 point masses (equation (25)) with that of the rigid body representation (left hand side of Figure 6), we get following conditions:
m mi + m2
T = m. | + m2 m2 xx/e/2
(26)
m. mix X1/c / m2X2/c
Since the last equation is vector equation, this gives us four equations in six unknowns which leaves us the freedom to choose two.
X1,
It follows from the third condition of (26) that points x,, X2 and x, should be collinear. To conclude, we can freely choose the positions the point masses (as long as x, is on the line between them), and will be able to model the rigid body as set of two point masses. In practice, KeyCLD will freely choose the keypoint positions to be able to model the dynamics. Depending on the constraints system, it is possible to further reduce the number of necessary keypoints. See Appendix A.4 for examples.
The interpretation of rigid bodies as sets of point masses allows expressing the kinetic energy the sum of the kinetic energies of the point masses. The mass matrix for 2D system thus defined as a diagonal matrix with masses my for every keypoint xt. leading to the following expression for the kinetic energy of the system:
m
0
0
0
m
0
0
X
T(x) x Mx
[x]
xn]
:
(27)
0
My
Xn
0
0
Min
To enforce positive values, the masses are parameterized by their square root and squared.
A.4 DETAILS ABOUT THE CONTROT. ENVIRONMENTS AND DATA GENERATION
We adapted the pendulum, cartpole and acrobot environments from di cont rol (Tunyasuvunakool al., 2020) implemented in MuloCo (Todorov al., 2012). Both are released under the Apache-2.0 license. Following changes were made to the environments to adapt them to our use-case:
Pendulum The camera was repositioned so that it in a parallel plane to the system. Friction was removed. Torque limits of the motor are increased.
15

---

Cartpole The camera was moved further away from the system to enable wider view, the two rails are made longer and the floor lowered so that they are not cut-off with the wider view. All friction is removed. The pole is made twice as thick, the color of the cart is changed. Torque limits are increased and actuation is added to the cart to make full actuation possible.
The camera and system are moved a little bit upwards. The poles made twice as thick, and one is changed in color. Torque limits are increased and actuation is added to the upper part make full actuation possible.
Acrobot
Data generation For every environment, 500 runs of 50 timesteps are generated with 10% validation split. Initial state for every sequence is at random position with small random velocity. The control inputs
constant throughout sequence, and uniform randomly chosen between the force and torque limits of the input. We set u = 0 for 20% of the sequences. We found this helps the model to learn the dynamics better, discouraging confusion of the energy models with external actions.
yy
X2
X2
X1
X1
X
e,X1 -1
||x||1 2
@(x) =
||x 4
@(x)
D(x)
1|x2 - -11
||x2
-x112
12
Figure 7: From left to right the pendulum, cartpole and acrobot dmncontrol environments. The respective constraint functions are given below each schematic.
The constraint function for each of the environments are given in Fig. As explained in Appendix A.3, every rigid body needs to be represented by two keypoints. But due to the constraints it is possible to omit certain keypoints, because they do not move or coincide with other keypoints. As experimentally validated, where the number of keypoints equals the number of bodies.
thus model all three systems with lower number of keypoints,
Pendulum One keypoint used model the pendulum. The second keypoint of this rigid body can be omitted because it can be assumed to be at the origin. Due to the constraint function, this point will provide no kinetic energy since it will not move. Since the other keypoints position and mass is freely chosen, any pendulum can be modelled. The constraint function expresses that the distance I from the origin to X1 fixed. The value of I the implementation irrelevant because it vanishes when taking the Jacobian.
Two keypoints are used to model the cartpole. The constraint function expresses that X1 does not move in the vertical direction and the distance between x₃ and X₂ is constant. Again, the values of I, and l in the implementation are irrelevant.
Cartpole
Acrobot Two keypoints are used to model the acrobot. The constraint function expresses that lengths ( and la are constant through time. Again, the values are irrelevant in the implementation.
A.5 TRAINING HYPERPARAMETER AND DETAILS
All models were trained on one NVIDIA RTX 2080 GPU.
KeyCLD, KeyLD and KeyODE2 We use the Adam optimizer (Kingma & Ba, 2015), implemented in Optax (Hessel et al., 2020) with learning rate of x 10 We use the exact same hyperparameters for all the environments and did not tune them individually. Dynamics loss weight À = 1, a 0.1
16

---

for the Gaussian blobs in s'. The hidden layers in the keypoint estimator and renderer model have the first block 32 features, this increases to respectively 64 and 128 after every maxpool operation. All convolutions have kernel size 3 x 3, and maxpool operations scale down with factor with kernel size of x 2.
The potential energy is modelled with MLP with two hidden layers with 32 neurons and celu activation functions (Barron, 2017). The weights are initialized with a normal distribution with standard deviation 0.01. Likewise, the input matrix modelled with an MLP similar the potential energy. The ouputs of this MLP are reshaped in the required shape of the input matrix.
The KeyODE2 dynamies model is MLP with three hidden layers with each 64 neurons. We chose higher number of layers and neurons, to allow this model more expressivity compared to the potential energy and input matrix models of KeyCLD.
Lag-ca VAE, Lag-VAL VAE and HGN For the Lag-caVAE VAE and Lag- VAE baselines, the official public codebase was used (Zhong Leonard, 2020). We adapted the implementation work with the higher input resolution of 64 by 64 (instead of 32 by 32), and 3 color channels (instead of 1).
For the HGN baseline, we used the implementation that was also released by Zhong & Leonard (2020). The architecture was adapted to work with the higher input resolution of 64 by 64 (instead of 32 by 32) by adding an extra upscale layer in the decoder, and a maxpool layer and one extra convolutional layer in the encoder.
A.6 ENERGY SHAPING CONTROL
major argument in favor of expressing dynamics in terms of a mass matrix and potential energy is the straightforward control design via passivity based control and energy shaping (Ortega al., 2001). Recent works of Zhong et al. (2020); Zhong & Leonard (2020) use energy shaping in generalized coordinates. In Cartesian coordinates, energy shaping can still be used. This easily seen from the fact that for the holonomic constraints ((x) O, we have the derivative D O, which means that the constraint forces equation (6) are perpendicular the path and hence do no work nor influence the energy (Lanczos, 2020).
Energy shaping control makes sure that the controlled system behaves according to a potential energy (x(x) instead of V(x): u [g g (VxV V
V Vv)
Ppasssve
(28)
where ypassive can be any passive output, the easiest choice being Upassive kag T xc, where ka is tuneable control parameter. The proposed potential energy Va should be such that:
x = argmin Vx 0 = --g- - (V
(29)
Where g is the left-annihilator of g, meaning that g-g O. For fully actuated systems, the first condition of equation (29) is always met and the easiest choice is:
Vx(x) = x-xxx)== kx
(30)
where hp is tuneable control parameter. The desired equilibrium position x" is obtained by putting a picture of the desired position the system into the keypoint estimator model. Finally, the passivity-based controller that is used is:
11
g) -g
V - kp(x x*)] - kag T
(31)
Changing the behavior of the kinetic energy is also possible (Gomez-Ester et al., 2001), but if left for future work. The passivity-based controller (see Appendix A.6 for full derivation and details):
{g g) -g V
kp(x x kug T
(32)
Many model-based reinforcement learning algorithms require the learning of full neural network as controller. Whilst in this work, due to knowledge the potential energy, we only need to tune two parameters ki and ka-
17

---

Figure 8: KeyCLD allows using energy shaping control because the learned potential energy model is available. Based on swing-up target image z", the target state x" is determined by the keypoint detector model. The sequences show that all three systems can achieve the target state. The control parameters k 5.0 and ky 2.0 are the same for all systems, demonstrating the generality of the control method.
KeyCLD
Keyl JD
KsyODE2
LagE
Lag-VAE
Figure 9: Actuated pendulum.
See Figure 8 for results of successful swing-up of the pendulum, cartpole and acrobot system. The same control parameters hip = 5.0 and kig 2.0 are used for all systems, demonstrating the generality of the control method.
A.7 ADDITIONAL EXPERIMENTAL RESULTS
Here we present additional qualitative results. Please refer to the supplementary materials for movies.
Future frame predictions We generate predictions of 50 frames, given the first frames of the ground truth sequence to estimate the initial velocity. Please compare these qualitative results for the unactuated and actuated pendulum environment(Fig. 5, 9), unactuated, underactuated and fully actuated cartpole environment (Fig. 10, 11, 12) and unactuated, underactuated and fully actuated acrobot environment (Fig. 13, 14, 15). Every third frame of the sequence is shown. See also the supplementary materials for movies of all sequences in the validation set. For every environment very long predictions of 500 frames are included, including visualizations of the keypoint representations and predictions.
Learned potential energy models Since the potential energy V is explicitly modelled, we can plot values throughout sequences of the state space. sequence of images is processed by the learned keypoint estimator model, and the states are then used to calculate the potential energy with the learned potential energy model. Absolute values of the potential energy are irrelevant, since the potential is relative, but we gain insights by moving parts of the system separately. See Figure 16 for results for the pendulum, Figures 17 and 18 for the cartpole and Figures and 20 for the acrobot.
18

---

CLDCLD
KsyODE2
IIGN
Figure 10: Unactuated cartpole.
Ground ruth
KLD
Keyl JD
KeyODE2
Figure 11: Underactuated cartpole.
Ground truth
KeyCLD
KeyLDD
KeyODE2
Figure Fully actuated cartpole.
Groad muth
KeyCLD
KeyLD
KsyODE2
HGN
Figure 13: Unactuated acrobot.
Ground mith
KeyCLD
KeyLDD
KeyODE2
Figure 14: Underactuated acrobot.
19

---

KeyCLD
KeyLD
KeyODE2
Figure 15: Fully actuated acrobot.
Figure 16: Potential energy of the trained KeyCLD model of the pendulum environment. The pendulum makes full rotation. As expected, the potential energy follows smooth sinusoidal path throughout this sequence. The maximum value reached when the pendulum upright, and the minimum value is reached when the pendulum is down.
Figure 17: Potential energy the trained KeyCLD model of the cartpole environment. The position of the cart is fixed, and the pole makes full rotation. As expected, the potential energy follows smooth sinusoidal path throughout this sequence.
Figure 18: Potential energy of the trained KeyCLD model of the cartpole environment. The pole fixed and the cart moves from left to right. As expected, the change in potential energy this sequence is very low (compare to Fig. 17 with the same axis). A horizontal movement has impact on the gravity potential.
20

---

Figure 19: Potential energy of the trained KeyCLD model of the acrobot environment. The first link makes full rotation, the second link is fixed relative to the first link. As expected, the potential energy follows a smooth sinusoidal path throughout this sequence.
Figure 20: Potential energy of the trained KeyCLD model of the acrobot environment. The first
link is fixed and the second link makes a full rotation. Again, the potential energy follows smooth sinusoidal path throughout this sequence. Please compare with Fig. 19, where both links moving. Here the potential energy changes less, because the first link is not moving.
21


Reviews:
[{"review_id": "tG4mxciFAI", "reviewer": "Reviewer_NWGA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to learn the dynamics of simple systems from visual observation. The proposed method is based on the detection of 2D keypoints in the image in an unsupervised way and the learning of the dynamics in its Lagrangian form. Prior knowledge of the structure of the problem is necessary.", "summary_of_the_review": "Finally, the main novelty of this paper consists in modeling the dynamics in its Lagrangian form. Even if the idea has a certain interest and seems reasonable, I am not convinced by the experiments of the practical advantages of such approach over existing baselines. Furthermore, the simplicity of the proposed tasks, the absence of relevant citations, and of comparisons with the state of the art do not allow me to recommend this work for a presentation at ICLR.", "strength_and_weaknesses": "In my opinion, the novelty of the proposed method is extremely limited. I list below a several papers (sometimes absent from the cited references) comparable to the work presented:\n\n- [1] introduces a very similar framework for unsupervised 2D keypoint detection. The presented model is a simplification in that visual information is no longer extracted from a source image, but simply learned as a fixed latent vector. The model is therefore unlikely to generalize to basic changes in the scene, such as the background color.\n\n- [2] is an extension of [1] presenting a keypoint detection model for control. Their manipulation task is quite similar to those used in paper, although in my opinion more complex. This reference is missing from the paper, and yet seems very relevant.\n\n- [3] offers a different keypoint-based approach in a realistic environment for controlling a 3D robot arm. It seems to me that the method is comparable with this reference (also missing from the citations), and address a much more complex problem.\n\n- [4] and [5] also propose to learn the dynamics of complex systems from the discovery of unsupervised 2D keypoints. However, these two papers use a much more challenging counterfactual context, and address 3D problems.\n\nMore generally, the paper is quite difficult to read, section 2 and 3, as well as figures 3 and 4 can in my opinion be re-worked to make understanding easier. Deep understanding of [1], [2] and cited works on Deep Lagrangian models feel absolutely necessary to fully understand the proposition. I list below  several remarks concerning the method:\n\n- The use of strong and demanding prior knowledge on the geometry of the problem seems to be a very strong assumption. Adding its constraints unsurprisingly improves overall performance, as a big part of the dynamic is manually encoded into the network. However, I wonder how precise these constraints need to be. It would have been interesting to show that the use of partial or slightly erroneous constraints made it possible to maintain good performance.\n\n- The authors detail in equation 11 the formula used to estimate the speed of keypoints by finite difference. The authors have chosen to use a forward Euler diagram, in other words, the speed (i) already contains the information of the position at the instant (i+1)! This effect fades when the prediction is made over a longer horizon, but it is a fairly significant design flaw.\n\n- The scenarios chosen to evaluate the model seem terribly simple to me regarding the tasks that SOTA address with this type of keypoints-based model (see references given above). These are simulated 2D problems, with relatively simple dynamics, with very simple visual output (no occlusion, constant background, fixed geometry, etc.)\n\n- It would have been interesting to study in more detail the behavior of detected keypoints. I wonder for example (1) what is the contribution of the BCE on the heatmaps, (2) what is the impact of the increase/reduction of the number of keypoints (3) Is the application of \"transformation of the $D_4$ dihedral group\" to $z$ useful in practice? It also seems to me that given the constraints on the keypoints, the learned dynamics should be very close to the physical equations, but this is not discussed in the paper.\n\n[1] Jakab, T., Gupta, A., Bilen, H., & Vedaldi, A. (2018). Unsupervised learning of object landmarks through conditional image generation. Advances in neural information processing systems, 31.\n\n[2] Kulkarni, T. D., Gupta, A., Ionescu, C., Borgeaud, S., Reynolds, M., Zisserman, A., & Mnih, V. (2019). Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32.\n\n[3] Manuelli, L., Li, Y., Florence, P., & Tedrake, R. (2021, October). Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning. In Conference on Robot Learning (pp. 693-710). PMLR.\n\n[4] Li, Y., Torralba, A., Anandkumar, A., Fox, D., & Garg, A. (2020). Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 9180-9192.\n\n[5] Janny, S., Baradel, F., Neverova, N., Nadri, M., Mori, G., & Wolf, C. (2022, April). Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space. In International Conference on Learning Representation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c2dtuOVY-a", "reviewer": "Reviewer_U2jo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to combine keypoint extraction from vision with physics-inspired deep networks to learn accurate dynamics models from images. The keypoint extractor extracts cartesian coordinates on the individual bodies. The consecutive dynamics model utilizes the key points to learn the constrained dynamics of these bodies. In addition, the proposed algorithm enforces Lagrangian dynamics on each body. In contrast to prior work, this paper uses a keypoint extractor while the other approaches utilized a VAE. Within the experiments, the authors show that the proposed algorithm can learn a good dynamics model.", "summary_of_the_review": "While I do not object accepting the paper, I don't think that the quality of this paper is ready for publication at ICLR. If the other reviewers, want the paper to be published, I am fine with that. ", "strength_and_weaknesses": "Dear authors, greeting from your Neurips reviewers. Please keep in mind that the review community for this topic is not soo big. Therefore, there is a high likelihood that you get the same reviewers. This paper deeply frustrates me, as this paper is soo close to the finish line but it seems like you cannot push it over the finish line. You had 2 months and 4 good reviews to adapt the paper. Some of the NEURIPS reviews might be a bit harsh on the baselines but overall fair, positive, and constructive. However, you decided to do very little and resubmit mostly the same paper. Yes, you cut a few equations, added a baseline (that nobody asked for), and reported VPT for no-actuation, under-actuated, and fully actuated. However, you failed to add a qualitative comparison of the prediction. You could easily plot the trajectories in phase space, which would enable a qualitative comparison, and plot the constraint violation of the key points. Another fun idea would have been to infer the constraints from the key points which should be straightforward (when the key points are good) compared to hard-coding the constraints. I still find the approach of learning dynamics from pixels but simultaneously requiring the knowledge of the kinematic chain a weird combination. Looking at the videos in the `extra_predictions` does not increase my confidence in the methods. The time horizons are very short and it often fails horribly for the under-actuated and un-actuated versions. \n\nI would urge the reviewers to go back to the NEURIPS reviews and see how the paper could be improved. To refresh your mind, this is the NEURIPS Meta reviews which provide a good summary and positiveness of the different reviewers. Unfortunately, the meta reviewer was a bit too confident as the work was not `substantially improved` but resubmitted with minor changes. \n\n> While all reviewers agree with the novelty and interestingness of this research direction, in its current form, my assessment is that it would make a good workshop paper proposing a 1st step into the direction of learning Lagrangian dynamics from images. The results are still in early stages, e.g. experiments using the simplest DM_control environments (even requiring modifications from the authors on friction to make it work). As uztn noted, “the system is built from parts that are well-described in the prior work, as described by the authors. The resulting system will not work outside of simple 2D, fully observable systems, and the generations produced are of comparable or lower quality than generations of dynamic systems both in and outside of the physically-inspired dynamics modeling community.”\n>\n> I also agree with reviewer sEpU that a comparison to video-prediction models (or image-based “world models”) is not outside the scope, since people will want to know where the approach stands compared with traditional methods (even at lower performance). As the reviewer mentioned, experiments which enable the comparison to other methods such as control / video-prediction, which are not necessarily Lagrangian-based or are Lagrangian based but do not learn from images (to answer questions like \"how much does the use of the Lagrangian improve the video-prediction/control performance\" or \"how large is the performance drop by learning from images instead states directly\") will be tremendously useful to achieve high impact within the broader ML research community.\n> \n> I look forward to seeing the authors improve the work, as we all agree that the direction is novel and I’m confident that the work will be substantially improved going forward.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IlxDIPVBK4", "reviewer": "Reviewer_ooc4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a method to learn Lagrangian dynamics from images. In line with previous work, the authors simplify the learning problem by: (1) representing the Lagrangian as the difference in kinetic and potential energy (both learned), and (2) using Cartesian coordinates to represent the positions of the system components. The main contribution of this work is the approach for identifying the positions of the system components. The authors do this by extracting keypoints from image inputs. The authors demonstrate the utility of their approach on the dm_control dataset.", "summary_of_the_review": "The method is well motivated and demonstrates strong performance relative to baseline models. Some of the claims/assertions are not well supported.\n", "strength_and_weaknesses": "*Strengths:* The authors address the challenging problem of designing physics informed deep neural networks with interpretable representations. The proposed method outperforms other methods on the proposed dataset and can learn to control underactuated systems.\n\n*Weaknesses:* I would have liked to see a comparison between the proposed approach and Lag-caVAE on the dataset given in Lag-caVAE. I would also be interested in a comparison in the number of parameters between models.\n\n*Notes:*\n- “and introduced energy-based control” – I think this phrasing may be confusing to some readers, since energy based control was not a contribution of Zhong et al.\n- “This enables more data efficient models, at the cost of providing extra knowledge about the system in the form of a constraint function.” – was this shown? \n- The authors claim that the use of a fully convolutional model is an advantage of the proposed approach over Zhong & Leonard, 2020 because of the translation equivariance. How does translation equivariance benefit the model when most of the experiments vary under rotation?\n- The authors note that additional information must be given to the renderer to compensate for translation equivariance, but an equivariant representation shifts under the relevant transformation, why would additional information be necessary?\n\n*Questions:*\n- What do the authors mean by interpretable coordinates in Table 1? For example, by construction, the latent representation learned in HGN is interpretable as the generalized position and generalized momentum; however, the authors do not mark it as having interpretable coordinates.\n\n*Possible typos:*\n- “wether they are” → whether they are\n- “changes in function of the state” → changes as a function of the state\n- “ are now in function of” →  are now a function of\n- “The Hamiltonian expresses the total energy of the system” – I think the citation used here is inappropriate; consider using a well known classical mechanics textbook instead.\n- “Lagrangian mechanics because it is more general” – I think the citation used here is inappropriate; consider using a well known classical mechanics textbook instead.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject