Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a method to learn Lagrangian dynamics from images. In line with previous work, the authors simplify the learning problem by: (1) representing the Lagrangian as the difference in kinetic and potential energy (both learned), and (2) using Cartesian coordinates to represent the positions of the system components. The main contribution of this work is the approach for identifying the positions of the system components. The authors do this by extracting keypoints from image inputs. The authors demonstrate the utility of their approach on the dm_control dataset.
strength_and_weaknesses: *Strengths:* The authors address the challenging problem of designing physics informed deep neural networks with interpretable representations. The proposed method outperforms other methods on the proposed dataset and can learn to control underactuated systems.

*Weaknesses:* I would have liked to see a comparison between the proposed approach and Lag-caVAE on the dataset given in Lag-caVAE. I would also be interested in a comparison in the number of parameters between models.

*Notes:*
- “and introduced energy-based control” – I think this phrasing may be confusing to some readers, since energy based control was not a contribution of Zhong et al.
- “This enables more data efficient models, at the cost of providing extra knowledge about the system in the form of a constraint function.” – was this shown? 
- The authors claim that the use of a fully convolutional model is an advantage of the proposed approach over Zhong & Leonard, 2020 because of the translation equivariance. How does translation equivariance benefit the model when most of the experiments vary under rotation?
- The authors note that additional information must be given to the renderer to compensate for translation equivariance, but an equivariant representation shifts under the relevant transformation, why would additional information be necessary?

*Questions:*
- What do the authors mean by interpretable coordinates in Table 1? For example, by construction, the latent representation learned in HGN is interpretable as the generalized position and generalized momentum; however, the authors do not mark it as having interpretable coordinates.

*Possible typos:*
- “wether they are” → whether they are
- “changes in function of the state” → changes as a function of the state
- “ are now in function of” →  are now a function of
- “The Hamiltonian expresses the total energy of the system” – I think the citation used here is inappropriate; consider using a well known classical mechanics textbook instead.
- “Lagrangian mechanics because it is more general” – I think the citation used here is inappropriate; consider using a well known classical mechanics textbook instead.

clarity,_quality,_novelty_and_reproducibility: *Quality:* For the most part, the submission seems to be technically sound. There are some errors in the text and some of the claims/assertions are not supported (see questions/comments)

*Clarity:* For the most part, the submission is clearly written and well organized (see questions/comments for concerns/suggestions)

*Originality:* The approach appears new and is well situated in the context of previous work.

summary_of_the_review: The method is well motivated and demonstrates strong performance relative to baseline models. Some of the claims/assertions are not well supported.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine keypoint extraction from vision with physics-inspired deep networks to learn accurate dynamics models from images. The keypoint extractor extracts cartesian coordinates on the individual bodies. The consecutive dynamics model utilizes the key points to learn the constrained dynamics of these bodies. In addition, the proposed algorithm enforces Lagrangian dynamics on each body. In contrast to prior work, this paper uses a keypoint extractor while the other approaches utilized a VAE. Within the experiments, the authors show that the proposed algorithm can learn a good dynamics model.
strength_and_weaknesses: Dear authors, greeting from your Neurips reviewers. Please keep in mind that the review community for this topic is not soo big. Therefore, there is a high likelihood that you get the same reviewers. This paper deeply frustrates me, as this paper is soo close to the finish line but it seems like you cannot push it over the finish line. You had 2 months and 4 good reviews to adapt the paper. Some of the NEURIPS reviews might be a bit harsh on the baselines but overall fair, positive, and constructive. However, you decided to do very little and resubmit mostly the same paper. Yes, you cut a few equations, added a baseline (that nobody asked for), and reported VPT for no-actuation, under-actuated, and fully actuated. However, you failed to add a qualitative comparison of the prediction. You could easily plot the trajectories in phase space, which would enable a qualitative comparison, and plot the constraint violation of the key points. Another fun idea would have been to infer the constraints from the key points which should be straightforward (when the key points are good) compared to hard-coding the constraints. I still find the approach of learning dynamics from pixels but simultaneously requiring the knowledge of the kinematic chain a weird combination. Looking at the videos in the `extra_predictions` does not increase my confidence in the methods. The time horizons are very short and it often fails horribly for the under-actuated and un-actuated versions. 

I would urge the reviewers to go back to the NEURIPS reviews and see how the paper could be improved. To refresh your mind, this is the NEURIPS Meta reviews which provide a good summary and positiveness of the different reviewers. Unfortunately, the meta reviewer was a bit too confident as the work was not `substantially improved` but resubmitted with minor changes. 

> While all reviewers agree with the novelty and interestingness of this research direction, in its current form, my assessment is that it would make a good workshop paper proposing a 1st step into the direction of learning Lagrangian dynamics from images. The results are still in early stages, e.g. experiments using the simplest DM_control environments (even requiring modifications from the authors on friction to make it work). As uztn noted, “the system is built from parts that are well-described in the prior work, as described by the authors. The resulting system will not work outside of simple 2D, fully observable systems, and the generations produced are of comparable or lower quality than generations of dynamic systems both in and outside of the physically-inspired dynamics modeling community.”
>
> I also agree with reviewer sEpU that a comparison to video-prediction models (or image-based “world models”) is not outside the scope, since people will want to know where the approach stands compared with traditional methods (even at lower performance). As the reviewer mentioned, experiments which enable the comparison to other methods such as control / video-prediction, which are not necessarily Lagrangian-based or are Lagrangian based but do not learn from images (to answer questions like "how much does the use of the Lagrangian improve the video-prediction/control performance" or "how large is the performance drop by learning from images instead states directly") will be tremendously useful to achieve high impact within the broader ML research community.
> 
> I look forward to seeing the authors improve the work, as we all agree that the direction is novel and I’m confident that the work will be substantially improved going forward.

clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: While I do not object accepting the paper, I don't think that the quality of this paper is ready for publication at ICLR. If the other reviewers, want the paper to be published, I am fine with that.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn the dynamics of simple systems from visual observation. The proposed method is based on the detection of 2D keypoints in the image in an unsupervised way and the learning of the dynamics in its Lagrangian form. Prior knowledge of the structure of the problem is necessary.
strength_and_weaknesses: In my opinion, the novelty of the proposed method is extremely limited. I list below a several papers (sometimes absent from the cited references) comparable to the work presented:

- [1] introduces a very similar framework for unsupervised 2D keypoint detection. The presented model is a simplification in that visual information is no longer extracted from a source image, but simply learned as a fixed latent vector. The model is therefore unlikely to generalize to basic changes in the scene, such as the background color.

- [2] is an extension of [1] presenting a keypoint detection model for control. Their manipulation task is quite similar to those used in paper, although in my opinion more complex. This reference is missing from the paper, and yet seems very relevant.

- [3] offers a different keypoint-based approach in a realistic environment for controlling a 3D robot arm. It seems to me that the method is comparable with this reference (also missing from the citations), and address a much more complex problem.

- [4] and [5] also propose to learn the dynamics of complex systems from the discovery of unsupervised 2D keypoints. However, these two papers use a much more challenging counterfactual context, and address 3D problems.

More generally, the paper is quite difficult to read, section 2 and 3, as well as figures 3 and 4 can in my opinion be re-worked to make understanding easier. Deep understanding of [1], [2] and cited works on Deep Lagrangian models feel absolutely necessary to fully understand the proposition. I list below  several remarks concerning the method:

- The use of strong and demanding prior knowledge on the geometry of the problem seems to be a very strong assumption. Adding its constraints unsurprisingly improves overall performance, as a big part of the dynamic is manually encoded into the network. However, I wonder how precise these constraints need to be. It would have been interesting to show that the use of partial or slightly erroneous constraints made it possible to maintain good performance.

- The authors detail in equation 11 the formula used to estimate the speed of keypoints by finite difference. The authors have chosen to use a forward Euler diagram, in other words, the speed (i) already contains the information of the position at the instant (i+1)! This effect fades when the prediction is made over a longer horizon, but it is a fairly significant design flaw.

- The scenarios chosen to evaluate the model seem terribly simple to me regarding the tasks that SOTA address with this type of keypoints-based model (see references given above). These are simulated 2D problems, with relatively simple dynamics, with very simple visual output (no occlusion, constant background, fixed geometry, etc.)

- It would have been interesting to study in more detail the behavior of detected keypoints. I wonder for example (1) what is the contribution of the BCE on the heatmaps, (2) what is the impact of the increase/reduction of the number of keypoints (3) Is the application of "transformation of the $D_4$ dihedral group" to $z$ useful in practice? It also seems to me that given the constraints on the keypoints, the learned dynamics should be very close to the physical equations, but this is not discussed in the paper.

[1] Jakab, T., Gupta, A., Bilen, H., & Vedaldi, A. (2018). Unsupervised learning of object landmarks through conditional image generation. Advances in neural information processing systems, 31.

[2] Kulkarni, T. D., Gupta, A., Ionescu, C., Borgeaud, S., Reynolds, M., Zisserman, A., & Mnih, V. (2019). Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32.

[3] Manuelli, L., Li, Y., Florence, P., & Tedrake, R. (2021, October). Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning. In Conference on Robot Learning (pp. 693-710). PMLR.

[4] Li, Y., Torralba, A., Anandkumar, A., Fox, D., & Garg, A. (2020). Causal discovery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 9180-9192.

[5] Janny, S., Baradel, F., Neverova, N., Nadri, M., Mori, G., & Wolf, C. (2022, April). Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space. In International Conference on Learning Representation
clarity,_quality,_novelty_and_reproducibility: The paper is not very well written and deserves improvements. The model description is hard to follow. Novelty is very limited, and the proposed method is not situated among previous works. Reproducibility seems possible, but code release would have been appreciated.
summary_of_the_review: Finally, the main novelty of this paper consists in modeling the dynamics in its Lagrangian form. Even if the idea has a certain interest and seems reasonable, I am not convinced by the experiments of the practical advantages of such approach over existing baselines. Furthermore, the simplicity of the proposed tasks, the absence of relevant citations, and of comparisons with the state of the art do not allow me to recommend this work for a presentation at ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The papers proposes an algorithm for learning keypoint representations of images with the assumption that the dynamics of the keypoints is constrained Lagrangian dynamics. The model is shown to produce good predictions on three environments (Pendulum, Cartpole and Acrobot).

The reviewers acknowledge the general direction of using physics-motivated inductive biases when learning representations. This work makes a small step in this direction which is appreciated by the reviewers. However, the setup considered in the paper is rather simple: 1) the model is applicable only to fully observed 2d systems and 2) the constraints are known (not inferred). Especially the second assumption seems unrealistic.

Given the simplicity of the considered setup, the reviewers would like to see more thorough evaluation of the model and qualitative assessment of the experimental results. For example: how good are the extracted key points, how much do they violate the constraints, qualitative assessment of the importance of the constraints, what is the main challenge of inferring the constraints from data. Such analysis would provide more insight on the model's strengths and limitations. For these reasons, the reviewers do not recommend acceptance of the paper but they encourage the authors to improve the paper and re-submit.
justification_for_why_not_higher_score: Too simple setup (observed 2d systems and known constraints).
justification_for_why_not_lower_score: The paper makes a small steps in an important direction.
Decision: Reject