Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed a novel method to solve dynamical systems through data. Instead of formulating the system Lagrangians or Hamiltonians, the method leverages the symmetry and conservation laws, thus does not assume any specific dynamical system forms. The proposed network predicts both the guess of the generalized velocity and the set of constants of the motions (COMs). Then, by exploring the orthogonality between the gradients of COMs and the generalized velocity, the final system velocity is projected through a QR decomposition.  The authors show that their method performs equally well or better than many baselines in many simulated dynamical systems. 

strengths_and_weaknesses: The strengths of the paper:

The method is quite novel yet simple, i.e. utilizes the system’s symmetry structure and conservation laws to better learn the system dynamics. 

The proposed method can handle many different types of dynamical systems, including Hamiltonian systems, dissipative systems, and even infinite dimension PDEs. 

The method outperforms many baselines in a wide range of tasks, and the generated trajectory has higher quality in some tasks (i.e. two body systems). 


The method works well even with a partial list of constants of motions, as shown in Figure 5. 


The weakness of the paper:

Most of the experiments are conducted in relatively simple dynamical systems. It will be interesting to compare the method with Lagrangian and Hamiltonian based methods in higher dimensional mechanical systems (such as robots).  I suspect that LNN may outperform the proposed method in some of these cases. 

In table 1, LNN is shown to not work with dissipative systems, which I am not convinced of.  It will be surprising that a Lagrangian based method cannot handle dissipations.

The proposed method, COMET, performs significantly worse than other baselines such as LNN in the 2D pendulum task, and even worse than NODE, which is surprising–According to the authors, if the system does not have any symmetry it will be the same as NODE. 
questions: Besides the main comment, I have a few questions and suggestions for the authors:

1, Explain a bit more on why QR decomposition is chosen as the way to orthogonalization. What other methods are available? Why this particular one (i.e. differentiability, computation simplicity, etc)? 
2, Explain a bit more on the “stiffness” of the dynamics. Why COMET, LNN, NST can produce stiffer dynamics?

3, Figure 5 shows that COMET works even without all COMs included. Can the authors add an ablation study to show how well COMET performs with a partial set of COMs?
limitations: N/A
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper is well written and easy to follow. It describes a methodology to learn a system of differential equations from data. The methodology can be used to capture the evolution of a given multi-dimensional time series s(t); possibly the time series can be influenced by an exogenous input (e.g. a force acting on the system). The working hypothesis is that the provided time series is the result of a constrained differential equation, i.e. there exists a function c(.) such that c(s) is constant on the trajectory s(t). 

The proposed approach is tested on motion data (i.e. frictionless mass-spring, 2D pendulum, 2D damped pendulum, two body interactions, 2D nonlinear spring, Lotka-Volterra dynamics). Results show that the proposed method performs better than previous approaches (i.e. NODE, HNN, NSF, LNN) in terms of mean squared error on prediction. Some heuristics for choosing the number of constraints acting on the system are proposed; generalization to systems with infinite number of states is also proposed. 
strengths_and_weaknesses: The paper is well written and easy to read. The structure of the paper is sound and results have been presented with enough details to understand the main point of the submitted paper. The major but fundamental weakness of the paper is that it focuses on relatively simple and artificial examples (e.g. mass-spring, pendulum) which aren't of practical use but merely useful for assessing the soundness of the approach. Even though I can see the value of these examples when explaining Lagrangian or Hamiltonian physics (where we need to understand how the different analytical components interact), I see very limited value in using these artificial examples when explaining how to use neural-networks to approximate a given time series. By focusing on this artificial examples, authors fall short in addressing fundamental questions listed below.

(1) ***Choosing the system state***. In classical mechanics  (e.g. Newton-Euler, Legrangian, Hamiltonian) the state of the system and its derived quantities (e.g. its derivative, the momentum) are fundamental quantities with specific properties. The choice of the state isn't trivial and it requires an analytical understanding of the system itself. In the proposed paper, authors assume that the state is given and often coincides with the one used in classical mechanics. However, in applications relevant to machine learning (as the one proposed), assuming that the state is given is quite a restrictive assumption. 

(2) ***Choosing the system state dimensionality and number of constraints***. Section 6 proposes an heuristic for finding the number of constraints of motion. The problem is quite fundamental and it deserves a more thorough investigation. Authors could try to get inspiration from solutions to a similar problem in the scope of system identification and model selection (e.g. MDL, minimum description language for model selection; AIC, Akaike information criteria). 

questions: - ***How robust is the proposed methodology to the choice of the system state?*** Do the authors have evidence that the proposed approach could potentially work replacing the given system state with an embedding (such as the latent of a VAEs) learnt from observations which are connected to the state but possibly higher dimensional (e.g. an image of the pendulum)?

- ***How robust is the proposed methodology to noise on the system derivative?*** The system state is always composed of the system generalized coordinates and their derivatives. As a result the system state derivative often contains the generalized coordinates accelerations (i.e. second order derivatives) which are extremely noisy when measured on real systems. Authors mentioned the states rate change was subject to a Gaussian noise with standard deviation of 0.05. Can authors elaborate on how this is representative of the typical noise observed in real systems when second order derivatives are approximated (e.g. finite differences)? 

- ***What is the role of $s_0(s)$ at test time?*** Looking at (4) and (2), it seems that the only role of $s_0$ is (at training time) to guide $s$ to $\hat{s}$ and to be sure that at convergence $\dot{s} \in \langle \nabla c_1, \dots, \nabla c_n \rangle^\perp$. After training it is unclear whether the function $s_0$ is needed. Can author describe better the need to explicitly learn the function $s_0(s)$?
limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In the context of learning models for dynamical systems, the authors propose to learn the constants of motion of the system by means of a neural mapping from states. The method is validated on several (albeit small) simulated dynamical systems, where it appears to improve upon previous approaches.
strengths_and_weaknesses: This paper was mostly clearly written (if occasionally abstract) and enjoyable to read. While mechanics is not my area of expertise, it proposes what seems like a novel idea for modelling of dynamical systems (learning constants of motion). It also empirically improves upon the results of the famous neural ODE paper and later developments. On the other hand, the evaluation is entirely on what seems like quite simple examples. 

Concerns:
-----------------
Presentation: 
- Some intuition of how the learning interacts with the integration scheme would have been helpful (c.f. NODE paper). It is quite abstract. 

Experiments: 
- The assumed number of constants of motion appears to have a significant effect (Fig. 5). How were these selected in the benchmark examples?

- It is a bit concerning that it degrades for 50 hidden neurons on the 2D non-linear spring system, there are still thousands of parameters in that network. Any idea why? Could this have something to do with biases induced by the QR decomposition? It is not obvious to me what kind of biases that will induce.

- As an engineer, I would have liked to see some example on how this approach can be used on some more complex real-world-like sysid problem.

Minor presentation issues:

"grad" -> writing out "gradient" in text would read better, or typeset it as a math operator (or nabla)

l282: works similar/is comparable? to

l290: which field, or just on science?
l290: more point*s* 


questions: The assumed number of constants of motion (n_c) appears to have a significant effect (Fig. 5). Maybe I missed it, but I couldn't find how these were selected in the benchmark examples?

It is a bit concerning that it degrades for 50 hidden neurons on the 2D non-linear spring system, with three layers there are still thousands of parameters in that network. Any idea why? Could this have something to do with biases induced by the QR decomposition? It is not obvious to me what kind of biases that will induce.

limitations: Mostly adequate.
review_title: null

Meta-review: metareview: 2 of the 3 reviewers highly appreciated the rebuttal and are now recommending the paper for acceptance without any reservations. The 3rd, most critical reviewer, FmUK did unfortunately not react. The new experiments "learning from pixels" nicely addresses the reviewer's concern about having to carefully choose the system state. Also the concern about the number of constants of motion is well addressed. The question about the sensitivity to noise could have been stronger: FmUK was talking about physical systems that typically have accelerations as part of their states, but typical sensors only measure positions/angles (which typically already produce slightly noisy measurements). Applying numerical differentiation twice to get to the accelerations often results in very noisy measurements. Hence the question how representative the experiments (where the states - that also include $\dot{x}$ - are assumed to be measured perfectly) are for real systems. I think this is still an interesting point to discuss - but no deal-breaker for me.
confidence: Less certain
Decision: Accept