Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to learn the first integrals of an ODE system by a neural network and projects
 the time evolution of the dynamical system onto the submanifold defined using the (discrete) gradients of the learned first integrals. With an appropriate number of assumed first integrals, FINDE predicted future states more accurately than base models.
strengths_and_weaknesses: Strengths:

1. This paper addresses an important topic in predicting an unknown physical system, that is symmetry, or the associated conservation law;

2. The Discrete-Time Projection Method for preserving the first integral of the discretized dynamical system seems to be novel;

3. The empirical results are through and sound.

Weaknesses:

1. The methodology section is hard to follow, no intuition is given after the formal introduction of the method;

2. The contribution of this article is entangled, the authors should separate the question of finding the right first integrals and making more accurate future predictions of the dynamical system. It's hard to say that making a better prediction comes from learning the right first integrals.

questions: 1. Can the authors explain more clearly how the implicit dFINDE can be implemented in an explicit way?

2. I don't understand this sentence: "Not only that, FINDE found and preserved the system energy and the total mass as first integrals".  Does it mean that FINDE can't guarantee to preserve other types of first integrals?
limitations: The limitations have be addressed in the Weaknesses summary.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel neural model for predicting the dynamics of a system by finding multiple unknown conserved quantities (first integral) from data. The discrete derivative-based formulation eliminates numerical errors, and the model can be trained to strictly satisfy the conservation laws. Experiments on physical systems with multiple conserved quantities show that the proposed model can capture multiple conserved quantities and predict the dynamics of the state more accurately than the base model.
strengths_and_weaknesses: - Originality
  - Pros 1: The problem setting of inferring multiple unknown conserved quantities at the same time is new and interesting.
  - Pros 2: Inforporating the prior knowledge of the first integral is reasonable.
  - Pros 3: The formulation based on discrete derivatives is useful.

- Quality
  - Pros 1: This submission is technically sound.
  - Pros 2: Experiments with physical systems with multiple conserved quantities have shown the effectiveness of the proposed model.
  - Cons 1: The description of the learning algorism is somewhat lacking.
  - Cons 2: Only a comparison with the base model is presented.

- Clarity
  - The presentation is excellent.

- Significance
  - Pros 1: Data-driven Approaches to modeling physical systems have attracted considerable attention. It is useful because it has the potential to construct simulators from data for physical phenomena for which mathematical models are unknown.
  - Cons 1: The application is not stated in detail.
questions: Q1. When the number K of conserved quantities is unknown, do you have a way to determine K?

Q2. It would be better to specify the objective function and/or the learning algorithm. I was wondering if it is really appropriate to use 1-step error for learning. In [R1,R2], learning is performed using time-window, which might be useful for this study.

Q3. The evaluation experiments are carefully conducted, but I was concerned that the only comparison method is the base model. If possible, it would be better to have a comparison with at least one of the latest models (e.g., DGNet).

Q4. How robust is the proposed model when varying the number of trajectories and time resolution in the training data? It might work well for sparse data because multiple conserved quantities are considered.

[R1] Kevin Course, Trefor Evans, and Prasanth Nair. Weak form generalized Hamiltonian learning. In Advances in Neural Information Processing Systems, volume 33, pages 18716–18726, 2020.

[R2] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Symplectic ODE-Net: Learn- ing Hamiltonian dynamics with control. In International Conference on Learning Representa- tions, 2020.
limitations: We believe that learning dynamics via a data-driven approach is effective in the following ways: 1) it allows us to construct simulators even when the mathematical model is unknown, and 2) it can lead to scientific discoveries (e.g., conservation laws). In order to clarify the significance of this research, we would like to ask for clarification of possible applications. In particular, please share your thoughts on how this could lead to scientific discoveries. If there is a gap, it would be good to mention it as a limitation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose modify the Neural ODE model to conserve the first integrals of the physical system, where the  first integrals are either known or learned by a neural network.
strengths_and_weaknesses: The authors propose a neat way to introduce the conservation constraint by adding the Lagrange multipliers to the derivative function of the Neural ODE. The first integrals are conserved exactly, which, to my knowledge, previous models do not provide. The first integrals can be learned from data, and the authors show that they correspond to the known conservation, such as conservation of mass and energy. It is notoriously hard to preserve the first integrals of the system in the learned physical simulations, and this work provides an elegant way to do so.
questions: - Can the authors clarify the difference between the cFINDE and dFINDE? cFINDE is derived for continuous functions f(u) and V(u), but in practice, it surely uses an integrator to compute the ODE solution. dFINDE also uses a numerical integrator for Psi: “We assume that the base model ψ is composed of a continuous-time base model and a numerical integrator.” Is the only difference between the models in the computation of the grad_V (u)?

- In case first integrals are learned, how do you prevent it from learning degenerate solutions, e.g. predicting zero function for all inputs u? My understanding is that function f(u) is trained together with the learned first integrals V(u). In the best case, the function f(u) can already preserve all or some of the first integrals, and V(u) can only learn a degenerate function.

- In Figure 1 authors demonstrate that leapfrog integrator does not preserve the total energy, while dFINDE does, while dFINDE uses Dormand–Prince method (dopri5)  (mentioned in section 4.1). Presumably, the ability to conserve the integral of dFINDE heavily depends on the integrator used. It is unfair to compare dFINDE equipped with dopri5 versus leapfrog integrator. Can authors provide the results or hypothesise about the following comparisons: 1) leapfrog versus dFINDE with leapfrog integrator 2) dopri5 versus on dFINDE with dopri5 integrator?

- Have authors tested the model on the systems with different masses and total energy than on the training set to verify that the learned constraints indeed learn to compute the  total energy/mass that need to be conserved?

limitations: I strongly suggest the authors to acknowledge the limitations of their work.

Some limitations worth mentioning in the paper:
- The ability of the cFINDE and dFINDE to preserve the first integrals depends on the numerical integrators. dFINDE also depends on the time discretisation delta_t. 
- The results in Table 3 suggest that the model is sensitive to K — the number of the first integrals chosen. Moreover, with K>3 the predictions “explode” in case of 2-body in 
- FitzHugh–Nagumo systems.
- By construction, enforces that certain quantities of the system have to be conserved exactly. The model is not applicable for the systems that are conserving, e.g. having dissipative energy.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The objective of the paper is to propose a method that enables to find (and preserve) invariant quantities in the observations of a dynamical system. 

The authors proposed in their work  the joint learning of a derivative ($f$) approximating the dynamics of the observations and parameterized first integrals $V=(V_1,..., V_k)$  that are used between each prediction step. Several technical developments are conducted to enforce the respect of the first integrals.


### POST REBUTTAL ###

I would like to thanks the authors for their rebuttal statements. The proposed explanations details and submission modification for the unclear parts are satisfying (notably on the projection and the discretization errors).  Therefore, I maintain my score.

strengths_and_weaknesses: S:  The paper and its objective are clearly presented and very relevant to the community and the development of further methods.
Moreover,  the paper is well-written and the technical contributions of the paper are clear.


W: I identify two major weaknesses. 
The first one lies in the presentation:  the loss objective is not clearly written in the main text which slightly hampers the understanding of the text. Indeed, the authors remain quite far from the application to focus on their technical contribution.Far from being essential, I nonetheless highly recommend the inclusion of an “algorithm” section to detail how the first integrals are learned practically using for instance NN. 

The experiments are conducted using low dimensional data (and it presents a clear case of the potential of the proposition in this case) but the application of the proposition to  higher dimension data is not answered.


questions: 1) Section 3.1 provides the dynamics that respects the invariants given $\hat{f}$ and lagrange multipliers. I suggest for clarity to present the optimization problem associated with these $\lambda$. 

2) Can the authors also detail why the continuous version is subject to numerical errors compared to the discrete ones ?

3) What is the method used to solve eq.3 (at each iteration) ? Or is the integration performed using the “modified” (i.e c/d Finde) version of the learned dynamics ?

4) L.144: it is unclear to me why gradients are not computable. Indeed solving eq.3 can be thought of as a meta-learning problem (yet the optimization would not be exact). 

5) I get that the whole point of the method is to provide a way to “respect” first integrals, i.e. to “alter” an initial integrator so that $(V_1,..., V_k)$  are constant. However, since it is done pointwise it allows potential spikes in the learned first integral as for instance illustrated in fig.4. Can the authors comment on this and potential implications (for instance it severely arms the interpretability of the learned first integral )? Can the authors think of any additional constraints or inductive biases to palliate such drawback ?

6) What is the influence of the hyper-parameters on the learning mainly on the relative influence of the LR of the networks associated with the first integral ? 

limitations: see questions.
review_title: null

Meta-review: metareview: Although all reviewers find an interesting point and the significance of this paper, some reviewers have several critical concerns in the paper such as the readability (very hard to follow, and lacking some important information) and the non-convincing empirical evaluations. Although it seems that the authors could answer parts of these concerns in their responses and , those seem to require much modification from the original draft. As a total, although this paper could be a good paper after reflecting the reviewers' comments, my recommendation for the current form of this paper is rejection from the relativistic perspective, compared with papers accepted to NeurIPS.
confidence: Certain
Decision: Reject