Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a neural network based approach to preserve and discover first integrals in the underlying target systems. In this paper, authors propose two instances of the approach, namely cFINDE and dFINDE, which works in continuous-time and discrete-time respectively. The proposed FINDE method is a great exploration in the area of preserving and/or discovering invariant quantities when learning dynamical systems with deep learning methods, and it is a generalization of existing deep learning based methods. As I see, the FINDE method makes a solid contribution.  
strength_and_weaknesses: The main strengths of this paper are 

$\mathbf{1}.$ on the proposal of enhancing the neural network based approach to learn dynamical systems by proposing an additional structure/equation to preserve and/or discover invariant quantities. 

$\mathbf{2}.$ This paper is very well organized, written and illustrated with graphic results. The background knowledge is adequate for readers (relevant in deep learning fields) to understand the rationale and the intuition driving the proposal, and the paper is very readable.  

There are a few weakness, which could make the paper better but should not overshadow the strength

$\mathbf{1}.$ The numerical results can be explained and presented better. For example, the switch between dFINDE and cFINDE could be explained better as they often make a bit confusion in the text; the plots can be improved with better presentation: for instance, Figure 3 - it can be shown as four subplots, each of which can compare HNN vs HNN+cFINDE; the plots could confuse audience a bit: for instance, it is hard to see the ground truth. 

$\mathbf{2}.$ It is very interesting to see the big difference induced by using different values of $K$. As discovering the invariant quantities is one of the cores of this approach and also as author addressed in the conclusion, conducting more comprehensive and in-depth numeric investigation on the role of $K$, with more detailed discussion, along with more analysis in theory, would benefit a lot in helping audience understand and the field leverage the proposed methods.    
clarity,_quality,_novelty_and_reproducibility: $\cdot$ This paper is very well clearly written and presented, with a great deal of details in introducing the background, literature review, intuitive explanation, theoretical analysis and numerical study. 

$\cdot$ The novelty of this paper is obvious as it proposed novel approach to generalize and enhance the existing literature to capture and discover first integral of the neural-network based dynamic system learning methods. The implementation is straightforward.

$\cdot$ This paper has detailed explanation of theoretical analysis, implementations, and experiments. And, it also listed most (if not all) of the used libraries that author used in the implementation.  
summary_of_the_review: As stated in the above comment sections, this paper has a solid contribution with a novel approach. It is a good paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper asks a very pertinent question as to how we can incorporate knowledge of conserved quantities when solving PDEs via nets. The suggested method is quite unclear. 
strength_and_weaknesses: The main strength of the paper is that it raises and important question and that it has tried some serious benchmark tests. Also interestingly this paper tries to marry the ideas of symbolic regression with ideas of neural training to discover conserved quantities. Definitely a worthwhile attempt. 

Prima facie there seem to be major weaknesses in the paper.  

For a start V seems to be a neural net whose output coordinates encode the conserved quantities. But is V being trained? If yes, how? If I understand correctly -- the function f in Theorem 1 is also a neural net. Is that being trained? Its entirely unclear to me as to how these nets are getting determined. If there is a loss function somewhere that is getting minimized then that is extremely unclear. The paper is in dire need of a pseudocode! 

The testing of preservation of the conserved quantities looks very ambiguous to me. While the authors want to use this as one of the main tests it doesn't seem to have a clear definition till Appendix A (Equation A13?) - and even then it seems to be a function of time while the Table 3 seems to suggest that this is a number that characterizes the whole solution.  And if we focus on Table 3 then it seems that at no value of K is the new method beating the benchmarks in this 1-step criteria for this 2-body example. Isn't that a strong argument against the new method? 

Further, in Appendix D.2 the authors seem to clearly state that their approach of symbolic regression is not leading to a correct identification of the conserved quantities for their 2-pendulum and the FitzHugh-Nagumo datasets. So what exactly is the claim of the paper at all? Doesn't this beat the central claim in the paper? 

Using KdV as a test-case is a quite tricky. Firstly there seems to have be a typo in equation A18 when quoting the KdV. Further KdV has infinite number of conserved quantities. The authors claim to have cut that down to some finite number by some discretization. This is entirely unexplained in the paper - and potentially a very fatal issue since that sounds like trying to entirely change the target from trying to solve KdV to something else! 
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is extremely low. 
I am unable to figure out even basic things here like explained in the above point. 

  

summary_of_the_review: Given the weaknesses described above, I feel there are major gaps here - either very fundamental issues have gotten overlooked from the study or the presentation has failed to communicate them to the reader.  The paper needs a major rewrite in the very least.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes first integral-preserving neural differential equation (FINDE) that can learn a dynamical system and its invariant quantities (i.e., first integrals) from observed data *without* prior knowledge (e.g., conservations of energy and momentum) or assumed geometric structures (e.g., Hamiltonian or Lagrangian) on the target system. FINDE consists of two neural networks; a neural network $\hat{f}(u) \in \mathbb{R}^N$ for modeling the target dynamics (i.e., neural ODE) and another neural network $V(u) \in \mathbb{R}^K$ that outputs $K$ first integrals. Then, FINDE projects the neural vector field $\hat{f}(u)$ to the directions in which $V(u)$ is preserved, by using the projection method. Thus, it guarantees the state $u$ evolves on $N-K$ dimensional sub-manifold (level set) which preserves first integrals. The optimization procedure of the projection method is well-formulated by using Lagrange multipliers, and it yields a novel type of neural ODE (continuous FINDE, cFINDE) that can learn conservative dynamics with repsect to the modeled frist integrals. The authors also present a discrete version of the proposed FINDE (dFINDE) to preserve the learned first integral under the discretization of numerical integrators. The authors theoretically show cFINDE and dFINDE exactly preserve the modeled first integrals $V(u)$ under the continuous and dicrete cases, repectively. The authors validate their approach with four dynamical systems including simple two-body problem and complicated KdV system.
strength_and_weaknesses: Strength:

1. The motivation of this paper is clear. Extracting underlying invariant quantities as well as learning dynamics *without* prior knoweldge on the target system is definitely important and challenging problem in physics. This paper provides a solid contribution to such an important problem. It might be one of good references for automated scientific discovery via machine learning tools.

2. The paper is technically sound. The used projection method seems to be a principled way to preserve arbitrary first integrals. The authors properly modifies the projection method optimization as a learnable model (FINDE) based on neural ODE framework. 

3. The paper is clearly written and easy-to-follow.

4. This work is highly reproducible. All used model architecture and hyper-parameters are well summarized in the main text of this paper. Experimental codes were made available.

Weakness:
1.  $K$, the output dimension of $V(u)$, is very important hyper-parameter because it directly determines the number of invariant quantities. Inappropriate setting of $K$ rather deteriorates the predictive performance of the model as depicted in the experimental results. In the current version of this paper, the authors do not clearly mention how one can determine such a critical hyper-parameter for unknown dynamical systems. It will be nice if the authors elarborate it.

2. I found a prior work [1] that seems to be very relevant to the proposed method. [1] uses PCA to find $N-K$ dimensional sub-manifold of trajectories and extracts $K$ invariant quantities. It will be nice if the authors cite [1] and compare pros and cons of the proposed method with [1] briefly.

***
[1] Liu, Z., & Tegmark, M. (2021). Machine learning conservation laws from trajectories. Physical Review Letters, 126(18), 180604.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written with sufficient details on the proposed algorithm.

Quality:

This paper tackles an important problem of physics and proposes a simple solution to resolve it. However, I have a concern regarding a hyper-parameter $K$ which might be very critical.

Novelty:

The use of the projection method for learning invariant quantities is straightforward (principled) yet novel to me. While the authors provide a thorough list of related literatures, however, [1] might be an important yet uncited work that very relevant to this work.

Reproducibility:

The authors provide a runnable code as well as a detaild description on the experimental setting.
summary_of_the_review: Overall, I believe the paper is very interesting, tackles an important problem, and thus is worthy of publication. However, the concern on $K$ tuning prevents me from recommending a clear accpetance of the paper in its current form. Currently, my evaluation is weak accept, however, I would like to increase my score if my concern is well addressed.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new method to preserver first integrals in neural network modeling called first integral-preserving neural differential equation (FINDE) with the claim that FINDE can preserve and discover first integral in data without needing explicit information about the underlying structure. The authors propose two versions of FINDE: cFINDE (continuous integration) and dFINDE (discrete integration). In its introduction, the paper outlines the importance and challenges related to finding first integrals from data and which first integrals can be especially relevant for computer simulations (Energy, momentum, mass, etc). Next, the authors formally define first integrals and describe related work of first integrals in numerical analysis and neural networks that preserve first integrals, including Neural Ordinary Differential Equations, as well as works that have built on top of it and different kind of architectures (holonomic neural networks - HNN). The paper then introduces and defines the two versions of FINDE: cFINDE and dFINDE with relevant theorems explaining important preservation properties related to first integrals for both formulations.

In their experiments, the authors evaluate cFINDE and dFINDE on relevant datasets that contain different kinds of first integrals. In their implementation, the authors leverage an HNN,  apply related loss formulations to train cFINDE and dFINDE, and define valid prediction time (VPT) as an evaluation metric. The first set of experiments investigate whether first integrals are preserved in FINDE via a mass spring example. The second set of experiments show results on finding known integrals with FINDE compared to using vanilla HNN with the results indicating that FINDE methods generally perform better. The third set of experiments deal with finding first integrals in unknown systems using a set of examples (KdV, pendulum). The results generally indicate that FINDE methods are able to reproduce the ground truth compared to neural ODEs.
strength_and_weaknesses: **Strengths**

* The paper introduces a novel formulation for a relevant problem in applying neural network to physical systems and computer simulations thereof. The FINDE methods are described and derived with detail and their properties tested as part of the experiments.
* The paper provides a relevant set of experiments to assess the properties and performance of FINDE, both cFINDE and dFINDE and seems to achieves superior performance compared to other methods.

**Weaknesses**

* The paper could be improved by having more detailed Figure captions across all figures, especially the one outlining the results (e.g. Figure 1, Figure 4 and Figure 5). More detailed captions would provide greater clarity on what the data show and how to interpret the results.
* The authors could further clarify potential sources of error in FINDE. The current version explains how dFinde resolve the time discretization error in cFINDE, but it would be good to have a more information on other sources of error created by the approximation and they can potentially be resolved. E.g. In any finite difference scheme there is approximation error, such as truncation errors coming from the discretization of the equation.

**Additional Questions**

* You mention that FINDE preserves multiple quantities at the end of Section 2. Could you explain more how that occurs? Specifically how does preserving multiple quantities (e.g. energy and mass) differ from preserving just one quantity (energy) related to your method?
* Given the additional advantages of dFINDE, in what situations would someone choose cFINDE over dFINDE?

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is generally well-written with relevant details described. Figures could be improved with more detailed captions to guide the reader.

**Quality**

The quality of the writing and experiments is generally strong.

**Novelty**

The novelty is generally strong and tackles a relevant problem.

**Reproducibility**

The authors provide relevant detail and code for their experiments, as well as a reproducibility statement.
summary_of_the_review: The paper introduces a novel method and provides a relevant set of experiments to showcase the abilities of the method. Improvement can be made to further strengthen the paper in subsequent revisions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper uses a neural network based approach to discover the structure of dynamical systems in terms of first integrals. A discrete and a continuous time version is presented. The paper also presents a nice set of experiments and benchmark for this domain.

Strengths + Weakneeses: The paper presents a solid framework for learning the structure of dynamical systems or ODEs. Most reviewers found the approach very interesting and the experiments exhaustive and compelling. On the negative side, reviewer pYdf had issues regarding the clarity of the paper and also misunderstood some of the results wrong. These issues have been properly addressed, yet, the reviewer did not respond to the rebuttal, hence I weight his review less. Overall, the remaining reviewers were enthusiastic about the paper and I recommend acceptance.
justification_for_why_not_higher_score: Reviewers mention issues in terms of presentation of the results as well as the importance of correctly chosen hyper-parameters.
justification_for_why_not_lower_score: The points of the negative review (score 3) were addressed in the rebuttal. Unfortunately, the corresponding reviewer was not responding and did not update his scores. All other reviewer have a good opinion about the paper (average score of 8) and were much more proactive in the discussion, hence, I weight there opinion higher in my evaluation.
Decision: Accept: poster