Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a method to impose an acquired conservation law on a learned time-evolving model such as a DNN.
The proposed method consists of a two-step approach. The first step is invariant learning, in which underlying conservation laws are estimated from trajectory data, and the second step is dynamic learning, in which the time evolution model is subjected to preservation of conservation laws. Numerical experiments show that the proposed method consistently outperforms the baseline method in both coordinate error and conservation measures and can be further extended to complex, large-scale dynamics by leveraging autoencoders. The proposed method may also be useful in discovering conservation laws in unknown dynamical systems.
strength_and_weaknesses: Strengths
The framework of estimating hidden conservation laws from data and then improving simulation accuracy based on the estimated conservation laws is highly novel.


Weaknesses

A weakness of the research is that the effectiveness of the proposed method has not been verified because there is no theoretical analysis to guarantee the effectiveness of the method and the numerical experiments are limited to simple systems for which the conservation laws are known.
Concrete weaknesses are that the following two points have not been validated.


1. the effectiveness of the proposed method for learning the energy function

The denominator of the "invariant learning" loss function (Eq. (3)) seems to be based on the good properties of the energy terrain.
For example, it is questionable whether it would work effectively in the case of a complex energy function with multiple peaks, such that starting from different initial conditions would result in the same energy value.
In addition, the behavior of the model when the number of data is small relative to the degrees of freedom the system has is unclear.
Numerical experiments with more complex energy landscapes and a discussion of the relationship between the number of data and learning performance would be needed. 


2. effectiveness of the proposed method as a method for estimating unknown conservation laws
The authors mention the application of the proposed method to the estimation of unknown conservation laws, but its effectiveness is questionable in terms of interpretability and feasibility of estimating complex conservation laws.


2-1. Interpretability
All of the previous studies cited in section "2.3 LEARNING WITH CONSERVED PROPERTIES" seem to achieve interpretable conservation law estimation.
On the other hand, the proposed method does not seem to be able to achieve interpretable conservation law estimation.
Methods for estimating interpretable conservation laws from DNNs trained on dynamical system data have been proposed [1,2].
I think that an additional discussion based on such previous studies seems necessary.


2-2. Possibility of Estimating Complex Conservation Laws
The authors claim that the proposed method can estimate complex conservation laws.
As an example, they demonstrate the estimation of an angular momentum conservation law for the Kepler system.
However, this task is not so difficult.
Conservation law estimation for the Kepler system has been realized in many studies [1,2].
In order to claim the effectiveness of the proposed method as a conservation law estimation method, it is necessary to at least realize conservation law estimation corresponding to symmetries for nonlinear transformations such as Runge-Lenz vectors.
Non-linear conservation estimation is also mentioned in [1][2]. 
Also, [3] achieves it using Hamiltonian Neural Networks by estimating the mass tensor.

[1]Ziming Liu and Max Tegmark, "Machine Learning Hidden Symmetries," PRL, 128, 180201, 2022.

[2]Yoh-ichi Mototake, "Interpretable conservation law estimation by deriving the symmetries of dynamics from trained deep neural networks," PRE, 103, 033303, 2021.

[3]Nate Gruver, Marc Finzi, Samuel Stanton, Andrew Gordon Wilson, "Deconstructing the indirect biases of hamiltonian neural networks," ICLR 2022.
clarity,_quality,_novelty_and_reproducibility: quality:

The manuscript was of high quality with no typographical errors or omissions.


clarity:

The paper was clear except for the following two points
The meaning of x∼C∼D in equation (3) was unclear.
In section 4.3, I did not understand why the reduced space was created.
If we bring all possible initial states, does the system still have 9 dimensions of freedom?


originality:

Originality is considered to be high enough to be published.


reproducibility:

Publication of the source code for the numerical experiments is desired.
summary_of_the_review: The approach of estimating hidden conservation laws from data and improving the accuracy of simulations based on these conservation laws is an innovative research, and the novelty of this research fully satisfies the conditions for acceptance for presentation at the conference.
On the other hand, the evaluation of the effectiveness of the proposed method is weak, and the validity of the study is questionable.
I believe that additional discussion and numerical experiments to improve these points would be a condition for the conference to accept the study.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Authors propose a contrastive learning approach to learn conserved quantities of a dynamical system using an auxiliary neural network. The neural network learns through positive samples from the same trajectory of the dynamical system and negative samples from different trajectories. The conservation is enforced during prediction by projecting the outputs from a forecasting neural network onto the conservation manifold. 
strength_and_weaknesses: Learning conservation laws from observed trajectories is an important task to study a dynamical system and can also help in generalization. To my knowledge, the contrastive learning approach is novel and interesting. 



**Weaknesses**

1. The paper does not adequately showcase the utility of learning conserved quantities this way. Since conserved quantities here are learnt as a NN representation, they are not interpretable. As mentioned by the authors, the learnt quantities may be a non-linear function of the (true) relevant conserved quantities. Additionally, as seen in Figure 4, the approach is not always able to enforce the learnt conservation law properly to the forecasting network.
2. Paper does not empirically compare with any other baselines that learn conservation laws. Only comparison provided is a standard neural network which does not learn the conservation law, as expected.
3. Writing in Section 3 is rushed (possibly with mistakes) and can be improved with more details about the equations. Section 4 (and Appendix) does not provide adequate details about the experiments & training/testing methodology, and a few plots are hard to parse due to no labels. 



**Questions**

1. Except the ideal spring mass experiment, every other experiment considers energy conservation where Hamiltonian/Lagrangian networks can be used, potentially with additional rotational symmetry [1, 2] for angular momentum conservation. I think these methods should be added as baselines in these experiments. Further, the paper would be much stronger if there were more experiments like spring mass experiment that conserve quantities other than energy. 
2. The experiment section should include comparison with other baselines that learn conservation laws from trajectories (e.g., [3, 4] cited in the paper). 
3. In Equation (2), $t_1$ is not defined, should this be $t$? Can $k = i$ in this equation? 
4. In Equation (3), the expectation is defined over $x^+$ and then the summation inside is also over $x^+$. There is the same issue for $x^-$. In Equation (4), there seem to be a lot more negative samples compared to positive samples, does this not introduce some bias in learning?
5. This framework does not seem to allow for the case when two different trajectories have the same conservation value. For example, trajectories with same energy but phase-shifted. Is this true?
6. Please provide details of the experiments (in Section 4 or Appendix), for example, what are the time steps the trajectories were observed during training and the same during testing. Specifically, are the methods asked to predict beyond the times seen during training?
7. Figure 3: Please provide y-labels for all plots. In Mass/Energy conservation plots, are the true values of mass/energy learnt by the neural network H?
8. Figure 4: Please provide x-labels for plots of top row; I am assuming it is time(s) but then there is discrepancy in x-axis between top and bottom rows. 
9. In Section 4.2, could you please elaborate on what type of trivial solution does the model learn. It is also not clear to me why Equation (3) will be biased toward “similarity” within trajectories, and why the proposed solution solves the issue. 


**References**

[1] Finzi, Marc, et al. "Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data." International Conference on Machine Learning. PMLR, 2020.

[2] Finzi, Marc, Max Welling, and Andrew Gordon Wilson. "A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups." International Conference on Machine Learning. PMLR, 2021.

[3] Ha, Seungwoong, and Hawoong Jeong. "Discovering conservation laws from trajectories via machine learning." arXiv preprint arXiv:2102.04008 (2021).

[4] Liu, Ziming, and Max Tegmark. "Machine learning conservation laws from trajectories." Physical Review Letters 126.18 (2021): 180604.
clarity,_quality,_novelty_and_reproducibility: Paper is clearly written but quality can be improved with more details in Section 3 & 4. Learning conserved quantities using contrastive learning is new. 
summary_of_the_review: My main concerns are regarding the utility of learning conservation laws this way as these NN representations are not interpretable and may not be easy to enforce onto the forecasting network. Also, paper does not adequately compare with any other baselines that learn conservation laws from trajectories.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a contrastive learning framework to model conservative dynamical systems with hidden conservative laws. More specifically, the proposed framework consists of two networks; an invariant representation network that maps elements of a certain trajectory to an identical latent features $H_{\theta_c}(x) \in \mathbb{R}^m$ which mimic $m$ invariant quantities, and a dynamics network  $\hat{f_{\theta_d}}(x) = \text{Projection}({f_{\theta_d}}(x), f: \langle f, \nabla_x H_{\theta}(x) = 0 \rangle)$ that models the targeted conservative dynamics by projecting a neural ODE vector field ${f_{\theta_d}}(x)$ to the orthogonal direction of  $\nabla_x H_{\theta}(x)$. The former and latter are trained via a neightborhood conservative loss and a standard MSE loss, respectively. The authors validate their proposed method for two simple dynamical systems (two variables with one conservative law), Kepler system (four variables with two conservative laws), and heat equation on a 1D rod (101 mesh points and one conservative law).
strength_and_weaknesses: Strength:
1. This paper tackles an important and challenging problem of physics (can machine learning models learn conservative dynamics and extract invariant quantities from observed trajectories without using prior knoweldge (e.g., conservation of mechanical energy) or assuming underlying structures (e.g., Hamiltonian)?).

2. To solve the above important problem, the authors introduce a constrative learning framework that can learn invariant quantities via a representation learning way. To me, this idea is novel.

3. The paper is generally easy-to-follow.

Weakness ( and Questions):
1. I am not sure whether the contrastively-learned invariant latent features, i.e., the modeled invariant quantities, can be valid for unseen initial conditions. I think the authors should report generalization performance of the proposed method to ensure it.

2. $m$, the latent feature dimension, seems to be very important hyper-parameter because it directly determines the number of invariant quantities. The authors do not explicitly state how one can tune such a important hyper-parameter when one does not know the exact value of $m$ for an unknown physical system, thus I am not sure the robustness of the proposed method in this paper.

3. Espeically, for a heat equation problem, the authors use an auto-encoder that maps an observation to $d$-dimensional laten space. It means that the modeled dynamics is $d - m$ dimensional sub-manifold and it seems that both $d$ and $m$ seem to be very sensitive hyper-parameters when learning the dynamics and invariant quantities accurately.

4. Experimental results seem to be weak. The proposed method fails to accurately learn the 4-variable Kepler system. Also, the authors only compare their model with a simple vanila network. It would be nice if the authors compared their model with some advanced models, e.g., HNN or [1].

5. Can the proposed model learn a non-conservative systems such as a dissipative one?

6. The core idea of this paper is that a certain state should have an invariant representation with respect to its time evolution. While the contrastive learning is one of good candidates to finding such a representation, there are various other methods that can enforce the invariance, e.g., consistency regularization [2]. What motivated the authors to use the contrastive learning?

***
[1] Liu, Z., & Tegmark, M. (2021). Machine learning conservation laws from trajectories. Physical Review Letters, 126(18), 180604.

[2] Sinha, S., & Dieng, A. B. (2021). Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34, 12943-12954.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is well-written and easy-to-follow.

Quality:

As I mentioned in the main review, I have some concerns on (1) generalization performance, (2) hyper-parameter tuning, and (3) weak experimental results with limited comparisons.

Novely:

The idea of using the contrastive loss to model and extract invariant quantities is novel for me.

Reproducibility:

The paper contains an incomplete set of used model architectures and hyper-parameters. Code is not made publicly available.


summary_of_the_review: Although the paper is interesting, I think there is still much room for improvement, e.g., see Weakness (and Questions). My current evaluation on this paper is borderline reject.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a technique to preserve physical conservation laws in neural network based dynamical systems. The authors propose to solve two tasks concurrently: (i) learning variables that remain constant for a given trajectory, and (ii) a dynamics function that respects the constant variables and that fits input trajectories. The first objective is achieved by adopting well-known contrastive learning objectives so that the new objective "pulls the latent factors" (constant) within a trajectory whereas it "pushes them away" if they belong to different sequences. The second goal is realized by a gradient-matching loss. The model is experimentally demonstrated to preserve constant terms oftentimes and hence leads to non-divergent trajectories.
strength_and_weaknesses: Strengths:
- The model surely learns _some_ constants that remain the same throughout integration.
- The framework is straightforward: Similar contrastive objectives are commonly used and well understood. The dynamics function is also easy to understand as it is built to preserve the constant variables.

Weaknesses:
- The paper suffers from clarity issues as well as grammatical mistakes. I believe it should undergo significant updates. See my comments at the end of this question.
- Methodology should be motivated. The authors argue that "this is the first work that studies discovery of conservation laws for general dynamical systems through contrastive learning." What if we had a simpler method in which certain global parameters remain constant throughout integration? Would the proposed approach still be preferable? If so, why? Furthermore, among all other techniques, why is contrastive learning chosen? It does come with a complicated and costly loss term; so what is the catch here instead of a simpler approach?
 - The gradient matching term (eq. 7) should be carefully analyzed. It is well known that this objective becomes problematic with increasing noise levels and bigger time differences as the differential function is directly fitted to the empirical gradients $\frac{x_{i+1}-x_{i}}{t_{i+1}-t_{i}}$. This is in contrast to learning a generative process, such as GP or neural ODEs, and is the very motivation of these approaches. 
- The paper lacks a whole discussion on causality. In my understanding, the method discovers certain factors on which the trajectory (forward simulation) is conditioned. This is the very definition of causal learning while the paper does not touch upon this connection. Similarly, the connections between invariance and causality could be included (in addition to the short discussion on invariance that is already included).
- The method should be compared against SOTA baselines. I do acknowledge that the literature on "black-box conservation learning" is quite limited but at least Lagrangian NNs or some adaptation of vanilla neural ODE systems could be considered here. This would help us judge the level of difficulty of the problem. Similarly, we should see how the performance would change if we consider higher dimensional latent spaces ($m>2$) in Sec 4.2. 


Writing issues:
 - "deploying the knowledge towards the new occurrence" is unclear.
 - "conservation law were" is wrong
 - The connection between the sentence starting as "On the other hand, data-driven" and the prior sentences is missing.
 - "one or more distinguishing features of physics-based system trajectories" This sounds mysterious, i.e., what are the examples of such features?
 - "By comparing the latent space distance of the system state observation and assigning them into their trajectory" Both parts of the sentence is unclear. What is latent space distance? How does "assignment" take place?
 - What does "ConCerNet" stand for?
 - "of the simulation in the longer term" Longer compared to what?
 - Naming the function approximations already on the first page would help follow the text.
 - "projecting the dynamical neural network output towards the" Perhaps "on the"?
 - "conservation manifold from the first one" is unclear. Perhaps something like "the manifold learned by the first module"? 
 - "guaranteeing" instead of "guarantying"
 - "Unlike discriminative models that explicitly learn the data mappings" What mapping is referred to here?
 - Sec2.1 could be organized chronologically (by moving the paragraph starting with "Despite the recent surge of" to the beginning)
 - Neural ODE paper (Chen, Ricky TQ, et al. "Neural ordinary differential equations." Advances in neural information processing systems 31 (2018).) should be cited in Sec 2.2.
 - "however, the soft Lagrangian treatment does not guarantee the model performance during testing" This should be elaborated.
 - "to propose a contrastive learning framework in a more generic form that is compatible with arbitrary conservation." What is meant by "arbitrary conservation"?
 - "total time step T" is not clear.
 - "We let the trajectory set ... be the simulation history starting". I'm not sure what this means: Do $C_k$'s refer to sequence ids?
 - "the initial conditions have various conservation values." What does this exactly mean?
 - "where the probability of point ... **belonging** to the trajectory"
 - Similar to a previously raised concern: What is the motivation for eq.2? What does it imply?
 - \log and \exp should be used instead of log and exp.
 - What does $x\sim C \sim D$ mean?
 - A short discussion on NCA would be nice as it is referred to multiple times.
 - "Softmax like function, ensuring the nice property of probabilistic distribution" What does this mean? What is nice here?
 - "learning of the m conservation terms" Perhaps an m-dim vector is referred to?
 - "by eliminating its parallel component" Parallel to what?
 - Eq,1 uses $f_\theta(x)$ whereas it later becomes $\tilde{f}_{\theta_d}(x)$. What is the difference?
 - Eq. 5-6 are unclear. A (verbal), more intuitive explanation should be given. Projection(.) function should be defined. 
 - "the orthonormalized set of vectors from" is unclear.
 - "conservation term in learned" 
 - x and y axes names and figure titles should be added/revisited.
 - What are the system states in Sec 4.1?
 - Why to use $x[1]$ instead of $x_1$?
 - "the learned conservation here is approximately the exact conservation" is unclear.
 - "coordinate mean square loss to the ground truth" is unclear.
 - Are the equations in the first paragraph of 4.2 really difficult to learn, given the recent success of neural nets on many tasks?
 - Why does the issue that is resolved by eq 8 take place in the first place? Why only on this experiment? Why does eq.8 help at all?
 - Better to use $m=2$ instead of $dim(H_{...})=2$.
 - Table1 could be given at the very beginning of Sec4 as it nicely summarizes all results.
clarity,_quality,_novelty_and_reproducibility: - Clarity: See above.
- Quality: The writing requires significantly more effort. Similarly the experiments require justifications and comparisons with baselines. I believe these considerably diminish the quality of the presented work.
- Novelty: The approach brings two rather known objectives together. Yet, the idea of explicitly encoding the conservation as well as the projection (of the dynamics) are timely and would be beneficial for the community.
- Reproducibility: None of the experiment details is visible in the main text.
summary_of_the_review: I believe encoding state-invariant variables is a nice idea that comes with immediate real-world applications. A contrastive objective to learn such variables is shown to be effective. However, we are unable to tell whether the benchmarks are sufficiently challenging (so that we can see the limits of the framework). We are also not in a position to decide whether simpler ideas or existing methods would perform any better. Moreover, the text requires significant updates, which is why I recommend a reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I will start with the clear strength of this paper, which is the interesting research question it addresses. In my opinion, the right research question is fundamental to interesting and impactful research work.

The submission has led to quite some commotion and back and forth between the authors and the reviewers. However, as also indicated by the scoring, the consensus is to not accept the current work in its current form. If anything, the changes over the original manuscript are significant so that to warrant a new round of reviews from scratch. What is more, motivations and theoretical backing must be sharpened. Quoting one of the reviewers

> The correct way of learning continuous-time systems is forward integration, which derives from the very definition of the learned function, i.e., time differential. However, in this work, a gradient matching (eq.7) based approach is presented, which is how discrete-time systems are learned. The authors response to this comment was "Hamiltonian/Lagrangian NNs also perform gradient-matching". Yet, these approaches do introduce new neural net architectures instead of manifesting themselves as new dynamical learning mechanisms.

What is more, I agree that more baselines should be included. I think the authors have got this suggestion negatively. However, in my opinion this will be a strength of their paper, when they show their method can work decently enough even if no explicit prior knowledge is provided. One cannot claim generality and at the same time do not stress test the model to harder/more general settings to show the benefits. In that case, unless there is clear theoretical proof that the proposed algorithm will work, it is not possible to derive solid conclusions.

Right now, a random reader would at the very least wonder how does the method compare to other works which do have this prior knowledge, and at worse mistrust the findings due to the lack of comparisons. In a way, this is similar to how one uses the accuracy obtained by supervised training to compare with unsupervised pre-trained models on downstream tasks: one gives the supervised training results to see how far is the model from what already exists out there, with the benefit of making fewer assumptions.

All in all, I believe it is fair for the authors to go through the paper once more, and I am sure it will be read with great interest.


justification_for_why_not_higher_score: As written in the metareview, there should at least be more extensive experimental comparisons, and certainly one more round of reviewing given the significant changes and the back and forth with the reviewers.
justification_for_why_not_lower_score: The paper is certainly interesting in its core and with a great research question.
Decision: Reject