Imposing conservation properties in deep dynamics modeling via contrastive learning
Authors: Wang Zhang
Subhro Das
Tsui-Wei Weng
Alexandre Megretski
Luca Daniel
Lam M. Nguyen
Venue: Submitted to ICLR 2023 (2023)
Keywords: dynamical system modeling
contrastive learning
learning conservation property

Abstract: Deep neural networks (DNN) has shown great capacity of modeling a dynamical system, but these DNN-based dynamical models usually do not obey conservation laws. To impose the learned DNN dynamical models with key physical properties such as conservation laws, this paper proposes a two-step approach to endow the invariant priors into the simulations. We first establish a contrastive learning framework to capture the system invariants along the trajectory observations. During the dynamics modeling, we design a projection layer of DNNs to preserve the system invariance. Through experiments, we show our method consistently outperforms the baseline in both coordinate error and conservation metrics and can be further extended to complex and large dynamics by leveraging autoencoder. Notably, a byproduct of our framework is the automated conservation law discovery for dynamical systems with single conservation property. 

Under review as a conference paper at ICLR 2023
IMPOSING CONSERVATION PROPERTIES IN DEEP DY- NAMICS MODELING VIA CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNN) has shown great capacity of modeling a dynamical system, but these DNN-based dynamical models usually do not obey conservation laws. To impose the learned DNN dynamical models with key physical proper- ties such as conservation laws, this paper proposes a two-step approach to endow the invariant priors into the simulations. We first establish contrastive learning framework to capture the system invariants along the trajectory observations. Dur- ing the dynamics modeling, system invariance. Through experiments, we show our method consistently out- performs the baseline in both coordinate error and conservation metries and can be further extended to complex and large dynamics by leveraging autoencoder. No- tably, byproduct of our framework is the automated conservation law discovery for dynamical systems with single conservation property.
design a projection layer of DNNs to preserve the
INTRODUCTION
With the quick growth of computational resources and massive prediction power of neural networks, recent times have seen great success of artificial intelligence in a wide range of applications such as image classification (He et al.) 2016). natural language processing (Vaswani et al., [017]] Devlin al. (2088) and reinforcement learning (Mnih al. 2013). Despite the scalability and diversity of modern machine learning tasks, extracting the underlying latent mechanism from training data and deploying the knowledge toward the new occurrence have always been the heart of artificial intelli- gence. The idea to work with compressed representation or prior information has been historically entangled with the development of machine learning, from earlier tools like clustering and principal component analysis (PCA) to more contemporary autoencoders or embeddings. In recent years, there is a surge in interest to discover knowledge from larger or even different domains leveraging techniques like representation learning (Bengio et al.) 2013) Chen & He. 2021) and transfer learn- ing (Weiss et al. 2016) to more general meta-learning (Rusu et al. 2018 Santoro et al., 2016) and foundation models (Bommasani et al. 2021) which are capable of handling wide range of tasks.
Many critical discoveries in the world of physics were driven by distilling the invariants from ob- servations. For instance, the Kepler laws were found by analyzing and fitting parameters for the astronomical observations, and the mass conservation law was first carried out by a series of exper- iments. However, such discovery usually requires extensive human insights and customized strate- gies for specific problems. This naturally raises a question, can we learn certain conservation laws from real-world data in an automated fashion? On the other hand, data-driven dynamical modeling is prone to violation of physics laws or instability issues (Greydanus et al.] 2019 Kolter & Manek, 2019), since the model only statistically learns the data or system state function without knowing physics prior.
In this paper, we provide a novel contrastive perspective to find one or more distinguishing features (i.e. conservation values) of physics-based system trajectories. By comparing the latent space dis- tance of the system state observations, we aim to learn a low-dimensional representation potentially serving as the invariant term for the system. With such inspiration, we propose ConCerNet con- sisting of two neural networks. The first network contrastively learns the trajectory invariants, the second network captures the nominal system dynamical behavior which will be corrected by the first network to preserve certain properties of the simulation in the long term prediction. The correction implemented by projecting the dynamical neural network output on the learned conservation man-

---

Under review as a conference paper at ICLR 2023
Step 1:
Unknown Dynamical System
Contrastive Learning Conservation
Learned
NNI: invariant learning
minimize
k K,A K
[[H(K) KODil
A
maximize
[1(()))) M(x))l
B A+ Constant
Prediction: better conformity to ground truth
Observed Noisy Trajectories
Dynamics Projection Conservation Manifold
NN2: dynamics learning
f(x)
Trush Bacefine NV
100
=
F(x)
((x) ((0)) (H))/))))) dH(x)
[dH()))dzll
dx
trustworthy dynamical system learning
Target:
Approach: ConCerNet with two step learning
Result: automated conservation discovery robust dynamics prediction
Figure I: Pipeline to learn the dynamical system conservation and enforce simulation. con- trastive learning framework is proposed extract the invariants across trajectory observations, then the dynamical model is projected to the invariant manifold to guarantee the conservation property.
ifold learned by the first module, and therefore enforcing the trajectory conservation for the learned invariants.
We summarize main contributions follows:
We provide a novel contrastive learning perspective of dynamical system trajectory data to capture the invariants of dynamical systems. One byproduct of this method that the learned invariant functions discover physical conservation laws in certain cases. To the best the authors' knowledge, this is the first work that studies the discovery of conservation laws for general dynamical systems through contrastive learning.
We propose a projection layer impose any invariant function for dynamical system tra- jectory prediction, guaranteeing the conservation property during simulation.
Based on the above two components, we establish a generic learning framework for dy- namical system modeling named ConCerNet (CONtrastive ConsERved Network) which provides robustness in prediction outcomes and flexibility to be applied to wide range of dynamical systems that mandate conservation properties. We conducted extensive experi- ments to demonstrate the efficacy of ConCerNet, especially its remarkable improvement over a generic neural network in prediction error and conservation violation metrics.
We draw inferences on the relationship between the contrastively learned function, the ex- act conservation law, and the logistics of contrastive invariant learning. Further, potential improvements to the proposed method are illustrated that can improve the automated sci- entific discovery process.
BACKGROUND AND RELATED WORK
2.1 CONTRASTIVE LEARNING
Unlike discriminative models that explicitly learn the data mappings, contrastive learning aims to extract the data representation implicitly by comparing among examples. The early idea dates back the 1990s (romley et al. 1993) and has been widely adopted in many areas. One related field to
2

---

Under review as a conference paper at ICLR 2023
work is metric learning (Chopra et al. 2005 Harwood et al. 2017 Sohn 2016). where the goal is to learn a distance function or latent space to cluster similar examples and separate the dis-similar ones.
Contrastive learning has been a popular choice for self-supervised learning (SSL) tasks recently, as it demonstrated its performance in many applications such computer vision (Chen et al. 2020a
He et al. 2020 Ho & Nvasconcelos) 2020 Tian et al.)
(020) and natural language processing (Wu
et al. [020b] Gao et al.). There are many existing works related to contrastive learning, covering the
design of contrastive loss (Oord et al. 20 Sa Chen et al.
2020a b). memory bank (Wu et al.
2018
sample bias (Chuang et al.) 2020 Arora et al. 2019 and momentum encoder (Chen al.
2020c
He et al. 2020).
2.2 DEEP LEARNING BASED DYNAMICAL SYSTEM MODELING
Constructing dynamical system models from observed data is a long-standing research problem with numerous applications such as forecasting, inference and control. System identification (SYSID) (Ljung 1998, Keesman & Keesman| 2011) was introduced a few decades ago and designed to fit the system input-output behavior with choice of lightweight basis functions. In recent years, neu- ral networks became increasingly popular in dynamical system modeling due to its representation power. In this paper, consider the following neural network based learning task to model an (autonomous and continuous time) dynamical system:
fo(x) 1 dx(t) dt
where TR" is the system state and i is its time derivative. So : R. IR" denotes the neural network model f with parameter to approximate ground truth dynamics.
The vanilla neural networks learn the physics through data by minimizing the step prediction error, without a purposely designed feature to honor other metrics such as conservation laws. One path to address this issue
include an additional loss in the training (Singh et al. [021 Wu et al. 2020a Richards et al. 2018: Wang et al. 2020);; however, the soft Lagrangian treatment does not guarantee the model performance during testing. Imposing hard constraints upon the neural network structures is a more desirable approach, where the built-in design naturally respect certain property regardless of input data. Existing work includes: Kolter & Manek (2019) learns the dynamical system and Lyapunov function to ensure the exponential stability of predicted system; Hamiltonian neural network (HNN, Greydanus et al. (2019)) targets at the Hamiltonian mechanics, directly learns the Hamiltonian and uses the symplectic vector field to approximate the dynamics; Lagrangian neural network (LNN Crammer et al. (2020) extends the work of HNN to Lagrangian mechanies. Although the above models are able to capture certain conservation laws under specific problem formulations, they are not applicable to general conserved dynamical systems (c.g. mass conservation). This motivates work in this paper propose contrastive learning framework in more generic form that is compatible with arbitrary conservation.
2.3 LEARNING WITH CONSERVED PROPERTIES
Automated scientific discovery from data without prior knowledge has attracted great interest to both communities in physics and machine learning. Besides the above-mentioned HNN and LNN. few recent works (Zhang et al. 2018 Liu & Tegmark 2021; Ha & Jeong 202T Liu et 2022) have explored automated approaches to extract the invariants or conservation laws from data. De- spite the promising results, the existing methods usually suffer from limitations including poor data sample efficiency and reliance on artificial prepprocessing and hereby difficult to extend to larger and more general systems. Note that the aim of our proposed framework is to provide a genuinely adaptable and highly automated tool for trustworthy data-driven dynamical systems modeling, rather than simply solving the above limitations. The other line of physics invariant learning focuses on discovering the hidden symmetries in physical systems (Liu & Tegmark] 2022 Mototake| (021). To the best of our knowledge, this is the first time conservation law discovery for general dynamical systems is studied through the lens of contrastive learning.
3

---

Under review as a conference paper at ICLR 2023
3 PROPOSED METHODS
3.1 CONTRASTIVE LEARNING FOR CONSERVATION PROPERTY FROM SYSTEM TRAJECTORIES
In the practice of dynamical system learning, the dynamics data is usually observed a set of trajectories of system state {a} IR" N.T fim1,t=1> where denotes the trajectory index of total trajectory
number N and &. is the time step with total time step number T. Ho. R -> I is the neural network parameterization to map the original state to latent representation with dimension m. We let the trajectory set { be the simulation history starting from initial conditions drawn from distribution D. and assume the initial conditions have various conservation values. Inspired by the concept of Neighborhood Component Analysis (NCA, GGoldberger et al. (2004)) and Neighborhood analysis Contrastive loss (NaCI, Ko et al. (2222), y, we analogize each trajectory as neighbor class where the individual points are drawn from and try to solve a N class classification problem by comparing the latent representation of every pair of points. Since the invariants are well conserved along the trajectory and differ among different trajectories, we aim to find the conservation laws which naturally serve as the latent representation of each class. We consider the nearest neighbor selection as a random event, where the probability of point ot belonging to the trajectory C₁ is defined as p(Ck) (2t) in the following design:
E exp(- - Ho. (z)) - Ho. (3) #
p(C+la/)==
ΣØ=1 Ei exp(- Ho. (at) Ho, (
2) + j or 12)
By traversing all pairs of points, we formulate the contrastive loss function:
Σ exp(- Ho, (x) - Ho, (1))
Loon =E E
a.C,C=D, EtEG
log
(3)
Σn exp(-Il/25500e (I) - No. (x 12))
EC C
+ Σx exp(- HHo
Ho. )11)
N
T
1== exp(- 11,. (xt,,) - Ho. (11,))13) (2)
T
log
NT
E Σ)=1
Ei exp( HH,, (at,) ) Ho. ()))){)))( + or + t2)
T
4=
(4)
Similar to NCA, we choose squared Euclidean norm the distance metric between point pairs] and the Softmax like function, ensuring the nice property of probabilistic distribution. To notice, in com- mon contrastive learning setting like SimCLR (Chnn al. (020a) or NCA, the classification target group only contains one element. In our setup, we consider the classification problem as assigning one point to a group of points in the same class, therefore we have the additional summation loop on both denominator and numerator. This setup is similar to the NaCl loss in Ko et al. (2022) with many positive pairs.
3.2 ENFORCING CONSERVATION LAWS IN DYNAMICAL SYSTEM PREDICTION
After the contrastive learning of the conservation terms Ha. (xr) R" we attempt to enforce the predicted trajectory along the learned conservation manifold in the simulation stage, s.t. dHo (w) dt
0. In the continuous dynamical system like Equation (I). we can project the nominal neural network output fe. (I) onto the conservation manifold by eliminating its parallel component to the normal
direction of the invariant planes (i.e. V, Ha, (x)). We define the projected dynamical model fo, (x) as following:
Positive pair denotes correlated views of the same example in contrastive learning literature (Chen et al. 2020a), in this paper define positive pair two system states from the same trajectory therefore they are assumed with same conservation property and considered in the same class.
4

---

Under review as a conference paper at ICLR 2023
for (x) Projection V., = =0]
(5)
(o, Σ (Ja,()) (1)) (V (V.HH (x))
where ((V,H) (z))-} denotes the orthonormalized of vectors from (V,H,, by process. The summation symbol indicates projections to each of the conservation terms and vanishes if only conservation term is learned (i.e. m 1). The projected model dy-
namics naturally satisfies (z), V Ho, (x)) and therefore guarantees He, (x) being constant
along the simulated trajectory. The intuitive diagram of projection is shown in Figure
!
For dynamical system learning, the loss function is simply the mean square loss between the neural network prediction and the observed system time derivative.
Cayn BenCaD [1Ffa(()) =01²]
4 EXPERIMENTS
To demonstrate the power our method, we first illustrate the procedure through two simple conser- vation examples, then we show technical details to overcome a more complex problem. In the end, we highlight the method is extendable to high-dimensional problems by leveraging an autoencoder. All the system and experiment details are listed in Appendix A
x(1
Learned Conservation
x[2]
Learned Conservation
1.0
1.0
05
0.8
0.6
00
-1
0.00
0.25
1.00
x33
Conservation
xF2
True Conservation
10
1.35
1.0
1.75
05
150
0.8
1.5
00
100
1.00
-1.
0.0
0.00
1.00
Figure 2: Learned conservation function vs ground truth, left: ideal spring mass, right: chemical kinetics
4.1 SIMPLE CONSERVATION EXAMPLES
In this section, we introduce two simple examples: Ideal spring mass system under energy conser- vation (x + xx2)²) and Chemical reaction under mass conservation (x[1] + x(2)). Both systems have 2D state space for casier visualization of the learned conservation function. Figure 2 shows the learned conservation compared with ground truth. The contrastive learning process captures the quadratic and linear functions, as the contour lines are drawn in circles and affine functions. To notice, the learned conservation here is approximately the exact conservation differing by some constant coefficient, further relationship between contrastively learning invariants and actual conservation, we delay the discussion to Section 5.1 In Figure 3 we compare the two methods by showing the trajectory, servation and coordinate error to the ground truth. The vanilla neural network is likely to quickly diverge from the conserved trajectory, and the error grows faster than proposed method.
the conservation is relative quantify instead of an absolute value. For
5

---

Under review as a conference paper at ICLR 2023
X01
Trepecorr
Energy
Groud Iuth
0040
Gaseline Mit
04
Beadlinc
1.5
0,710
035
Groand Trll
800
Hona oe
06
6,000
10/00/000
035
951
114
314
Thepeciory
Maasi Comservation
bettomeen cocrcc
Druund
135
112
Mh
4012
380
154
Figure 3: Simulation comparisons of two simple examples; upper row: ideal spring mass system, lower row: chemical kinematics. 1st column: state trajectories, 2nd column: violation of conserva- tion laws to ground truth, 3rd column: mean square eтror to ground truth.
(chmmWVis))
(dlmmitta))
0.25
0.15
0.18
0.00
-0.04
-0.05
-0.00
0.00
0.30
-0.08
30
15
100
14
121
Energy
Conservation
MSE coordinates
Boseline
-0.55
Ous dimtrtaai
0.5
Our ctettita))
0.-
-0.5
-0.75
Ours diceMta
Eaze lin in
Oura d
0.-
Gound
d
-0.05
Ocn destto--
-10
-05
0.0
05
1.0
15
Figure 4: Kepler system results: First rows (contrastive learning) learned invariants for 6 sampled trajectories. Last row (dynamics simulation): state trajectories, violation of conservation laws to ground truth, mean square error to ground truth.
4.2 COMPLEX CONSERVATION FUNCTIONS
In this section, we tackle a more complex system with more than one conservation laws with com- plicated representations. The Kepler system describes planet orbiting around with ellip- tical trajectories. The planet has four dimensional states, including both coordinates in the 2D plane and the corresponding velocity. The system has two conservation terms (energy conservation =
and angular momentum conservation xr x[]]x[4] x[2]=[3]).
+=
The major challenge for Kepler system conservation learning is data requirement and the repre- sentation power of the simple neural network to capture the complex energy conservation function including the square root on the denominator. Besides, if we use the standard contrastive loss from Equation (3). we found it is possible that the learned conservation might converge to a trivial solution as the loss function might encourage more of "similarity" within trajectories than "discrepancy" be-

---

Under review as a conference paper at ICLR 2023
tween trajectories. To address this issue, propose to batch normalized latent function during training by replacing (x) with (x) with
(x,) Ho. (1) Mhatch
Obatch
where Pbatch ΣV Hg, (zi) and Obatch
Σ. (Hi) #batch)2 are the element-wise
mean and standard deviation of the mini-batch.
From Figure 4 we use different dimensions (dim= 2 and 1) for the latent space and plot the learned conservation along trajectories. We found that for both dimensions, the learned conservations are distinguishable but struggling to follow the flat line, where the periodic pattern from the orbit loops challenges the prediction power of the neural network to find the correct law. terms of the rela- tionship between learned and actual conservation, it is difficult to draw any conclusions other than the ordering of the trajectories. Despite not learning the exact conservation law, in the simulation stage, method still outperforms the vanilla neural network by large margin in both metrics with the learned invariant. Interestingly, the conservation laws are better enforced with dim (2,(())) comparing dim Hg, (z)) 1, indicating the system naturally follows conservation laws.
4.3 LARGER SYSTEM: HEAT EQUATION
To further extend our model to larger systems, we test our method on solving the Heat Equation on ID rod. The ID rod is given some initial temperature distribution and insulated boundary condition on both ends. The temperature Uy. f), as a function of coordinate and time, gradually evens up following the heat equation aú 3'y By The total internal energy along the rod does not vary because the heat flow is blocked by the boundary. We use system state consisting of overall 101 nodes to discretize y E [-5, 5| and compress system states to a 9 dimension latent space with an autoencoder pair (E De,). For both contrastive conservation learning and dynamical system learning, original space state and time derivative (I, () are mapped to the autoencoder latent space (z, 8) by
2 = Eeg (z)
1 E (x)
x +
(10)
dr
where denotes the matrix multiplication by chain rule, the partial derivative from latent space to original space can be calculated by auto-differee dittreiiferentiitionn package. After simulation, the latent space trajectory will be mapped back the original space by Dap
Wherswew Usyll
Mops Mnel
Temperature uBC
0.30
08
8.024
Vl00
0.566
Temoeraatue
8.800
Conevvttonn
22.0
Cansevation
22.0
Ground Thath Basekin KM
Ground Tur Daseline NN
Ours
0.
0.2
044
0.6
1.5
0.5
04
06
0.8
1.0
-6.1
Figure 5: Heat Equation Simulation. Left column: vanilla neural network coordinate error and conservation violation ground truth. Mid-column: our method. Right column: initial and final temperature distribution
Figure 5 shows the simulation result and conservation metric comparison between vanilla neural network and method. For both methods, the initial conservation violation error was introduced

---

Under review as a conference paper at ICLR 2023
by autoencoder. In general, our method conforms to ground truth trajectory and conservation laws much better than the vanilla method.
We generalize the quantitative results for all the experiments above in Table | Each of the experi- ments is performed over three random seeds. During testing, we integrate the trajectory with Runge- Kutta method and compare the trajectory state coordinate and conservation error to the ground truth. Our method outperforms the baseline neural network by a large margin, and the error is often multi- ple times smaller. One may notice that the standard deviation is comparable with the error metric in the experiments, this is due to the instability of the dynamical system. The trajectory tracking error will exponentially grow as function time, and the cases with outlier initialization are likely to dominate the averaged results and lead to large variance. We append the same results in log scale in Appendix A to help clarification. In practice, our method is capable to control the tracking deviation better than the baseline method across almost all the cases.
Table 1:
Simulation error over the tasks Mean square error
Violation of conservation laws
Task
Baseline NN
ConCerNet
Baseline NN
ConCerNet
Ideal spring mass system
0.209 = 0.172
0.076
0.063
0.096 0.080
0.002 = 0.002
Chemical kinematics
0.064 = 0.006
0.031 + 0.033
0.025 = 0.012
0.003 = 0.001
Kepler system
0.854 + 0.103
0.328 0.026
0.060 0.011
0.009
0.012
Heat equation
0.133 + 0.054
0.098 + 0.073
0.686 + 0.546
0.178 0.045
DISCUSSIONS
5.1 LEARNED INVARIANTS VS EXACT CONSERVATION LAWS
To notice, the learned invariants are different from exact physics conservation laws. From Figure 2 can tell the learned quantity is approximately linear function of the conserved quantity. We will give glimpse of the intuition of why linear function reaches the minimal loss with one-dimensional conservation assumption in Section [5.2] In fact, the linear coefficient is not always positive, therefore the sign of the conservation function is not guaranteed. This is the natural result for a perfectly conserved system, lacking of time evolution information for conservation quantities. In real world cases, many systems are dissipative, with directional time derivative on nominal conservation. We design ranking loss function to utilize the directional information and encourage the correct sign of the learn invariants, the result is delayed to Appendix B
Despite the similarity between learned invariants and conservation laws for the above cases, for general system with more than one conservation terms, the learned invariant can be non-linear function the conservation terms. Therefore, might not preserve the linear relationship. this paper, we foeus improving the conservation performance for dynamical modeling and leave the task to find the multiple conservation laws to future work. Regardless of the mapping between the learned function and the exact conservation function, the simulation is guaranteed preserve the conservation property if the mapping is bijection.
5.2 CONTRASTIVELY LEARNED LATENT SPACE FOR CONTINUOUS LABELS
As the original contrastive learning framework is designed for discrete labels, we would like to vestigate how the learned latent space looks like when a continuous conservation function implicitly labels the observations. Let a system with state x has distribution D) on a compact set X' C IR", there exists continuous scalar function g _ + E denoting the unknown conservation function mapping X' to another compact set Z = fz|2 = g(x), x E X¹). Consider Z as a continuous label space, where close examples in the Z space are considered noisy observations to each other and belong to the same class. Formally, (x, x²) E {positive pairs}, VVlg(=)))) g(x) < e. Then we can write the "continuous" version of the contrastive loss function Equation
Coontinuous =-/ pD(x))lo(( -(alll("'- g( PD PD(ao)erpp (epepp(((hooo ho ho(
dx (11)
fx PD (xx" eezp( (ho ho(z"))2))dd"
8

---

the result is consistent with our Under review as a conference paper at ICLR 2023
where he h) is the parameterized conservation function and PD() is the probability over the distribu- tion D. The numerator traverses the similarity on the neighborhood x' assumed the same class of x, the denominator calculates the same integral over the entire space. As attempt to analytically solve the optimization problem for Equation (11) but only come up with a trivial maximum solution, conduct numerical experiments and make the following conjecture:
Conjecture 1. Let D be uniform distribution, optimize Leomtimouus over bounded set of then (o( is an affine function of g(x:) (e.g. ho(x) = Cig(z) + C2 when L.continour achieves a minimum point.
We illustrate the conjecture through the following numerical example. We consider ID uniform distribution over the set X [-1, 1] and g(x) = x for simplicity. We parameterize ho(x) with quadratic function and show the numerical results of Leentmoous in Figure O The trivial maximum point is when ha(2) being constant (blue). The minimum point reaches when ha (2) is a linear function with the largest absolute coefficient (green). Interestingly, the red solution performs worse than green with the same starting and ending points, indicating a deviation from the linear transform will increase Conntinuous For more generality, we provide another numerical experiments with g(z) x² without bijection relationship between A' and Z in Appendix C conjecture.
20
10.00-100.cm0
==0b=0x=0
10
-5
1.5
-10
10
-1
-0.5
0.5
Figure 6: Right: (x))) = ax + bx + c on different parameterization. Left: Contrastive loss on 1D
continuous label space function of (a, b) E [-10,, 10j2, 0.05.
5.3 CONTRASTIVE LEARNING TRAINING LOGISTICS
During the training trajectory invariants with contrastive learning, we observe certain interesting phenomenon and would like to involve the discussion here for readers' reference. We use the linear regression error to the exact conservation function as metric and delay the experiment results under different hyper-yppaate saper-parrmeters Appendix We found the fitting eтror decays with O(N 1/2). where N is the training trajectory number. This echoes with the supervised deep learning generalization bound in literature (Yehudai training. we also use small sizes (10-50) large sizes slightly compromise the performance. Our intuitive explanation is that large batch is likely to involve similar but less distinguishable trajec- tories.
Shamir 2019, Cao Gul [219] 2020). practical contrastive
CONCLUSION
In this paper, we propose ConCerNet, a generic framework learn the dynamical system with de- signed features to preserve the invariant properties along the simulation trajectory. We firstly learn the conservation manifold in the state space with contrastive view over the trajectory observation, then purposely enforce the dynamical system to stay in the desired subspace by appending projec- tion layer after the nominal neural network. We show the advantage of our proposed method both simulation error and conservation metrics and extendibility to be incorporated into larger models. Despite the paper presents an end-to-end approach, both contrastive learning on system invariants and projected dynamical system learning be different direction. We believe these ideas represent a promising route in automated system property discovery and practical dynamical system modeling.
independent procedure and open up

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. 36th International Conference on Machine Learning, ICML 2019, pp. 9904-9923, International Machine Learning Society (IMLS), 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv2200..07556 2021.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances neural information processing systems, 32, 2019.
Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over- parameterized deep relu networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 3349-3356, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information pro- cessing systems, 33:22243-2223 22255, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint Stti:20003.04299 2020c.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattem Recognition (CVPR'OS 05), volume 1, pp. 539-546. IEEE, 2005.
Ching- Yao Cao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775. 2020.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003 04630, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 2018.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings.
Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. Advances in neural information processing systems, 17, 2004.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
10

---

Under review as a conference paper at ICLR 2023
Michael Gutmann and Aapo Hyvärinen. for unnormalized statistical models. In Proceedings of the thirteenth international conference on artifficia intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
estimation: A new estimation principle
Seungwoong Ha and Hawoong Jeong, Discovering conservation laws from trajectories via machine learning. arXiv preprint arXiv:2102.04008, 2021.
Ben Harwood, Vijay Kumar BG, Gustavo Carneiro, lan Reid, and Tom Drummond. Smart mining for deep metric learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2821-2829, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17081-17093, Curran Asso- neurips httpcccccpppere/2///eeee pdf
ciates, Inc., 2020. URL
(c68c9c8258ea (68088255e7655772200000111000-20
Karel J Keesman and Karel J Keesman. System identification: introduction, volume 2. Springer, 2011.
Ching-Yun Yun Ko, Jeet Mohapatra, Sijia Liu, Pin-Yu Chen, Luca Daniel, and Lily Weng. Revisiting contrastive learning through the lens of neighborhood component analysis: an integrated frame- work. In International Conference on Machine Learning, pp. 11387-11412. PMLR, 2022.
J Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. Advances in neural information processing systems, 32, 2019.
Kookjin Lee and Kevin T Carlberg. Deep conservation: a latent-dynamics model for exact sat- isfaction of physical conservation laws. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 277-285, 2021.
Ziming Liu and Max Tegmark. Machine learning conservation laws from trajectories. Physical Review Letters, 126(18):180644 2021.
Ziming Liu and Max Tegmark. Machine learning hidden symmetries. Physical Review Letters, 128 118):180201, 2022.
Ziming Liu, Varun Madhavan, and Max Tegmark. Ai poincaré 2.0: Machine learning conservation laws from differential equations. arXiv preprint arXiv:2203.12610 2022.
Lennart Ljung. System identification. In Signal analysis and prediction, pp. 163-173. Springer, 1998.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013.
Yoh-ichi Mototake. Interpretable conservation law estimation by deriving the symmetries of dynam- ics from trained deep neural networks. Physical Review E, 103(3):033303, 2021.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018a.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018b.

---

Under review as a conference paper at ICLR 2023
Spencer M Richards, Felix Berkenkamp, and Andreas Krause. The Iyapunov neural network: Adap- tive stability certification for safe learning of dynamical systems. Conference on Robot Learn- ing. pp. 466-47666 476.. PMLR, 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05566. 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentee neural networks. In International conference machine learn- ing, pp. 1842-1850. PMLR, 2016.
Sumeet Singh, Spencer M Richards, Vikas Sindhwani, Jean-Jacques E Slotine, and Marco Pavone. Learning stabilizable nonlinear dynamics with contraction- based regularization. The Interna- tional Journal of Robotics Research, 40(10-11): 123-1150, 2021.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Naoya Takeishi and Yoshinobu Kawahara. Learning dynamics models with stable invariant sets. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9782-9790 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1457-1466, 2020.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40, 2016.
Jin-Long Wu, Karthik Kashinath, Adrian Albert, Dragos Chirila, Heng Xiao, et al. Enforcing sta- tistical constraints in generative adversarial networks for modeling chaotic dynamical systems. Journal of Computational Physics, 406: 409:10 2020a.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision 2018.
and pattern recognition, pp.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 2020b.
Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. Advances Neurul Information Processing Systems, 32, 2019.
Pengfei Zhang, Huitao Shen, and Hui Zhai. Machine learning topological invariants with neural networks. Physical review letters, 12066):06440,, 2018.
12


Reviews:
[{"review_id": "eneicaBgfW", "reviewer": "Reviewer_vHDM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a contrastive learning framework to model conservative dynamical systems with hidden conservative laws. More specifically, the proposed framework consists of two networks; an invariant representation network that maps elements of a certain trajectory to an identical latent features $H_{\\theta_c}(x) \\in \\mathbb{R}^m$ which mimic $m$ invariant quantities, and a dynamics network  $\\hat{f_{\\theta_d}}(x) = \\text{Projection}({f_{\\theta_d}}(x), f: \\langle f, \\nabla_x H_{\\theta}(x) = 0 \\rangle)$ that models the targeted conservative dynamics by projecting a neural ODE vector field ${f_{\\theta_d}}(x)$ to the orthogonal direction of  $\\nabla_x H_{\\theta}(x)$. The former and latter are trained via a neightborhood conservative loss and a standard MSE loss, respectively. The authors validate their proposed method for two simple dynamical systems (two variables with one conservative law), Kepler system (four variables with two conservative laws), and heat equation on a 1D rod (101 mesh points and one conservative law).", "summary_of_the_review": "Although the paper is interesting, I think there is still much room for improvement, e.g., see Weakness (and Questions). My current evaluation on this paper is borderline reject.", "strength_and_weaknesses": "Strength:\n1. This paper tackles an important and challenging problem of physics (can machine learning models learn conservative dynamics and extract invariant quantities from observed trajectories without using prior knoweldge (e.g., conservation of mechanical energy) or assuming underlying structures (e.g., Hamiltonian)?).\n\n2. To solve the above important problem, the authors introduce a constrative learning framework that can learn invariant quantities via a representation learning way. To me, this idea is novel.\n\n3. The paper is generally easy-to-follow.\n\nWeakness ( and Questions):\n1. I am not sure whether the contrastively-learned invariant latent features, i.e., the modeled invariant quantities, can be valid for unseen initial conditions. I think the authors should report generalization performance of the proposed method to ensure it.\n\n2. $m$, the latent feature dimension, seems to be very important hyper-parameter because it directly determines the number of invariant quantities. The authors do not explicitly state how one can tune such a important hyper-parameter when one does not know the exact value of $m$ for an unknown physical system, thus I am not sure the robustness of the proposed method in this paper.\n\n3. Espeically, for a heat equation problem, the authors use an auto-encoder that maps an observation to $d$-dimensional laten space. It means that the modeled dynamics is $d - m$ dimensional sub-manifold and it seems that both $d$ and $m$ seem to be very sensitive hyper-parameters when learning the dynamics and invariant quantities accurately.\n\n4. Experimental results seem to be weak. The proposed method fails to accurately learn the 4-variable Kepler system. Also, the authors only compare their model with a simple vanila network. It would be nice if the authors compared their model with some advanced models, e.g., HNN or [1].\n\n5. Can the proposed model learn a non-conservative systems such as a dissipative one?\n\n6. The core idea of this paper is that a certain state should have an invariant representation with respect to its time evolution. While the contrastive learning is one of good candidates to finding such a representation, there are various other methods that can enforce the invariance, e.g., consistency regularization [2]. What motivated the authors to use the contrastive learning?\n\n***\n[1] Liu, Z., & Tegmark, M. (2021). Machine learning conservation laws from trajectories. Physical Review Letters, 126(18), 180604.\n\n[2] Sinha, S., & Dieng, A. B. (2021). Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34, 12943-12954.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VibZD658aMi", "reviewer": "Reviewer_kRTH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a method to impose an acquired conservation law on a learned time-evolving model such as a DNN.\nThe proposed method consists of a two-step approach. The first step is invariant learning, in which underlying conservation laws are estimated from trajectory data, and the second step is dynamic learning, in which the time evolution model is subjected to preservation of conservation laws. Numerical experiments show that the proposed method consistently outperforms the baseline method in both coordinate error and conservation measures and can be further extended to complex, large-scale dynamics by leveraging autoencoders. The proposed method may also be useful in discovering conservation laws in unknown dynamical systems.", "summary_of_the_review": "The approach of estimating hidden conservation laws from data and improving the accuracy of simulations based on these conservation laws is an innovative research, and the novelty of this research fully satisfies the conditions for acceptance for presentation at the conference.\nOn the other hand, the evaluation of the effectiveness of the proposed method is weak, and the validity of the study is questionable.\nI believe that additional discussion and numerical experiments to improve these points would be a condition for the conference to accept the study.", "strength_and_weaknesses": "Strengths\nThe framework of estimating hidden conservation laws from data and then improving simulation accuracy based on the estimated conservation laws is highly novel.\n\n\nWeaknesses\n\nA weakness of the research is that the effectiveness of the proposed method has not been verified because there is no theoretical analysis to guarantee the effectiveness of the method and the numerical experiments are limited to simple systems for which the conservation laws are known.\nConcrete weaknesses are that the following two points have not been validated.\n\n\n1. the effectiveness of the proposed method for learning the energy function\n\nThe denominator of the \"invariant learning\" loss function (Eq. (3)) seems to be based on the good properties of the energy terrain.\nFor example, it is questionable whether it would work effectively in the case of a complex energy function with multiple peaks, such that starting from different initial conditions would result in the same energy value.\nIn addition, the behavior of the model when the number of data is small relative to the degrees of freedom the system has is unclear.\nNumerical experiments with more complex energy landscapes and a discussion of the relationship between the number of data and learning performance would be needed. \n\n\n2. effectiveness of the proposed method as a method for estimating unknown conservation laws\nThe authors mention the application of the proposed method to the estimation of unknown conservation laws, but its effectiveness is questionable in terms of interpretability and feasibility of estimating complex conservation laws.\n\n\n2-1. Interpretability\nAll of the previous studies cited in section \"2.3 LEARNING WITH CONSERVED PROPERTIES\" seem to achieve interpretable conservation law estimation.\nOn the other hand, the proposed method does not seem to be able to achieve interpretable conservation law estimation.\nMethods for estimating interpretable conservation laws from DNNs trained on dynamical system data have been proposed [1,2].\nI think that an additional discussion based on such previous studies seems necessary.\n\n\n2-2. Possibility of Estimating Complex Conservation Laws\nThe authors claim that the proposed method can estimate complex conservation laws.\nAs an example, they demonstrate the estimation of an angular momentum conservation law for the Kepler system.\nHowever, this task is not so difficult.\nConservation law estimation for the Kepler system has been realized in many studies [1,2].\nIn order to claim the effectiveness of the proposed method as a conservation law estimation method, it is necessary to at least realize conservation law estimation corresponding to symmetries for nonlinear transformations such as Runge-Lenz vectors.\nNon-linear conservation estimation is also mentioned in [1][2]. \nAlso, [3] achieves it using Hamiltonian Neural Networks by estimating the mass tensor.\n\n[1]Ziming Liu and Max Tegmark, \"Machine Learning Hidden Symmetries,\" PRL, 128, 180201, 2022.\n\n[2]Yoh-ichi Mototake, \"Interpretable conservation law estimation by deriving the symmetries of dynamics from trained deep neural networks,\" PRE, 103, 033303, 2021.\n\n[3]Nate Gruver, Marc Finzi, Samuel Stanton, Andrew Gordon Wilson, \"Deconstructing the indirect biases of hamiltonian neural networks,\" ICLR 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JsOK7oDTrgs", "reviewer": "Reviewer_m8A3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a contrastive learning approach to learn conserved quantities of a dynamical system using an auxiliary neural network. The neural network learns through positive samples from the same trajectory of the dynamical system and negative samples from different trajectories. The conservation is enforced during prediction by projecting the outputs from a forecasting neural network onto the conservation manifold. ", "summary_of_the_review": "My main concerns are regarding the utility of learning conservation laws this way as these NN representations are not interpretable and may not be easy to enforce onto the forecasting network. Also, paper does not adequately compare with any other baselines that learn conservation laws from trajectories.", "strength_and_weaknesses": "Learning conservation laws from observed trajectories is an important task to study a dynamical system and can also help in generalization. To my knowledge, the contrastive learning approach is novel and interesting. \n\n\n\n**Weaknesses**\n\n1. The paper does not adequately showcase the utility of learning conserved quantities this way. Since conserved quantities here are learnt as a NN representation, they are not interpretable. As mentioned by the authors, the learnt quantities may be a non-linear function of the (true) relevant conserved quantities. Additionally, as seen in Figure 4, the approach is not always able to enforce the learnt conservation law properly to the forecasting network.\n2. Paper does not empirically compare with any other baselines that learn conservation laws. Only comparison provided is a standard neural network which does not learn the conservation law, as expected.\n3. Writing in Section 3 is rushed (possibly with mistakes) and can be improved with more details about the equations. Section 4 (and Appendix) does not provide adequate details about the experiments & training/testing methodology, and a few plots are hard to parse due to no labels. \n\n\n\n**Questions**\n\n1. Except the ideal spring mass experiment, every other experiment considers energy conservation where Hamiltonian/Lagrangian networks can be used, potentially with additional rotational symmetry [1, 2] for angular momentum conservation. I think these methods should be added as baselines in these experiments. Further, the paper would be much stronger if there were more experiments like spring mass experiment that conserve quantities other than energy. \n2. The experiment section should include comparison with other baselines that learn conservation laws from trajectories (e.g., [3, 4] cited in the paper). \n3. In Equation (2), $t_1$ is not defined, should this be $t$? Can $k = i$ in this equation? \n4. In Equation (3), the expectation is defined over $x^+$ and then the summation inside is also over $x^+$. There is the same issue for $x^-$. In Equation (4), there seem to be a lot more negative samples compared to positive samples, does this not introduce some bias in learning?\n5. This framework does not seem to allow for the case when two different trajectories have the same conservation value. For example, trajectories with same energy but phase-shifted. Is this true?\n6. Please provide details of the experiments (in Section 4 or Appendix), for example, what are the time steps the trajectories were observed during training and the same during testing. Specifically, are the methods asked to predict beyond the times seen during training?\n7. Figure 3: Please provide y-labels for all plots. In Mass/Energy conservation plots, are the true values of mass/energy learnt by the neural network H?\n8. Figure 4: Please provide x-labels for plots of top row; I am assuming it is time(s) but then there is discrepancy in x-axis between top and bottom rows. \n9. In Section 4.2, could you please elaborate on what type of trivial solution does the model learn. It is also not clear to me why Equation (3) will be biased toward “similarity” within trajectories, and why the proposed solution solves the issue. \n\n\n**References**\n\n[1] Finzi, Marc, et al. \"Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Finzi, Marc, Max Welling, and Andrew Gordon Wilson. \"A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups.\" International Conference on Machine Learning. PMLR, 2021.\n\n[3] Ha, Seungwoong, and Hawoong Jeong. \"Discovering conservation laws from trajectories via machine learning.\" arXiv preprint arXiv:2102.04008 (2021).\n\n[4] Liu, Ziming, and Max Tegmark. \"Machine learning conservation laws from trajectories.\" Physical Review Letters 126.18 (2021): 180604.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "94t93Gs9e3m", "reviewer": "Reviewer_qJvt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a technique to preserve physical conservation laws in neural network based dynamical systems. The authors propose to solve two tasks concurrently: (i) learning variables that remain constant for a given trajectory, and (ii) a dynamics function that respects the constant variables and that fits input trajectories. The first objective is achieved by adopting well-known contrastive learning objectives so that the new objective \"pulls the latent factors\" (constant) within a trajectory whereas it \"pushes them away\" if they belong to different sequences. The second goal is realized by a gradient-matching loss. The model is experimentally demonstrated to preserve constant terms oftentimes and hence leads to non-divergent trajectories.", "summary_of_the_review": "I believe encoding state-invariant variables is a nice idea that comes with immediate real-world applications. A contrastive objective to learn such variables is shown to be effective. However, we are unable to tell whether the benchmarks are sufficiently challenging (so that we can see the limits of the framework). We are also not in a position to decide whether simpler ideas or existing methods would perform any better. Moreover, the text requires significant updates, which is why I recommend a reject. ", "strength_and_weaknesses": "Strengths:\n- The model surely learns _some_ constants that remain the same throughout integration.\n- The framework is straightforward: Similar contrastive objectives are commonly used and well understood. The dynamics function is also easy to understand as it is built to preserve the constant variables.\n\nWeaknesses:\n- The paper suffers from clarity issues as well as grammatical mistakes. I believe it should undergo significant updates. See my comments at the end of this question.\n- Methodology should be motivated. The authors argue that \"this is the first work that studies discovery of conservation laws for general dynamical systems through contrastive learning.\" What if we had a simpler method in which certain global parameters remain constant throughout integration? Would the proposed approach still be preferable? If so, why? Furthermore, among all other techniques, why is contrastive learning chosen? It does come with a complicated and costly loss term; so what is the catch here instead of a simpler approach?\n - The gradient matching term (eq. 7) should be carefully analyzed. It is well known that this objective becomes problematic with increasing noise levels and bigger time differences as the differential function is directly fitted to the empirical gradients $\\frac{x_{i+1}-x_{i}}{t_{i+1}-t_{i}}$. This is in contrast to learning a generative process, such as GP or neural ODEs, and is the very motivation of these approaches. \n- The paper lacks a whole discussion on causality. In my understanding, the method discovers certain factors on which the trajectory (forward simulation) is conditioned. This is the very definition of causal learning while the paper does not touch upon this connection. Similarly, the connections between invariance and causality could be included (in addition to the short discussion on invariance that is already included).\n- The method should be compared against SOTA baselines. I do acknowledge that the literature on \"black-box conservation learning\" is quite limited but at least Lagrangian NNs or some adaptation of vanilla neural ODE systems could be considered here. This would help us judge the level of difficulty of the problem. Similarly, we should see how the performance would change if we consider higher dimensional latent spaces ($m>2$) in Sec 4.2. \n\n\nWriting issues:\n - \"deploying the knowledge towards the new occurrence\" is unclear.\n - \"conservation law were\" is wrong\n - The connection between the sentence starting as \"On the other hand, data-driven\" and the prior sentences is missing.\n - \"one or more distinguishing features of physics-based system trajectories\" This sounds mysterious, i.e., what are the examples of such features?\n - \"By comparing the latent space distance of the system state observation and assigning them into their trajectory\" Both parts of the sentence is unclear. What is latent space distance? How does \"assignment\" take place?\n - What does \"ConCerNet\" stand for?\n - \"of the simulation in the longer term\" Longer compared to what?\n - Naming the function approximations already on the first page would help follow the text.\n - \"projecting the dynamical neural network output towards the\" Perhaps \"on the\"?\n - \"conservation manifold from the first one\" is unclear. Perhaps something like \"the manifold learned by the first module\"? \n - \"guaranteeing\" instead of \"guarantying\"\n - \"Unlike discriminative models that explicitly learn the data mappings\" What mapping is referred to here?\n - Sec2.1 could be organized chronologically (by moving the paragraph starting with \"Despite the recent surge of\" to the beginning)\n - Neural ODE paper (Chen, Ricky TQ, et al. \"Neural ordinary differential equations.\" Advances in neural information processing systems 31 (2018).) should be cited in Sec 2.2.\n - \"however, the soft Lagrangian treatment does not guarantee the model performance during testing\" This should be elaborated.\n - \"to propose a contrastive learning framework in a more generic form that is compatible with arbitrary conservation.\" What is meant by \"arbitrary conservation\"?\n - \"total time step T\" is not clear.\n - \"We let the trajectory set ... be the simulation history starting\". I'm not sure what this means: Do $C_k$'s refer to sequence ids?\n - \"the initial conditions have various conservation values.\" What does this exactly mean?\n - \"where the probability of point ... **belonging** to the trajectory\"\n - Similar to a previously raised concern: What is the motivation for eq.2? What does it imply?\n - \\log and \\exp should be used instead of log and exp.\n - What does $x\\sim C \\sim D$ mean?\n - A short discussion on NCA would be nice as it is referred to multiple times.\n - \"Softmax like function, ensuring the nice property of probabilistic distribution\" What does this mean? What is nice here?\n - \"learning of the m conservation terms\" Perhaps an m-dim vector is referred to?\n - \"by eliminating its parallel component\" Parallel to what?\n - Eq,1 uses $f_\\theta(x)$ whereas it later becomes $\\tilde{f}_{\\theta_d}(x)$. What is the difference?\n - Eq. 5-6 are unclear. A (verbal), more intuitive explanation should be given. Projection(.) function should be defined. \n - \"the orthonormalized set of vectors from\" is unclear.\n - \"conservation term in learned\" \n - x and y axes names and figure titles should be added/revisited.\n - What are the system states in Sec 4.1?\n - Why to use $x[1]$ instead of $x_1$?\n - \"the learned conservation here is approximately the exact conservation\" is unclear.\n - \"coordinate mean square loss to the ground truth\" is unclear.\n - Are the equations in the first paragraph of 4.2 really difficult to learn, given the recent success of neural nets on many tasks?\n - Why does the issue that is resolved by eq 8 take place in the first place? Why only on this experiment? Why does eq.8 help at all?\n - Better to use $m=2$ instead of $dim(H_{...})=2$.\n - Table1 could be given at the very beginning of Sec4 as it nicely summarizes all results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject