Towards Cross Domain Generalization of Hamiltonian Representation via Meta Learning
Authors: Yeongwoo Song
Hawoong Jeong
Venue: ICLR 2024 poster (2024)
Keywords: hamiltonian dynamics
cross domain generalization
learning physics
meta learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We explore cross domain generalization across dynamical system of diverse functional form of Hamiltonian.

Abstract: Recent advances in deep learning for physics have focused on discovering shared representations of target systems by incorporating physics priors or inductive biases into neural networks. While effective, these methods are limited to the system domain, where the type of system remains consistent and thus cannot ensure the adaptation to new, or unseen physical systems governed by different laws. For instance, a neural network trained on a mass-spring system cannot guarantee accurate predictions for the behavior of a two-body system or any other system with different physical laws.
In this work, we take a significant leap forward by targeting cross domain generalization within the field of Hamiltonian dynamics. 
We model our system with a graph neural network (GNN) and employ a meta learning algorithm to enable the model to gain experience over a distribution of systems and make it adapt to new physics. Our approach aims to learn a unified Hamiltonian representation that is generalizable across multiple system domains, thereby overcoming the limitations of system-specific models. 
We demonstrate that the meta-trained model captures the generalized Hamiltonian representation that is consistent across different physical domains.
Overall, through the use of meta learning, we offer a framework that achieves cross domain generalization, providing a step towards a unified model for understanding a wide array of dynamical systems via deep learning.

Published as a conference paper at ICLR 2024
TOWARDS CROSS DOMAIN GENERALIZATION OF HAMILTONIAN REPRESENTATION VIA META LEARNING
Yeongwoo Song' Hawoong Jeong! 2+
DDeparmmnt of Physics, KAIST CCntter for Complex Systems, KAIST
ywsong1 025knngst kr. Ajeong@kaist. edu
ABSTRACT
Recent advances deep learning for physics have focused on discovering shared representations of target systems by incorporating physics priors or inductive biases into neural networks. While effective, these methods are limited to the system domain, where the type of system remains consistent and thus cannot ensure the adaptation to new, or unseen physical systems governed by different laws. For instance, neural network trained on mass-spring system cannot guarantee accurate predictions for the behavior of a two-body system or any other system with different physical laws. In this work, we take a significant leap forward by targeting cross domain generalization within the field of Hamiltonian dynamics. We model our system with graph neural network (GNN) and employ meta learning algorithm to enable the model to gain experience over distribution of systems and make it adapt to new physics. Our approach aims to learn a unified Hamiltonian representation that is generalizable across multiple system domains, thereby overcoming the limitations of system-specife models. We demonstrate that the meta-trained model captures the generalized Hamiltonian representation that is consistent across different physical domains. Overall, through the use of meta learning, offer framework that achieves cross domain generalization, providing a step towards a unified model for understanding a wide array of dynamical systems via deep learning.
INTRODUCTION
Deep learning has succeeded in many application areas such as image classification, image generation. natural language processing, and so on (Reed et al. 2016 Tan & Le 2019 Raffel al. [020 Gu
et al. 2022). One of the major roles of such accomplishment was capable of parameterizing useful representations from data with neural networks (Chen al. 2020 Van Den Oord al. 2017
Hamilton et a (017). However, grafting deep learning to physics is yet another problem. They struggle to learn conservation laws or implicit physical geometries or symmetries. Although various studies make the model learn the conservative quantities or symmetries inside the system (Greydanus) al.. 2019: Liu Tegmark (02T), these approaches are system- specific. which means that if model is trained for system, it cannot easily adapt another system with different physical laws. Since systems whose physics is unknown have more sparse data, these flaws make the standard supervised learning methods harder to learn the physics of the system.
jet al. 2019. Sanchez-Gonzaler
One of the current extents of this limitation is crafting the generalization of fixed type of system across various parameter environments (Lee et al.) 202 YYn et al.) 2021 Kirchmeyer al. 2022 Liet al. 2023). Here, one can think of an extension of building a unified model that is not limited adapting to various parameter environments within single system type, but instead generalizes across the vast landscape of different system types. In this paper. we focus our objectives on systems that Hamiltonian mechanies can formulate. Systems governed by Hamiltonian mechanics inhere symmetries, and their state trajectories are laid on the symplectic manifold (Libermann & Marle 2012, Arnol d. 2013). Moreover, almost all of the physics our nature has its own conservation laws, and Hamilonnian mechanics relates the state of the system to its corresponding conservative quantities (usually energy).
"Correspondence Hawoong Jeong.

---

Published as a conference paper at ICLR 2024
In this work, we search for the generalized representation that shares the essence of Hamiltonian mechanics that a neural network can learn. doing so, we aim to make the model easily adaptable to unseen physics.
Our contributions are summarized as follows.
Utilizing the meta learning algorithm, we extend its application the data distribution across the Hamiltonian manifold, encompassing various types of physical systems. This enables our model to generalize effectively to system data with unknown physies, enhancing its versatility.
We begin by evaluating the model's performance on simpler examples, such as the mass- spring and pendulum systems. Subsequently, we expand our evaluation more complex scenarios characterized by chaotic behavior, such as the Hénon-Heiles and magnetic-mirror systems. This demonstrates the robustness and applicability of our approach across wide range of dynamical systems.
To gain insights into the superior performance of the meta meta-traines trainer model compared to other baselines, conduct in-depth analysis of the learned representations within the neural network. This investigation provides a deeper understanding of the underlying factors contributing to the model's effectiveness due meta learning.
We use the term domain to refer to different types of physical systems governed by distinct laws. To the best of our knowledge, we are the first to explore cross-domain generalization across diverse dynamical system types enabling easy adaptation physics, surpassing prior studies limited to fixed system types.
PRELIMINARIES
2.1 HAMILTONIAN MECHANICS
Hamiltonian mechanics is of the concrete frameworks describe the state of dynamical
system. It begins by defining a vector composed of canonical coordinates = (q. Here and denotes the generalized coordinate and its canonical momentum respectively; q - (g), .2. ..... gN), (P1. P2.... )N), where N is the degree of freedom of the given system. Then, Hamiltonian (H) defined such that Hamilton's 's equation (Equation TT is held. Usually, the Hamiltonian of the system corresponds to its energy where it is to be conserved.
dq
0H dt 8p" dt dq
0H
dp
(I)
I,
Here, the Hamiltonian vector field X
87 HH
JV Ho. where J -1
is
0p
0
known as symplectic, which allows the Hamiltonian to on a symplectic manifold. Symplectic manifold is one of the essences of differential geometry, which naturally arises from the formulations of Hamiltonian mechanics (Libermann & Marle 2012 Arnol'd 2013). provides the generalization of the phase space of a closed system, allowing
probe the time evolution of the system.
2.2 LEARNING PHYSICAL DYNAMICS FROM NEURAL NETWORKS
Various works use neural networks to analyze the time evolution, or the conservation law of the physical system (Greydanus et 2019 Gonzalez et al. 2019 Cranmer al., 2020 Kasim
& Lim 2022 Greydanus et al. 2023). For example, Hamiltonian Neural Network (HNN) (Greydanus) ct al., 2019) parameterizes the Hamiltonian of the system with a neural network. HNN learns the dynamics of the given system by inducing physical bias to the loss function. Making use of Equation] T HNN predicts the dynamics the system incorporating the symplectic gradient inside the loss function as in Equation
07
0q
07 0q
dp
CHNN =
+
+
(2)
Öp
Øt
Øt
2
2

---

Published as a conference paper at ICLR 2024
As an alternative way encode the physical bias into the objective function, the Variational Integrator Network (VIN) (Saemundsson et al. 2020) has been introduced, which preserves the geometric structure of physical systems. The architecture of VIN is designed to fit the discrete-time equations of motion of the given dynamical system. Thus, it provides interpretability and more efficient learning by preserving the manifold geometry inherent in physical systems directly in the model architecture.
While the theoretical understanding of inductive biases that underlie the advantages of neural networks remains elusive, recent studies attempt to shed light on the influence of such specific biases. For example, one demonstrates that contrary to conventional wisdom, the performance improvement in HNNS arises from the second-order structure, rather than the symplectic bias (Gruver et al., (022). Nevertheless, a number of works focus on leveraging the inherent nature of physical systems, such as their symplectic structure and the principle of least action, to enhance the predictive capabilities of neural networks for system dynamics (Chen et al. 2021 Greydanus et al. 2023).
In opposition implying relevant biases in neural networks, recent studies suggest a data-driven ap- proach to learning the dynamics of one type of system generalized across different environments (Yin) et al. 2021 Kirchmeyer et al. 2022). These focus on dynamical systems that are governed by the form dm,(t) = fe (t), where xx)) is a time-dependent state in the phase space X, and fa : - TX maps (t) to its time-derivative state dz,(t)
of the following differential equation; dt
dt which lies the tangent space TX. Here, the subscript e denotes an environment for given dynamics (e.g. different a, 3, 7, parameter configuration of Lotka- KolterVoleera system; du/ dt = au Buv, du = duv - yv).
2.3 META LEARNING FOR GENERALIZING DYNAMICAL SYSTEM
Meta learning, also referred to as "learning to learn," " focuses on training a model that exhibits strong generalization across diverse data distributions, enabling it to effectively adapt even with a small amount of previously unseen samples. As a result, meta learning can serve as a promising data-driven approach for generalizing the dynamics of physical systems. Among various methods are particularly versatile and compatible with differentiable models (Hospedales et al. 2020). For example, CoDA (Kirchmeyer al. 2022)
tasks
approaches of meta learning, optimization-aaseeee
is a representative optimization- based meta learning method for multi-environmeee formul lation of
the generalization problem for dynamical systems. Particularly, previous studies (Lee et al.) 202T
et al. 2023) have demonstrated the ability of a broadly applicable aptimization-baser meta-learning algorithm, Model-Agnostie Meta-Learning (MAML) (Finn al. (017]. in uncovering the physical laws of given system across different environments. There also exists Mon-MAML based methods for generalizing dynamics. Sequential Neural Processes (Singh et al. 2019) aims to build a meta-
transfer learning framework for stochastic processes, Dy (Wang et al. (022)) is a model-based meta learning method that learns the shared dynamics of given domain, and meta-SLVM (Jiang et al. 2022) utilizes Bayesian meta learning for learning to adapt latent dynamic functions.
The approaches mentioned above are all limited to scenarios where the systems share the same functional form of Hamiltonians, such set of pendulum systems with varying physical parameters (e.g. mass and length). Expanding upon these limitations, we aim to investigate whether a shared neural representation can be discovered among diverse functional forms of Hamiltonians. In this study, we explore the portrayal of a symplectic manifold governed by Hamilton's 's equation within neural networks by incorporating MAML with GNNs, Further details regarding the MAML algorithm can be found in Section 3,2 and 3.3
3 METHODS
3.1 PREPARING THE DATASET FOR META LEARNING
We constructed a task distribution comprising six distinct types of physical systems: mass-spring. pendulum, Hénon-Heiles, magnetic-mirror, two-body, and three-body. Given that these systems feature separable Hamiltonians, which inherently limit the diversity in the kinetic energy terms, we intentionally selected systems with varying complexities in their potential energy terms. Starting from the simplest mass-spring system, extend the type of system to more difficult systems such as Hénon- Heille or magnetic-mirror system. The details for each system can be found in the Supplementary Materials (SM)A.) A.1
3

---

Published as a conference paper at ICLR 2024
This variety of systems encompasses a range of complexities, featuring simple physical systems like the mass-spring and the pendulum, as well as more intricate systems exhibiting chaotic behavior such as the Hénon-Heiles and the magnetic-mirror. Additionally, the task distribution extends to systems involving more than two particles, specifically the two-body, and the three-body, body, thereby providing a comprehensive set of challenges. For all of our systems, we generated dataset with N 10, 000 trajectories confined to the wwo-dimensional space, where the canonical coordinates a = (q. p) as the input, and their corresponding time derivatives provide the detailed procedure for generating the dataset in the SM A.2
(á. p) as the output of a neural network. We
In the language of meta learning, several different types of systems will be used for meta training. while one of the remaining unselected systems will be used for meta testing (i.e. evaluation). We will denote each trajectory as Ts.; which sampled from the task distribution ((Ts) (where S E (mass-spring, pendulum, Hénon-Heiles, magnetic-mirror, two-body, saree-body). and i E [1..N]]. Then tasks from Tge will compose the task distribution for meta training, and tasks from T3 will form the data distribution for meta testing. For example, we will use the data distribution Tga: ge (mass-spring, pendulum, Hénon-Heiles) for meta training, Tg; S {magnetie-mirror for evaluating the magnetic-mirror system. The exact dataset configuration for meta training scenario is described in SM A.3
3.2 META TRAINING THE NEURAL NETWORK
We use simple graph convolutional network (GCN) (Kipf Welling/ 2016) to parametrize the given physical system in which we implement it using PyTorch (Paszke et al. 2019). Using GNNs to describe system several particles mediates the model to handle various degree-of-freedomm inputs such that the model can be fed with data from a diverse type of system, unlike from the previous works listed in Section 2.2 and 2.3 that dealt with fixed system scenarios. In line with the previous methods (Sanchez- Gonzalez al. 2019 Bishnoi et al. 2023) that represented the physical systems as graphs, we represent the state variable (q.P) as the node feature, while currently, edge features are not utilized. Our model is composed of three GCN layers to extract features of the input states, preceded by three fully connected linear layers for the regression of Hamiltonian value. We choose the mish function as the non-linear activation function. Prior works (Greydanus al.) 2019; Sanchez- Gonzalez et al. 2019: Lee et al. 2021) have utilized tanh or softplus activations, hypothesizing that the relu activation may hinder parameter optimization due to its piecewise linear nature, as the HNN loss defined in Equation 2 involves calculating derivatives of the output with respect to the inputs. After conducting several trials, we observed that the mish activation produced more stable and improved results compared to tanh or sofiplus activations in our case. More details regarding the network architecture are further described in the SM
Upon the framework that we described so far, we represent our model as fo; parameterized by neural network parameters 0. To account for the varying input (x') and output (œ') scales across different systems, we employ the log-cosh function our loss function. This choice enhances the robustness of our model to data with diverse system dynamics, as it mitigates the impact of differing data scales.
(Se = Σ log cosh( - JV vVelo((z)
(a (xr.
In the inner loop of our variation of MAML. the model adapts to a new task Tgo by updating the parameters with one gradient step using K 50 phase space points for each task.
8. = --0-aVoly aVoL (fo)
(4)
In the outer loop using the Adam optimizer (Kingma & Ba 2014), meta optimization across systems is done by updating the parameters as below.
0 NV0 Σ CT., (fe)
(5)
4

---

Published as a conference paper at ICLR 2024
mass-spring
pendulum
Hénon-Heiles
magnetic-mirro
Rel. err. btw. coords.
10-1
10-1
10-1
10
10-3
10-3
10
10
10.5
10:5
10-5
10
10
5
10
5
10
10
Rel. err. btw. energy
10-1
10
10-1
10"2
10
10
10
10-3
10-3
10-3
10-3
10
10
10
10
Time
Time
Time
Time
meta-trained model
pre-trained model
random-initalized model
Figure 1: The relative error throughout the time rollout for the meta-trained, pre-trained, and random- initialized model at adaptation step 50 for the predicted coordinates (top) and energy (bottom). The solid line and the shaded area each represent the average and the standard error of 10 runs.
3.3 EVALUATING THE META-TRAINED MODEL
We evaluate the performance by making a few-step gradient descent on the unseen type of system during the meta training phase with K 50 data points. The model adaptation to the unseen system done by updating the parameters as follows.
0 - 0 VoLT, (Jo)
Using the adapted model each step, integrate model output using the LSODA integrator (Pet- zold 1983) to achieve the system dynamics. Then, the performance was achieved by calculating the relative error between the predicted and ground truth trajectory points. We perform our experiments on NVIDIA RTX A6000 by performing 10 independent runs as default to ensure the stability of our results and report the average and standard error values. We again further describe the related details for the training and adaptation process in the SM B.2
RESULTS
4.1 QUANTITATIVE ANALYSIS OF THE MODEL PERFORMANCE
For each system, we evaluated the adaptation performance on unseen systems for the meta-trained model and compared it with the pre-trained model that doesn't utilize meta learning, and the scratch model which was adapted from random-initialized weights. We conducted up to 50 adaptation steps and computed the relative error between the predicted and the ground truth coordinates and energies at each time rollout and adaptation step. The measured relative eттor is defined as Err(t) = ||²(t) 22) ([] + [[=(t)] (12), as described in (Finzi al. 2020). This error metric enables us to quantify the error independent of the data scale and tends towards unity as the predictions become orthogonal to the ground truth or || >> ||=||- Compared metrics like mean squared error, the above notion of bounded relative performance. Finally, we calculated the geometric moving average of the relative error to account for the multiplicative accumulation of the error from the numerical integration involved.
provides
accurate representation of the prediction
In Figure the relative error in the predicted trajectories serves as an indicator the overall accuracy of the adapted models, while the energy metric demonstrates that the physics of the system remains well- belavee during the prediction. Considering both predicted coordinates and energy, our results demonstrate that the meta-trained model robustly outperforms the other two baselines.
5

---

Published as a conference paper at ICLR 2024
mass-spring
pendulum
Hénon-Heiles
magnetic-mirror
Rel. err. btw. coords.
10-1
10-1
10-1
10-1
10
25
50
25
50
25
50
25
50
Rel. err. btw. energy
10-1
10-1
10-1
10-1
25
50
25
50
25
50
50
Adaptation step
Adaptation step
Adaptation step
Adaptation step
meta-trained model
pre-trained model
random-initalized model
Figure 2: The relative error across the adaptation step for the meta-trained, pre-trained, and random- initialized model at adaptation step 50 for the predicted coordinates (top) and energy (bottom). The solid line and the shaded area each represent the average and the standard error of 10 runs.
We further examine the capability of fast adaptation of each model throughout the adaptation steps. Figure 2 illustrates the decreasing trend of the relative error as the number of adaptation steps increases from 0 to 50. Note that the meta-trained model requires significantly fewer adaptation steps compared to the randomly initialized model, which corresponds to the vanilla HNN model. Also, the relative еrror between the predicted coordinates shows that the fast adaptation shows significant difference as the system becomes more complex. Within a gradient step of less than 100, the meta-tras meta-trained model demonstrates faster and superior adaptation compared to both the pre- tre-trained model without meta learning and the HNN baseline.
4.2 QUALITATIVE ANALYSIS OF THE PREDICTED DYNAMICS
we observed that the meta-trained model consistently achieved lower relative error between the predicted and ground truth dynamics across both time rollouts and adaptation steps. However, a small error in the predicted trajectory does not always guarantee accurate system dynamics prediction. To provide more conclusive results, examined the exact predicted dynamics of the three adapted models after 50 steps. We conducted this analysis for the mass-spring, pendulum, Hénon-Heiles, and magnetic-mirror systems, and the corresponding results shown in Figure 3
In Section 4.1
For the mass-spring system, both the meta-trained model and pre-trained model accurately predicted the coordinates (x, P&) within 50 adaptation steps, while the iiniom-iitiaalized model required more steps to adapt. Here, it is worth noting the behavior of the prediction of the surplus coordinates (y, P.). In the real world, we cannot readily distinguish what coordinates remain physically meaningful among different phase variables. Thus, it would be beneficial if our trained neural network could also distinguish which one is necessary for describing the system. In the experiments, confined the dynamics of the system to two-dimensional space, thus y. Py should remain constant for the mass-spring system. Comparing the predicted trajectories of the redundant coordinates between the meta-trained model and the pre-trained model, observed that the meta-trained model had much smaller scale in the prediction of unnecessary coordinates (y, P,).
Moving to the pendulum system, the meta-trained model also provides better predictions than the other two baselines. While the pre-trained model performed better than the aundom-intializze model, the dynamics of and pa were still out of phase compared to the meta-trained model. We also observed a similar behavior in the prediction the surplus coordinates (r, p,), as observed in the mass-spring system.

---

Published as a conference paper at ICLR 2024
0.03
0.40
0.02
0.20
0.02
0.01
0.00
P.
y 0.01
Py0.00
0.20
0.00
0.00
0001
0.2
0.40
-0.50
-0.01
10
10
10
10
1.00
1.00
0.40
0.50
0.50
0.50
0.00
0.00
0.00
0..00
-0.50
1.00
10
2.50
0.40
JAN
MA
2.00
0.20
0.05
1.50
0.00
0.10
1.00
P.
0.20
y
Py
0.00
0.50
0.40
0.00
0.60
0.20
10
10
2.00
0.00
0.00
1.00
2.00
Pa1.00
-4.00
0.00
6.00
-6.00
8.00
-3.00
Time
Time
Time
Time
ground-truth
meta-trained model
pre-trained model
random-initialized model
Figure 3: System dynamics through time rollout 0 10, generated from the meta-trained, pre-trained, and random- random-initialize initialized model each after 50 adaptation steps. Each row (a) to (d) corresponds to the predictions with mass-spring, pendulum, Hénon-Heills, and the magnetic mirror system respectively.
For the more complex systems exhibiting chaotic behavior, such as the Hénon-HHille, and magnetic- mirror system, all four coordinates necessary to describe the dynamics. In these scenarios, the meta-trained model significantly outperformed the pre-trained model and random-initialin model. Although the meta-trained model did not perfectly match the trajectory, it captured the overall behavior of each coordinate dynamics more reasonably compared to the other baselines.
4.3 ANALYSIS ON THE LEARNED REPRESENTATION
Our objective in this section is to understand why the meta-trained model outperforms the other baselines, particularly the pre-trained model. The existing difference between the meta-trained model and the pre-trained model lies in the training strategy during the training process, while factors such as the total iterations, along with other training conditions remained the same. To investigate the impact of the training algorithm, we conducted the centered kernel alignment (CKA) (Kornblith et al. 2019) analysis on the three models. CKA offers a means to compare the representations of two layers within neural network, providing similarity score ranging from 0 to 1. Furthermore. it is widely recognized that early layers tend to learn general representations, while later layers tend to specialize in specific features/ Yosinski et al. 2014 Raghu et al. 2019 2020 Alabdulmohsin al. 2021).
7

---

Published as a conference paper at ICLR 2024
mass-spring
pendulum
Hénon-Heiles
magnetic-mirro
10-1
10-1
10-1
10
1-CKA
10 33
10
10-5
10-5
10-5
10"5
500
1000
1000
500
1000
500
1000
Adaptation step
Adaptation step
Adaptation step
Adaptation step
meta trained model
pre trained model
random initialized model
Figure 4: The measured CKAs in the last layer between the model before adapting (i.e. right after finishing meta- meta-traina training or pre-training), and during the adaptation up to 1000 steps. The CKA values are transformed to 1 1-CKA A to make clearer visualization. The solid line and the shaded area each represent the average and the standard error of 10 runs.
Upon this observation, we examine the CKA between the model's last layer before adaptation and its state each adaptation step, ranging from 1000. Figure shows these 1000 individual 1-CKA measurements, each comparing the model before adaptation to its state at a specific step during the adaptation process. In all four cases, the meta-trained model consistently exhibited substantially lower 1 1-CKA values compared to the other models. This finding suggests that the representations learned during the adaptation process closely resemble those of the meta-trained model when serves as the starting point. Drawing from insights gained in the prior works utilized meta learning (Lee et al. [2021] et al. 2023), and the manifold hypothesis (Fefferman et al., 2016), we interpret the traanntaiieed model gives the representation of the fundamental structure of underlying Hamiltonian dynamics. Thus making the meta-trained model acquire a generalized understanding of the diverse domain of distinct physical law. During the evaluation, the model adeptly adapts to unknown dynamics by leveraging the generalized physics learned during the meta training phase. Consequently, this observation provides further support for our initial assumption that meta learning can capture general representations across a broad domain of Hamiltonian manifolds.
shows
4.4 ABLATION ANALYSIS AND COMPARISON WITH EXISTING METHODS
First, we present the ablation studies to scrutinize the impact of both the choice of integrator and the number of data points in the adaptation task. We experimented with symplectic integrators ranging from 1st to 4th order to ensure energy conservation compared to the LSODA integrator employed in our primary experiments. And, since we are considering scenarios where abundant data is available for known systems but sparse data for the target unknown system, we varied the test data size as E (5, 10, 20, 50) to investigate the data efficiency of the meta-trained model relative to other baselines. The ablation results for varying integrator and K each detailed in SM C. and C.2 reveal several key findings. 1) LSODA integrator performs adequately for dynamics that do not involve large time scales, making it reliable choice for most scenarios considered in this study. 2) Meta-trained model demonstrates the data efficiency when adapting to unseen tasks. This suggests that the meta-trained model is not only data-eflicient but also robust, capable of adapting well when data for the target system is sparse.
Second, we demonstrate the generalization ability of the meta-trained model by not just testing the adaptation to a single-held-out system scenario as in the previous sections but also checking the adaptation performance on various types of unseen systems (i.e. performing the adaptation task to multi-held-ood held-out-system scenario). This task involved four models, each of which was initially trained for adaptation to one of the following systems; mass-spring, pendulum, Heinns and magnetic-mirror. We conducted an additional adaptation task using namely the 2D harmonic oscillator system and the Kepler system. We evaluated the performance these models in terms relative error for both predicted coordinates and energy, with detailed results presented in SM C.3 The results indicate that the meta eetantrained model is capable of effectively adapting to more than just a single type of unseen system, further suggesting the potential of general model for real-world applications. This underscores the robustness and versatility of the generalized representation learned by the meta-trained model.
8

---

Published as a conference paper at ICLR 2024
Third, we compared the meta-trained model to the existing baselines in domain generalization for learning the physical dynamics. Although there is none that aims to generalize across the type of dynamical systems we have already discussed in Section 2.3 it is important to recognize that the methods from previous works are not appropriate in our broader scope of generalization. We considered CoDA (Kirchmeyer et al.] 2022). which showed the best performance across prior methods for generalizing dynamics across multi-environment within fixed system setting, and DyAd (Wang) (model-based) meta- learning method as our baselines. The results in CM and C.5 show that the meta-trained model outperforms the CoDA and DyAd baselines for the generalization problem in cross domain settings. Moreover, note that in some cases, the existing baselines struggled to adapt.
al. 2022), which is a representative example of a non-optimization-bai
4.5 LIMITATIONS OF OUR APPROACII
In our experiments, we treated all systems as conservative. However, data from unknown physics may not satisfy these conditions, raising concerns about its upplicabiliity to real-world scenarios. Correspondingly, we performed the adaptation task to damped mass-spring system with a damping term -cù. From the results in SM 6M.
see that the non-conservative feature disturbs the adaptation. This is due to the damping term -cė, Equation i violated, which hinders the validity of Equation Saroonnnnuurii during adaptation. Thus to make use of our approach, both the data used in meta training and adaptation should not differ in their intrinsic nature.
Furthermore, we extend our analysis to two-body and three-body systems, going beyond the scenarios on single-partich tasks, thereby opening the door to real-world applications. This extension allows us to explore the dynamics of two-body systems as a direct continuation, and three-body systems as more challenging problem due to their chaotic behavior. However, we acknowledge that further refinement required for the meta learning configuration for non-single-particn tasks. Here, we present the preliminary results for the two-body and three-body systems in the SM C.7
5 DISCUSSION
While previous studies have focused on generalizing system dynamics across different environments under consistent physical laws, our work takes a step further by aiming for generalization across diverse system domains. Specifically,
tackle the challenge of identifying the shared representation of physical systems across various functional forms of Hamiltonian. In this perspective, our work presents unique contributions compared to existing literature. Here, we leverage the power of meta learning algorithm to uncover the representation of the Hamiltonian manifold within neural networks. In our comparison between the meta- trainet model and other baseline models, we provide compelling evidence that the neural network learns the Hamiltonian of an unseen system by identifying the general representation for Hamiltonian dynamics itself, rather than directly constructing the functional form of the Hamiltonian of the given system. This finding highlights the ability of meta learning to capture and exploit the underlying structure of Hamiltonian mechanics. To support our claim, we conduct CKA analysis on the trained networks. This analysis provides valuable insights into why the meta-trained model outperforms the other baselines across different physics. By examining the similarity between representations before and during the adaptation process, gain understanding of how the meta- trained model effectively learns and adapts to diverse physical systems. Overall, our work contributes the field by demonstrating the efficacy of meta learning in discovering shared representations of physical systems across distinct functional forms of Hamiltonian.
In nature, each physical system operates according to its unique governing law, which is rooted in fundamental principles. In the scope of this work, systems with different functional forms of Hamiltonian are expressed with one Hamilton's equation. Building upon this observation, we propose an approach that harnesses the power of meta learning algorithms to embed these fundamental physical principles into neural networks. Our method allows us to capture and represent these general physical principles within the neural network architecture, thereby providing a means to interpret neural networks physical context. key advantage of our approach that we imply minimal predefined physical priors or inductive biases. Instead, it leverages the data-driven capabilities of deep learning to uncover the underlying physics. This marks a significant step forward in developing a more comprehensive and general model for discovering physics through the lens of neural networks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This research was supported by the Basic Science Research Program through the National Research Foundation of Korea N
REFERENCES
Ibrahim Alabdulmohsin, Hartmut Maennel, and Daniel Keysers. The impact of reinitialization on generalization in convolutional neural networks. arXiv preprint arXiv:1109.00267 2021.
Vladimir Igorevich Amol'd. Mathematical methods of classical mechanics, volume 60. Springer Science & Business Media, 2013.
Suresh Bishnoi, Ravinder Bhattoo, Jayadeva Jayadeva, Sayan Ranu, and N M Anoop Krishnan. Learning the dynamics of physical systems with hamiltonian graph neural networks. In ICLR 2023 Workshop on Physies for Machine Learning. 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Yuhan Chen, Takashi Matsubara, and Takaharu Yaguchi. Neural symplectic form: Learning hamilto- nian equations general coordinate systems. Advances in Neural Information Processing Systems, 34:16659-16670, 2021.
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.044 04630, 2020.
Ch Efthymiopoulos, M Harsoula, and G Contopoulos. Resonant normal form and asymptotic normal form behaviour in magnetic bottle hamiltonians. Nonlinearity, 28(4):851, 2015.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2944):983- 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1126-113 1135. PMLR, 2017.
Marc Finzi, Ke Alexander Wang, and Andrew G Wilson. Simplifying hamiltonian and lagrangian neural networks via explicit constraints. Advances in neural information processing systems, 33: 13880-13889, 2020.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
Samuel James Greydanus, Timothy David Strang, and Isabella Caruso. Nature's cost function: Simulating physics by minimizing the action. In ICLR 2023 Workshop on Physics for Machine Learning, 2023.
Nate Gruver, Marc Anton Finzi, Samuel Don Stanton, and Andrew Gordon Wilson. Deconstructing the inductive biases of hamiltonian neural networks. International Conference on Learning Representations, 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10696-10706, 2022.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Michel Hénon. Numerical exploration of hamiltonian systems. Chaotic Behavior of Deterministic Systems (Les Houches, 1981), pp. 53-170, 1983.
Michel Hénon and Carl Heiles. The applicability of the third integral of motion: some numerical experiments. Astronomical Journal, Vol. 69, 73 (1964), 69:73, 1964.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, 2020.
Xiajun Jiang, Ryan Missel, Zhiyuan Li, and Linwei Wang. Sequential latent variable models for few-shot high- nigh-dimensional time-series forecasting, In The Eleventh International Conference on Learning Representations, 2022.
Muhammad Firmansyah Kasim and Yi Heng Lim. Constants of motion network. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint 1rX1::1609.0299000 2016.
Matthieu Kirchmeyer, Yuan Yin, Jérémie Donå, Nicolas Baskiotis, Alain Rakotomamonjy. and Patrick Gallinari. Generalizing new physical systems via context-informed dynamics model. arXiv preprint arXiv:2202.01889, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019.
Seungjun Lee, Haesang Yang, and Woojae Seong. Identifying physical law of hamiltonian systems via meta-learning. In International Conference on Learning Representations, 2021.
Qiaofeng Li, Tianyi Wang. Vwani Roychowdhury, and M. K. Jawed, Metalearning generalizable dynamics from trajectories. Phys. Rev. Lett., 131:067301, Aug 2023.
Paulette Libermann and Charles-Michel Marle. Symplectic geometry and analytical mechanics, volume 35. Springer Science & Business Media, 2012.
Ziming Liu and Max Tegmark. Machine learning conservation laws from trajectories. Physical Review Letters, 126(18): 10:184 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Linda Petzold. Automatic selection of methods for solving stiff and nonstiff systems of ordinary differential equations. SIAM journal on scientific and statistical computing. 4114 1983.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):54855555) 2020.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2024
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text image synthesis. In International conference on machine learning. pp. 1060-1069. PMLR, 2016.
Steindor Saemundsson, Alexander Terenin, Katja Hofmann, and Mare Deisenroth. Variational integrator networks for physically structured embeddings. International Conference on Artificial Intelligence and Statistics, pp. 3078-3087. PMLR, 2020.
Alvaro Sanchez-Gronzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint urXiv:1909.11 2019.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. Advances in Neural Information Processing Systems, 32, 2019.
Mingxing Tan and Quoc Le. Efficientner: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pp. 6105-61 14. PMLR, 2019.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, -722, 2020.
Rui Wang, Robin Walters, and Rose Yu. Meta-learning dynamics forecasting using task inference. Advances in Neural Information Processing Systems, 35:21640-21653, 2022.
Yuan Yin, Ibrahim Ayed, Emmanuel de Bézenac, Nicolas Baskiotis, and Patrick Gallinari. Leads: Learning dynamical systems that generalize across environments. Advances in Neural Information Processing Systems, 34:7561-7573, 2021.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
12

---

Published as a conference paper at ICLR 2024
A DATASET SPECIFICATION
A. TYPES OF SYSTEMS USED
Below are the descriptions of the systems that are used for the dataset.
Mass-Spring One of the most simplest physical system is particle, frictionless mass-spring system, where m, k are the mass of the particle, and spring constant respectively.
P2 2m
kr
H
Pendulum The Hamiltonian a pendulum system slightly more complex than the mass-spring case. Here, m, 9; denotes the mass of the particle, gravitational acceleration, and length of the pendulum respectively.
p
+ mgl(1 - cos 0)
2ml
Hénon-Heiie Up to here, the given dynamical system was rather simple. Hénon- gives the chaotic dynamics of star around galactic center with its motion constrained on a 2D plane (Hénon Heiles 1964] Hénon 1983). In the below Hamiltonian of the Ménonn-Heiles system, À is conven-
tionally taken unity.
1 H = 502 p2) 2222 +2
y
+ + x2y
Magnetic-Mirror Here we introduce a system that has the most complicated form of Hamiltonian in our experiments. From the works (Efthymiopoulos et al. 2015), the Hamiltonian of the magnetic bottle type system is given as follows.
1 H = - + 2) + 20² 2 g0" 0
16 0
128
Two-Body From now on, we expand our system with more than two particles. the two-body system case, we consider the gravitational interaction between two particles. Then the Hamiltonian of the two-body system can be written as follows. Note that is the gravitational constant, and mi,, m2 the masses for each of the two bodies.
Pi 2mm
P2
Gmima
2mn
rr
r2
Three-Body Adding particle to the two-body system gives the three-body system. Although the Hamiltonian of the three-body system is an incidental extension of the two-body case, its dynamics cannot be described by closed-fome expression, thus exhibits chaotic behaviour.
p 2ml
P
p3
Gm1m2
Gm2mg
Gmym1
H
2m2
2m3
r1
r2
12
rs
Fr3 ri
A.2 DATASET GENERATION
obtained the state trajectories (q. p) employing the LSODA integrator implemented in SciPy (Virtanen et al., 2002).. The trajectories were integrated over the interval [0, 10] with 200 steps. Subsequently, (q, P) using JAX (Bradbury et al. 2018) and Equation T For simplicity. all the constants from the Hamiltonian (i.e., mi, k. k, g, G) set to The initial conditions employed for each system described below.
Using the Hamiltonian described in Section 3.1
calculated the corresponding time derivatives
Mass-Spring The initial state (Zu, Pæo) is randomly sampled from uniform distribution over the interval [1,, The redundant coordinate (yo, Pyo) set to (0, 0) as a fixed value.
Pendulum
The initial state Poo) is randomly sampled from a uniform distribution over the
interval
[- - x] [-1,1] 1). The redundant coordinate (ro, Pro) is set (1, 0) as fixed value.
13

---

Published as a conference paper at ICLR 2024
Hénon-Heiles The initial state (Io, you Pro+ Pvo) is randomly sampled from uniform distribution over the interval |-1,
Magnetic-Mirroo The initial state (Po. 20. Pro 20) is randomly sampled from a uniform distribution over the interval 1|4.
Two-Body The initial coordinate for the first body (I10: V1o) is randomly sampled from uniform distribution over the interval [0.5, 1.5]², and (Pe.o- is calculated to obtain nearly-circular orbit. Then, the initial state for the second body (z20- y20= Przu Pyan) is to (-I10- -Mio- PP Pys o Then we slightly perturbed Px-Py by adding a Gaussian noise (multiplied by a constant of 0.1) to the velocity of both two bodies. Here, the velocities are equivalent to the canonical momentum Pa and Py, we assume the masses mı and m2 to be equal to 1.
Three-Body The initial state for the three-body system obtained in a similar way to that of the two-body, except the initial coordinate for the first body (I10- V/g) is randomly sampled from uniform distribution over the interval [0.8, 1.2 Again, (Px, o- Pyio) ) is set to obtain a nearly-circular orbit. The initial state for the second and third body is obtained by rotating the first and second body each by an angle of 2 Here, the Gaussian noise term that is added to each of the bodies is multiplied by constant of 0.05.
3.
Following the above description, we generated 10000 trajectories for each system.
A.3 DATASET CONFIGURATION FOR META TRAINING
The composition of the system types used in meta training scenario is listed in Table 1
Table 1: Dataset configuration for meta learning Hamiltonian systems
system to test
systems for meta training
mass-spring
pendulum, Hénon-Heiles, magnetic-mirro mass-spring, Hénon-Heiles, magnetic-mirror mass-spring, pendulum, magnetic mass-spring, pendulum, Hénon-Heiles
pendulum
Hénon-Heiles
magnetic-mirror
two-body
mass-spring, pendulum, Hénon-Heiles, magnetic-mirror
three-body
mass-spring, pendulum, Heinns, magnetic-minet mirror
B NEURAL NETWORK IMPLEMENTATION
B.1 NETWORK ARCHITECTURE
In our experiments, we constructed our model as follows: GCNConv(4, 200) Mish GCNConv(200, 200) Mish GCNConv(200, 4) Mish Global Linear(4, 200) Mish Linear(200, 200) Mish Lincar(200, 1) Mish, where GCNConv and Linear correspond to the graph convolutional layer and the fully connected layer implemented in PyTorch respectively. Thus, the model outputs a single-dimensional scalar value. After the forward pass, the derivative of the output with respect to the input computed to obtain the -ime-derivative of the input state, utilizing Equation I Note that we did not heavily tune the hyperparameters regarding the network architecture, because we focus on the difference between the meta-trained, pre-trained, and aanom-initialiin model, not obtaining state-of-the- results.
B.2 TRAINING PROCESS
Mass-Spring For both meta-training and pre-training, we used learning rate of = 0.001 for the gradient step on the inner loop, and the Adam optimizer on the outer loop with learning rate of = 0.0005. The inner gradient update was step, with a total of 5000 iterations on the outer loop. The number of task batches is set to 10, and the number of phase points used for each task was 50 (i.e. among the 200 points in each trajectory, 50 points were randomly sampled for meta training). For evaluation, used the Adam optimizer with learning rate of 0.0001.
14

---

Published as a conference paper at ICLR 2024
mass-spring
pendulum
Hénon-Heiles
magnetic-miroe
Rel. err, btw. coords.
10-1
10-1
10
10
x 10
6 10
10-2
4x
4
x
LSODA
$2
$3
$4
LSODA S1
52
53
54
LSODA
S3
S4
LSODA
S2
S3
54
Rel. err. btw. energy
4x
4
10-
3 3
10-1
-
x
10-1
10-1
LSODA S1 $2
$3
S4
LSODA
$2
$3
S4
LSODAS1
S2
S3
S4
LSODA $1
S2
S3
54
integrator
Integrator
integrator
Integrator
meta-trained model
pre-trained model
random-inttillzee model
Figure S.I: The relative error for the meta-trained pre-trained, and respect the predicted coordinates (top) and energy (bottom) using different integrators. The solid line and the shaded area each represent the average and the standard error of 10 runs.
model with
Pendulum All conditions were set to be the same as those of the mass-spring system.
Hénon-Heiles All conditions set to be the same those of the mass-spring system, except for the iterations on the outer loop which was set to 10000.
Magretic-Mirror All conditions were set to be the same as those of the mass-spring system, except for the iterations on the outer loop which was set to 30000.
Two-Body For both meta-training and pre-training, we used a learning rate of a 0.001 for the gradient step on the inner loop, and the Adam optimizer on the outer loop with learning rate of 0.0005. We set 10000 iterations the outer loop. The other conditions were set to be the same.
Three-Body All conditions were set to be the same those of the two-body system, except for the 3 which is set to 0.0005.
C ADDITIONAL EXPERIMENTS
C.1 ABLATION ON INTEGRATORS
Figure shows the evaluation performance of meta-trained, pre-trained, and randomly-inittalizze models across scenarios using different integrators; LSODA, and the 1st-4th order symplectic integrator. The LSODA integrator consistently yields the best results in most cases. While symplectic integrators known for their ability maintain energy conservation, their advantages become more apparent in long-term dynamics. However, the time range considered in this study is not sufficient to fully capture the benefits of symplectic integrators. Nonetheless, their impact could be significant when extended to longer time scales. So in our experiments, use LSODA primary integrator.
C.2 ABLATION ON
presents the evaluation performance of meta-trained, pre-trained, and randomly-initialized models under varying K in the adaptation task. The ablation results indicate that the meta-traine model consistently performs well at low K values across adaptation tasks. Furthermore, its perfor- mance markedly improves as the data size K increases, which reflects the efficient adaptability the meta-trained model.
Figure S.2
15

---

Published as a conference paper at ICLR 2024
Rel. err. bbw. energy Rel. err. btw. coords.
mass-spring
pendulum
Hénon-Heiles
magnetic-miroo
10-1
6x-0
6
10"2
10
4x
3x00-2
50
10
50
50
40
4x
10-1
3x
10-1
x 10-1
x
10-1
10-1
10
50
10
50
10
50
10
50
K
K
mneta-trained model
pre-trained model
random-initilizzed model
Figure S.2:
The relative error for the meta-trained, pre-trained, and respect the predicted coordinates (top) and energy (bottom) with various number of data points K used. The solid line and the shaded area each represent the average and the standard error of 10 runs.
model with
C.3 GENERALIZATION FOR MULTI-HELD-OUT SYSTEM
Here, we present the adaptation performance the meta-trained models Section 4 to the multi- held-out system scenario for the extent of the single-held-out scenario demonstrated in Section 4 For example, we performed the adaptation task of the meta-trained model trained with pendulum, Hénon-Heiles, and the magnetic-mirror system only for the mass-spring system, but also with the newly introduced 2D harmonic oscillator, and the Kepler system as well.
The relative errors terms both predicted coordinates and energy for this are presented in Figure SS.3 and SS44 The results demonstrate that the meta-trained models exhibit proficient adaptation both to the newly introduced 2D harmonic soscillator system and the Kepler system.
Trained w/ (pd, hh, mm) Trained w/ (ms, hh, mm)
Trained w/ (ms, hh, mm)
Trained (ms, pd, hh)
Rel. err. btw. coords.
10-1
10-
10-1
10-1
10-3
10
10
10
10-5
10
10-5
10
10
5
10
10
5
10
Rel. err. btw. energy
10
10-1
10-1
10"
10-2
10-2
10-2
10-7
10-3
10-3
10-1
10
10
5
10
0
5 Time
10
Time
Time
Time
meta trained model
pre-trained model
uundominitializen model
Figure S.3: The relative error throughout the time rollout of the 2D harmonic oscillator system for the meta-trained, pre-trained, and andom-initializen model at adaptation step 50 with respect the predicted coordinates (top) and energy (bottom). The solid line and the shaded area each represent the average and the standard error of 10 runs.
16

---

Published as a conference paper at ICLR 2024
Trained w/ (pd, hh, mm)
Trained w/ (ms, hh, mm)
Trained w/ (ms, hh, mm)
Trained w/ (ms, pd. hh)
Rel. err. btw. coords.
10-1
10"1
10-1-
10
10-1
10-1
10-5
10-
10
5
10
5
10
0
5
10
0
5
10
Rel. err. btw. energy
10"1
10
10-1_
10-1
10a2
10
10"2-
10
10
10
10-3
10
10
10
10
5
10
Time
Time
Time
Time
meta-trained model
pre-trained model
random-inittllzedd model
Figure S.4: The relative error throughout the time rollout of the Kepler system for the meta-trained, pre-trained, and random-initialized model at adaptation step 50 with respect to the predicted coordi- nates (top) and energy (bottom). The solid line and the shaded area each represent the average and the standard error of 10 runs.
1.00
1.00
0.00
0.50
0.00
P,1.00
2.00
-050
33.00
-0.50
0.60
0.50
0.40
1.00
0.40
0.25
0.50
0.00
0.00
0.00
0.00
4.20
0.50
00.50
-4.40
-0.00
-3.60
10
10
1.00
0.50
1.00
0.40
0.00
0.50
0.20
Pxo.00 0.00
y 0.00
Py0.00
1.00
0.50
0.50
0.20
11.50
-1.00
0.40
0.60
5
10
5
10
10
5
10
ground-trui truth
meta trained model
pre-trained model
handom-initializen model
Figure S.5: The example predicted dynamics for the 2D harmonic oscillator system (a), and each good (b) and bad (c) examples of Kepler system.
We also show the example of the predicted dynamics for both the 2D harmonic oscillator and the Kepler system in Figure SS.3] There exists a situation for the Kepler system where the meta-trained model does not perform well, which implies insufficient adaptation steps for this case.
17

---

Published as a conference paper at ICLR 2024
0.40
0.40
0.00
V
V
0.20
0.20
40.0
0.00
Pv
0.00
Py
0..00
-0.00
-0.0
0.40
0.40
-0.40
10
10
5
10
0.25
0.50
1.00
0.50
0.25
0.25
P4 00
Pro.00
0.50
0.00
1.00
10
10
10
5
10
0.15
0.40
0.05
0.05
Pk 0.00
0.00
0.00
-0.00
0.20
0.05
0.10
10
5
10
1.00
1.50
W
1.00
1.00
0.50
M
0.50
0.00
Pr 0.00
0.00
0.00
0.550
-1.00
0.50
00.50
-1.00
-1.00
-1.00
Time
Time
Time
Time
ground-truth
meta-trained model
CoDA, steps
CoDA, 10000 steps
Figure S.6: The prediction each system dynamics from the meta-trained after 50 adaptation steps, and CoDA after 50 and 10000 steps. Each row (a) (d) corresponds to the predictions tested with mass-spring, pendulum, Hénon-Heiles, and the magnetic mirror system respectively.
True dynamics
True dynamics
Predicted ted dynamics
Predicted dynamics
Y
Pr
y
Pr
-1.5
P,
Px
Figure S.7: The predicted dynamics of the magnetic- mirror system using DyAd.
C.4 COMPARISON WITH CoDA
The comparison of the meta-trained model between CoDA and DyAd is depicted in Figure S.6 From the results, the dynamics from CoDA after 50 steps (red dotted line) not sufficient match the performance of the meta-trained model, so we also made a comparison with the CoDA after 10000 steps (red dashed line), which also failed to meet the expectations.
18

---

Published as a conference paper at ICLR 2024
a
6==10 10
1=200-
0.3
0.3
0.3
0.3
Px
Px
Px
Px
0.0
-0.3
-0.3
-0.3
0.0
0.3
-0.3
0.0
0.3
-0.3
0.0
0.3
10-1
10
10-1
10
err.
3e-1
1e-1
1e-1
1e1
10
10
3e-2
10-1
3e-2
10
3e2
1e-2
1e-2
1e2
10-1
10-1
10-1
10
10
10
5 Time
Time
Time
meta-trained model
pre-trained model
random-initialized model
Figure S.8: (a) shows the dissipative dynamics of the mass-spring system as the damping coefficient changes. The leftmost subfigure in (b) indicates the geometric moving average of the relative error time for different values of damping coefficient c. The three right subfigures in (b) represent the relative error through the time rollout with different values of e for each three models.
Although CoDA is reported to outperform MAML-based methods the scope generalization across environments within fixed system, our results show that MAML-based method advantages CoDA in our broader scope of definition (i.e. generalization across different types of systems).
C.5 ADAPTATION WITH DYAD
Contrary to CoDA, where inputs are represented by the state variables (g, p) of the system, Dy Ad relies on image sequence inputs. To this end, we generated sequences of images with dimensions 2 x 128 128, where each channel corresponds to the ay space and PxP, space, respectively.
However, the weak supervision term DyAd's encoder loss (the first term in Equation 7
Cens
Σ - + Σ 1 AEC
2000
+ Σ
!1 m||
$.1.c
where 9 is the encoder network, x is the input, 200 = (x) and 2(0) = w + an affine transformation of &e which as a hidden feature for task c, was directly applicable our scenario due to the varying nature of physical parameters across different systems. To adapt this notion to our needs, we have chosen to represent the physical parameter as the energy the system. While it may not be exact to say that the explicit functional form of energy is common to all systems, the use energy as form of weak supervision can effectively utilize the minimal concept of physical parameters across different dynamical systems. We conducted adaptation task for the bagnetic-mirre system using the meta-trained model, which was previously trained on mass-spring, pendulum, and Hénon-Heiles systems. The results show that, illustrated in Figure S.7 DyAd is not appropriate to our scope of domain generalization.
C.6 ADAPTATION TO DISSIPATIVE SYSTEM
Here, we performed adaptation task to the damped mass-spring system with damping term -ci with the same meta-trained model that was used to adapt to the original mass-spring system (i.e. meta-trained with conservative pendulum, Hénon-Heiles, magnetic-miror system). From Figure [S8 we can see that the meta-trained model with conservative systems cannot easily make an adaptation dissipative system, which indicates definite limitation in problem setting.
19

---

Published as a conference paper at ICLR 2024
C.7 DEMONSTRATION ON LARGER SYSTEMS
In this section, we present preliminary results for the two-body and three-body tasks. We recognize that the training conditions, such as the learning rate for adaptation, a result, while the CKA exhibits behavior similar to that observed in other single-particle tasks as shown in Figure S.10 the prediction accuracy remains suboptimal, as illustrated in Figure S.9 The predicted trajectories are depicted in Figure It should be noted that while none of the models successfully capture the overall dynamics, the meta-trained trained model tries to capture the trend than the others. This suggests that with further refinement of the training conditions, the meta-trained model holds promise for achieving more accurate predictions.
sufficiently refined. As
Rel. err. btw. coords.
10
two-body
three-body
10-2
10-2
10-3
10-2
Time
Time
meta-trained model
pre tre-trained model
random-inital innttaliee iized model
Figure S.9: The relative error throughout time rollout for the meta-trained, pre-trained, and random-
initialized model. The solid line and the shaded area each represent the average and the standard error of 10 runs.
two-body
three-body
10-1
10-1
1-CKA
10
10-1
10-5
10-5
Adaptation step
Adaptation step
meta-trained model
pre-trained model
random-inits initialized model
Figure S.10: The measured CKAs in the last layer between the model before making adaptation and during adaptation. The solid line and the shaded area each represent the average and the standard error of 10 runs.
20

---

Published as a conference paper at ICLR 2024
02
0.5
X1
P.0.5
Y1
Py. 0.0
-.8
0.0
02
-100
1.0
4.4
1155
10
1.5
1.2
1.0
1.0
0.5
0.2
X2
5/2
Pr
0.0
11.0
0.0
-0.5
-1.0
10
10
0.75
0.5
Y1 0.0
0.00
-0.25
-0.5
-0.50
1.0
2.0
0.5
0.5
1.5-
Pr
Y2 0.0
Py,
0.0
0.0
-0.5
0.5
0.0
-0.
10
10
10
10
1.5
1.0
1.0
1.0
0.0
0.0
-033
-0.5
-1.0
005
-1.
-1.0
-1.5
-1.0
Time
Time
Time
Time
ground-truth
meta-trained model
pre-trained model
random-initialized model
Figure S.11: The example predicted trajectories of (a) two-body, (b) three-body systems.
21


Reviews:
[{"review_id": "ux7s4C5vaw", "reviewer": "Reviewer_hDu2", "summary": "This paper proposes to use meta-learning to learn generalized representations across different types of dynamical systems. The meta-learning step helps improve the adaptation to unknown systems with fewer data points (compared to randomly initialized and pre-trained baselines) by virtue of generalized representations. The authors also analyze the representations learned by different baselines and meta-learning by using centered kernel alignment (CKA) to gain insights into better performance by the meta-learning model.", "questions": "Please check the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow and the motivation to learn a generalized model is clear. \n- The experiments are performed with different numbers of data-points for the adaptation task to evaluate the robustness of the approach. \n- The analysis using CKA gives further insight into how the meta-learning model learns closer representations of the adaptation task. \n- The implementation and task curation are described in detail for reproducibility.", "weaknesses": "- The main contribution of the paper seems to be utilizing meta-learning to efficiently adapt to new systems. However, it is not clear from the paper if it is as simple as just using off-the-shelf implementation or if there are some challenges to doing this. \n- Also, I would like to see some discussion around why meta-learning is preferred over other representation learning methods e.g. Domain Generalization, and why optimization-based methods surpass other approaches in meta-learning.\n- The experiments are not sufficient. First, there is no comparison with existing baseline models in domain generalization or meta-learning. Second, the comparison of the meta-model and pre-trained model is not fair, and I would suggest the author fine-tune the pre-trained model on the K-shot support set. Third, no visual comparison of the predicted dynamics and the ground truth, making the conclusion less convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FBXRP3lnzq", "reviewer": "Reviewer_jozM", "summary": "The paper reports the performance of MAML applied to domain generalization over different Hamiltonian dynamics. The performance gain by MAML is analyzed with multiple indices over different combinations of meta-training and meta-test data. The key findings include the superiority of the meta-trained models compared to the pre-trained and randomly initialized models and an implication that by meta-learning, the representations obtained by the models tend to be more specific to each system.", "questions": "I don't have particular questions. It would be great if the authors could additionally report the results of some experiments to investigate the limitation of meta-learning in this context, although this is not a question.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The experiments clearly show the superiority of meta-learning, at least within the limited number of Hamiltonian systems. \n\nThe paper is well written. The motivation, the method, and the experimental results are very clearly reported.", "weaknesses": "The limitation of meta-learning for (Hamiltonian) dynamics is not clearly investigated. This makes it difficult to assess the range where the claims made in the paper should be valid. In other words, the claims are rather weak because their applicability seems unbounded with the current set of experiments. When the meta-learning approaches for dynamics may not be beneficial? For example, what happens if you meta-train a model only with conservative systems and try to adapt it to a dissipative system? Such experiments to investigate the limitations of the empirical findings would strengthen the paper.\n\nThe paper only reports the performance of a well-known method (MAML) merely applied to a particular setting. This could certainly be a kind of contribution in which ICLR audience may be interested, but I think that in such a paper, with a purely experimental point of view, the claims should be made more carefully. Specifically, as stated above, the cases where meta-learning is not necessarily beneficial should also be revealed, with which the claims would become more falsifiable and convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPDVvgTzbR", "reviewer": "Reviewer_xLig", "summary": "This paper proposed a novel meta-learning method aiming to learn a unified Hamiltonian representation such that it can be generalized to unseen physical systems. Hamiltonian Neural Networks were utilized as the backbone of the method for learning the unified Hamiltonian representations of different physical systems, via meta-learning pipelines of a variation of MAML. Experiments demonstrated the proposed methods achieved lower relative error of trajectories and energy when adapted to different systems.", "questions": "In Figure 4, why does a lower CKA value of the meta-trained model suggest it learned more similar representations during adaptation? As the authors mentioned, should a low CKA value indicate different representations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Unlike many existing works that focus on learning system dynamics under similar physical law, the proposed methods aim to learn the unified representations across diverse system domains via meta-learning the Hamiltonian of the given system. This sounds significant and promising.\n\n(2) Both quantitative and qualitative results demonstrated the proposed method achieved better adaptation to various new systems compared with baselines.", "weaknesses": "The evaluation can be strengthened by considering comparing the proposed methods with other Few-shot Learning and Physics-informed Neural Networks methods for system domain generalization under both \"consistent\" and \"different\" physical laws.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWxMIfTpWd", "reviewer": "Reviewer_GtBu", "summary": "This work leverages the power of meta-learning algorithms coupled with graph neural networks to find a shared representation of physical systems across various functional forms of Hamiltonian. In contrast to previous work, here the focus is on obtaining a representation which is valid across different physical systems. The performance of the framework is evaluated over a range of physical systems aiming to showcase its adaptivity to unseen settings.", "questions": "1. Can the authors provide additional ablation studies following the weaknesses presented above? Specifically, it will be beneficial to present the performance as a function of the number of systems used in training (see weaknesses 1.) and add additional baselines  (see weaknesses 2.) \n2. Judging from the presented results the current framework is not suitable for larger systems, could the authors suggest possible extensions that may allow? What would be the necessary refinements that could be incorporated in the meta-learning configuration to allow for that?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "$\\underline{\\textrm{Originality}}$: the paper presents great originality in providing a Hamiltonian representation learning framework that can generalize across multiple system domains. This is in contrast to the common approach of providing system-specific models.\n\n\n$\\underline{\\textrm{Quality}}$: the paper is very well written. First, it provides the reader with all necessary background as well as motivation for the task addressed. Next, methods and results are well constructed. \n\n\n$\\underline{\\textrm{Clarity}}$:  the main ideas conveyed in this paper are clearly constructed and explained and supplementary information assists with providing further details and ablation studies. \n\n\n$\\underline{\\textrm{Significance}}$:  the main significance of the paper is in defining a new task, generalizing upon existing approaches, and suggesting to derive a framework that learns a representation that is not domain-specific.", "weaknesses": "The paper presents an appealing goal, providing generalized Hamiltonian representations consistent across different physical domains. However, given the presented quantitative and qualitative results it is hard to judge the actual generalization and performance of the framework as detailed in the following points:\n\n1. The notion of $\\textit{generalization}$: ideally when discussing generalization in DL we would like to obtain a single pre-trained model which can then be used for diverse applications. With respect to the presented framework, this would suggest training the model(s) on a single task and then using the same network for prediction on all held-out systems. Similar to the setting presented in Ricci et al. (2023). However, here presented results always consider a single held-out-system. Providing an ablation over the number of systems used for training will allow for strengthening the claim of generalization and applicability for real-world applications.\n\n2. Baselines: it would be beneficial to extend the baselines presented in the paper in two directions:\n(i) optimal;  Training over the tested task, using all regimes (meta-, pre-, and vanilla HNN). This will allow a better assessment of the quality of the generalized model and (ii) within system generalization; following the background presented in section 2 it will be valuable to add a comparison to frameworks that are similar in nature and allow $\\textit{within}$ model generalization, e.g. CoDA (Kirchmeyer et al. 2022) or within the same functional form of the Hamiltonian, e.g. iMODE (Li et al. 2023). Here training over the same train-test splits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)