Decomposing heterogeneous dynamical systems with graph neural networks

Authors: Cedric Allier, Magdalena C. Schneider, Michael Innerberger, Larissa Heinrich, John A. Bogovic, Stephan Saalfeld
Primary area: learning on time series and dynamical systems
Keywords: graph neural networks, gnn, dynamic system, latent parameter discovery

Abstract:
Natural physical, chemical, and biological dynamical systems are often complex, with heterogeneous components interacting in diverse ways. We show how simple graph neural networks can be designed to jointly learn the interaction rules and the latent heterogeneity from observable dynamics. The learned latent heterogeneity and dynamics can be used to virtually decompose the complex system which is necessary to infer and parameterize the underlying governing equations. We tested the approach with simulation experiments of interacting moving particles, vector fields, and signaling networks. While our current aim is to better understand and validate the approach with simulated data, we anticipate it to become a generally applicable tool to uncover the governing rules underlying complex dynamics observed in nature.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730361579294

thread_type:
official_review

<thread_object>
creator: reviewer_kdqj

<content>
summary:
This paper demonstrates that GNNs can be designed to jointly learn both interaction rules and heterogeneous structures directly from data. The learned latent structures and dynamics can then be used to decompose complex dynamic systems and infer the underlying governing equations. To evaluate the proposed approach, simulation experiments on moving particles and vector fields are conducted, highlighting its potential for capturing intricate dynamics.

strengths:
The paper introduces a novel approach where GNNs jointly learn interaction rules and heterogeneous structures directly from data, enabling the decomposition of complex dynamic systems and inference of underlying governing equations. Simulation experiments on moving particles and vector fields demonstrate the model's effectiveness in representing complex dynamics, with promising potential for broader applications across various types of dynamic systems.

weaknesses:
Novelty: Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs) are well-established frameworks for modeling the evolution of dynamical systems over time. Several ODE/FDE-based GNNs, such as Continuous Graph Neural Networks (CGNN), have leveraged this approach to simulate time-evolving, interacting components in these systems. While the authors mention using simple MLPs instead of differentiable functions, it remains unclear what advantages MLPs offer over ODE-based methods. A discussion on this aspect would clarify the benefits of MLPs in the proposed approach and strengthen the novelty claim. Is it possible to provide a comparative analysis between their MLP-based approach and ODE-based methods like CGNNs, and also discuss the relative advantages and disadvantages in terms of computational efficiency, expressiveness, or ease of implementation? 

Lack of experiments over real data: While the approach is tested on simulated data, it lacks validation on real-world data observed in natural systems. This limitation raises concerns about the method’s practical effectiveness and generalizability to real-world dynamics, where noise and complexities may differ significantly from simulations. Including experiments with real data would strengthen the evidence for the approach’s applicability, for example, applying the proposed method to data from physical experiments, biological systems, or social networks is more convincing.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730396657955

thread_type:
official_review

<thread_object>
creator: reviewer_ca5v

<content>
summary:
This paper demonstrated that a graph neural network can both learn the dynamics and structure of a dynamical system. The parameters showed both underlying structure and differential equations. The proposed model was tested on several simulated datasets, and visually showed a meaningful relationship between the true and predicted values.

strengths:
This model can be applied to various complex realistic systems to reveal the underlying structure and dynamics.

weaknesses:
1. The presentation is poor. It is hard to track simulations they did for the model. People need to check figure, table, video and supplementary figure to understand what they have done without any hint or explanation.
2. The interpretations are lack to explain their results. Only showing several latent representations are not enough to convince or help understand the model's strength.

questions:
1. Have you compared your proposed model with a conventional GNN, i.e., the loss is l2-norm between x and x_hat? What is the significant improvement by the proposed model?
2. How do you choose to use first-order or second-order derivative to train the model?
3. At a single time point, why don't you use both first-order and second-order or even with higher-order derivative to train the model, where the information should be more compact?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695630919

thread_type:
official_review

<thread_object>
creator: reviewer_laea

<content>
summary:
This paper fits GNNs to dynamic systems and showed that it is feasible.

strengths:
- The figures are well-presented.
- The experiment details are clear.

weaknesses:
- First of all, what does heterogeneity mean in this context? Meaning that the particles are of different types? The closest thing I can find to a definition is "The latent heterogeneity of the particles is encoded by a two-dimensional learnable embedding $a_i$ that is part of the node features." But why two?
- The simulation of dynamic systems has been routinely done by GNNs, most notably Sanchez-Gonzalez et al., 2020, as cited in the paper. The only difference seems to be that they do not consider this "heterogeneity" whose definition is not clear? But does this method introduced here even outperform the baselines without such input?
- This paper uses a very classical method that is not equivariant on 3-dimensional space. How does it deal with rotational equivariance?
- There is no baseline nor quantitative comparison.
- No code is provided.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731395986276

thread_type:
official_review

<thread_object>
creator: reviewer_tcqf

<content>
summary:
The paper proposes using graph neural networks (GNNs) to jointly learn interaction rules and heterogeneous structure in complex dynamical systems from data alone. Extensive experiments on simulated systems including particle interactions, wave propagation, reaction-diffusion, and signaling networks, showing its good performance.

strengths:
1. The paper is in general easy to follow and with clear writing flow. The problem is well-motivated, by using GNN to learn system dynamics over time and in the meanwhile, uncover the underlying latent properties in an interpretable way that facilitates further analysis.

2. The evaluation of dynamical systems in the experiment sections are extensive, though adding some baselines for comparison would be better.

weaknesses:
1. There is no related work section. Some works are discussed in the introduction part, but there are many existing neural simulators that use GNN to rollout trajectories of multi-agent dynamical systems [1,2,3,4]. Discussion about existing work and comparison in the experiment section are helpful to provide a comprehensive analysis.

2. As mentioned above, for rollout MSE across different datasets, it is suggested to compare against representative baselines. Also the run time comparison can be included across compared methods. 





[1]  Learning Continuous System Dynamics from Irregularly-Sampled Partial Observations.

[2] Interaction Networks for Learning about Objects, Relations and Physics.

[3] Learning to simulate complex physics with graph networks.

[4] HOPE: High-order Graph ODE For Modeling Interacting Dynamics

questions:
The authors mention the method can infer the underlying governing equations, (line 17) but I do not see any analysis in the experiment part. It would be interesting to see how can we extract formula from a learned GNN.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732571643671

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Working on improving the text

comment:
Dear reviewers, thanks a lot for your candid reviews.

We realize that we failed to make it clear how our approach is different from the related work that you mention and that we discuss in the introduction.  We are in the process of changing the text to make this more clear, but in the meantime, here is a short summary of what's new:

1. Simulation: We simultaneously learn unknown interaction- and update functions and an embedding of latent heterogeneous properties from observations of dynamical systems.
2. Interpretation: We do this in a way that makes it easy to use the learned embedding and functions to infer the underlying rules and properties governing the dynamical system.

We achieve this by training simple GNNs with single MLPs for interactions and/or updates of particle states that are parameterized by the observable states and a low dimensional learnable latent vector for each particle.  We then sample the learned functions from the learned latent embedding space and infer symbolic rules and their parameters, similar to the work on orbital dynamics by Lemos et al (2023).  The significant difference to their approach is that they chose to explicitly learn a latent scalar $b$ corresponding to a latent factor multiplied with a learnable function $b F(x)$.  In contrast, we learn a latent vector $a$ that the learnable function $F(a,x)$ uses to map out an arbitrary latent parameter space.  We chose the latent vector $a$ to be 2-dimensional which was the lowest dimensionality that worked in all our experiments (it also prints well on paper and is easy to interpret). 

We kept the examples as simple as possible to demonstrate how this can be used for a diverse set of simulated quasi-physical simulations whose interactions depend on 1 to 4-dimensional latent parameters, and show how the learned MLP and the embedding can be used to infer symbolic interaction functions and their parameters.

To demonstrate how this approach can be extended for more complex systems, we added an example that includes latent external inputs that affect---but themselves are not altered by---the dynamical system (Fig. 4), and an example where the interaction matrix has to be inferred (Suppl. Fig. 18).

To our knowledge, this approach is new, so there is no meaningful baseline to compare to.  E.g., if we train LG-ODE with different spring constants or Learning-to-simulate without known particle types, we do not get meaningful results, because those methods are not designed to infer these latent parameters.

Even though we were not successful at generating reasonable results from training data with variable spring constants with LG-ODE, we think that it should be possible for methods without a dedicated latent state embedding to infer plausible behavior from extended temporal context if sufficient training data is provided (e.g., LG-ODE uses the entire time series, whereas we use a single time step).  However, the underlying properties responsible for this aspect of the behavior (e.g., variable spring constants) would be significantly more difficult to extract from the trained parameters of the network.  Interestingly, one could likely use our method to infer them from the learned simulation.
We think that those topics would be super interesting to investigate in future work but are not within the scope of this manuscript.

We hope that you agree with us that our presented method is an exciting new way to use GNNs to infer the rules and latent properties of physical and biological dynamical systems and we would love to discuss future experiments and ideas with conference participants at ICLR.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a graph neural network framework for learning the dynamics of a physical system, which is tested on several simulations.

We had four reviews, all negative. All reviewers are unanimous concerning the lack of novelty, poor writing and presentation (e.g., Figure 1 in the paper), lack of real-world simulations, missing baselines, and several other concerns.

I see no reason to overrule this consensus, and I suggest to reject the paper.