Learning system dynamics without forgetting
Authors: Xikun ZHANG
Dongjin Song
Yushan Jiang
Yixin Chen
Dacheng Tao
Venue: ICLR 2025 Poster (2025)
Keywords: graph neural networks
AI4Science
physics
biology
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Observation-based trajectory prediction for systems with unknown dynamics is essential in fields such as physics and biology. Most existing approaches are limited to learning within a single system with fixed dynamics patterns. However, many real-world applications require learning across systems with evolving dynamics patterns, a challenge that has been largely overlooked. To address this, we systematically investigate the problem of Continual Dynamics Learning (CDL), examining task configurations and evaluating the applicability of existing techniques, while identifying key challenges. In response, we propose the Mode-switching Graph ODE (MS-GODE) model, which integrates the strengths LG-ODE and sub-network learning with a mode-switching module, enabling efficient learning over varying dynamics. Moreover, we construct a novel benchmark of biological dynamic systems for CDL, Bio-CDL, featuring diverse systems with disparate dynamics and significantly enriching the research field of machine learning for dynamic systems. Our code available at \url{https://github.com/QueuQ/MS-GODE}.

Published as a conference paper at ICLR 2025
LEARNING SYSTEM DYNAMICS WITHOUT FORGETTING
Xikun Zhang
Dongjin Song & Yushan Jiang
Generative Al Lab
Department of Computer Science and Engineering University of Connecticut Storrs, CT USA [dongj joongjin.oo song, yushan, jiang) @uconn, edu
College Computing and Data Science Nanyang Technological University Singapore 639798 xikun w eg
Yixin Chen
Dacheng Tao
Washington University in St. Louis St. Louis, MO
Generative Al Lab
College of Computing and Data Science Nanyang Technological University Singapore 639798 dacheng t com
USA chenècse. wustl. edu
ABSTRACT
Observation-base trajectory prediction for systems with unknown dynamics is essential in fields such as physics and biology. Most existing approaches are limited to learning within a single system with fixed dynamics patterns. However, many real-world applications require learning across systems with evolving dynamics patterns, a challenge that has been largely overlooked. To address this, we systemat- ically investigate the problem of Continual Dynamics Learning (CDL.), examining task configurations and evaluating the applicability of existing techniques, while identifying key challenges. In response, we propose the Mode-switching Graph ODE (MS-GODE model, which integrates the strengths LG-ODE and sub-network learning with a mode-switching module, enabling efficient learning over varying dynamics. Moreover, we construct novel benchmark of biological dynamic sys- tems for CDL, Bio-CDL, featuring diverse systems with disparate dynamics and significantly enriching the research field of machine learning for dynamic systems. Our code available at fttttts:////o //ss:///iit htt///gttttt githubb..... hubb com QueuQ/uu-
INTRODUCTION
Scientific research often in- volves systems composed of interacting objects, such as multi-body systems in physics and cellular systems in biology, with their evolution governed by underlying dynamic rules. However, due to potentially un- known incomplete dynamic rules or incomplete observa- tions, deriving explicit equa- tions to simulate system evolu- tion can be extremely challeng- ing. As result, data-driven approaches based on machine learning have emerged as promising solution for predict- ing the future trajectories of system states purely from ob- servational data. For instance, the Interaction Network (IN) model (Battaglia al.
Cytoplasm
Exportin: The cargo to be translocated the membrane.
Translecation Dissadiation
RAN: type of protein that facilitates the translocation cargo.
Exportin
RAN
RAN*: RAN protein activated with the assistance RCC1.
Nuclear Membrone
RAN
Exportin
type of protein activates RAN through phosphorylation.
Cytoplasm: substance cell the cell membrane and the nucleus.
YAM RCC1
MANAAY
Nuclear Membrane: The layer that encloses the nucleus cell
RAN
RAN
RAN Activation
Nucleus: The structure within a cell contalining
information
Figure 1: Illustration of the key components of biological cellular system studied in our work: the RAN-regulated nucleocytoplasmic transport (Moore 2013). Briefly speaking, this model depicts the translocation of cargo proteins (Exportin) via nuclear pores with the assistance of RAN proteins. RAN is first activated (RAN*) and then binds to cargo forming complex. Next, the complex translocated across the nuclear membrane into the cytoplasm with assistance RAN. Finally, RAN Exportin are dissociated.
2016) explicitly

---

Published as a conference paper at ICLR 2025
learns interactions between pairs of objects and has demonstrated superior performance in simulated physics systems, showcasing the potential of machine learning in studying physical system dynamics. IN has inspired many subsequent works (Kipf et al., 2018 Sunchez-Gonzaler et al. 2019 Huang et al. 2022 Liu et al. 2024). Later, to enable the modeling of incomplete and temporarily irregular system observations, ODE-based models (Huang et al. 2020 2021 were proposed to learn the continuous dynamics of the systems.
Despite the success of these methods, existing approaches are often limited to learning within single system with a fixed type of dynamics. However, in many real-world scenarios, system dynamics are subject to change over time. For example, in cellular systems (Figure [ the dynamics of a set of variables are subject to change when the kinetic factors are altered. Besides, as a widely adopted task in the field, predicting the trajectories of n-body physical systems (Huang et al. 2020 Battaglia et al. (016)) may also require a model to learn over systems governed by different dynamics factors including interaction types (e.g., elastic force or electrostatic force) and interaction strengths (e.g., different amounts of charges charged objects), as illustrated in Figure In this work, we term these learning scenarios as continual dynamics learning (CDL), and for the first time formally formulate the setting of CDL. In CDL, continually learning from systems with varying dynamics may overwrite a model's 's knowledge encoded in the model weights and trigger the catastrophic forgetting problem. In other words, the model may only accurately predict the most recently observed system dynamics while failing on earlier systems. This phenomenon is empirically verified and reported in Section 4.6 Moreover, many real-world systems exhibit repeated dynamics, and mitigating the forgetting is actually crucial in broad range of scenarios. For example, dynamics of many physics systems are controlled by environmental factors, c.g. temperatures (Huang al. 2023). The values of these factors like temperatures typically oscillate within a certain range, therefore the dynamies of the systems will also repeat. This also true in biological systems. Moreover, biological cellular systems also go through different phases of cell cycle. Catastrophic forgetting phenomenon has also been observed in other fields like computer vision (Van de Ven & Tolaas 2019 Wang et al., 2024) and graph learning (Zhang et al., 2024b), and different approaches have been proposed to facilitate the models in the continual learning setting. However, as demonstrated in our experiments (Section 4.3). most existing continual techniques, which are based on regularization or memory-replay (Yoon) et al. 2017; Kirkpatrick et al. 2017) Lopez-Pazz & Ranzato 2017). are designed to accommodate the patterns of different tasks] within one set of model weights and fail to effectively alleviate the forgetting issue in the context of dynamics learning, especially when the consecutive systems contain different number of objects and exhibit significantly different dynamics patterns.
Targeting the challenge, we turn to the parameter-isolation based continual learning techniques, and propose a novel mode-switching graph ODE (MS-GODE) model, which can continually learn over varying system dynamics and automatically switch to the optimal mode during the test stage. MS-GODE consists of three major components: a prediction network serving as the backbone, a sub-network learning module for encoding the observed dynamics into masks, and mode-switching module for switching the sub-network mode based on the observation. To support irregular and incomplete observational data in the practical scenario studied in our work, our backbone network is built based on LG-ODE model (Huang et al. (020), which has a Variational AutoEncoder (VAE) (Kingma & Welling 2013) structure facilitated by ODE-based prediction (Rubanova et al.) 2019). The basic workflow of MAS-GOOE is as follow: Given the observational data, an encoder network first encodes the data into latent states, upon which ODE-based generator predicts the future system trajectories within the latent space. Finally, a decoder network maps the predicted latent states back into the data space. Upon this framework, we adopt the sub-network learning strategy (Wortsman et al. 2020; Ramanujan et al. 2020; Zhou et al. 2019), which fixes the model weights after initialization and optimize a unique binary mask over the parameters for each system during training. In this way, unlike standard training strategy that encodes data patterns solely in one set of model weights, MS-GODE encodes different types of dynamics in different sub-networks by the collaboration between the binary masks and the fixed-weight backbone network. In the test stage, the model will be switched to the optimal mode by the switching module via selecting the the most suitable mask that can most accurately reconstruct the given observation. Moreover, targeting that the existing entropy-basee mask selection technique is only applicable to classification tasks, we further develop the novel observation reconstruction based mask selection strategy in the mode-switching
In our work, a task refers to system with specific dynamics patterns. While in other fields, egg. Computer Vision, a task could refer certain categories of images.
2

---

Published as a conference paper at ICLR 2025
module. In this way, despite the significant difference between consecutive systems, catastrophic forgetting is avoided.
Besides innovatively formulating the CDL setting and the technical contribution of an effective model in CDL scenario, we have also created a novel dynamic system benchmark, Bio-CDL consisting of
biological cellular systems based on the VCell platform (Schaff et al.. 1997: Cowan et al. 2012 Blinov et al.] 2017). Compared to the widely adopted simulated physics systems, cellular systems contain heterogeneous objects and interactions, offering richer and more challenging patterns of system dynamics. Therefore, Bio-CDL will significantly enhance the research of machine learning- based system dynamics prediction. In experiments, we thoroughly investigate the influence of the system sequence configuration on model performance in CDL with both the widely adopted physics systems and our newly constructed BioCDL., which demonstrates the advantage of MS- over existing state-of-the-art techniques.
2 RELATED WORKS
2.1 LEARNING BASED DYNAMIC SYSTEM PREDICTION
In recent years, graph neural networks (GNNs) have been proven to be promising in modeling and predicting the
Gas
complex evolution of systems consisting of interacting
Depesstion
Veporization
objects (Battaglia et al.
2016 Kipf et al. 2018 Sanchez.
Sublimation
Condensation
Gonzalez et al. [019 H fuang et |l.]| [202]]
Liu et al. [024]
Luo al. 2224bb Zhang et al. 2019;] Luo et al.
2024a] Wu
Freestng
et al.
2021 Luo et al. 2025
Jiang et al.
2024
Guo et al.
Melting
2024
Zhang al. 2020b; Shen al.
2024 Yin et al.
Solid
Liquld
2024
Chen et al.)
2025). This was firstly demonstrated
by (Battaglia et al. (016)) with Interaction Network (IN),
Figure 2: A molecule system may enter dif- ferent phases and exhibit different dynamics environmental factors (e.g. temperature) change. Molecules in solid state can only vi- brate at fixed locations because of the strong interaction between them. Upon entering the liquid state, the interaction strength de- creases and molecules can move around. In gas state, molccules move more freely with little molccule wise interaction.
which iteratively infers the effects of the pair-wise inter- actions within a system and predicts the changes of the system states. Following IN, (Kipf et al. 2018) proposed neural relational inference (NRI) to predict systems con- sisting of objects with unknown relationships. (Mrowca jet al. 2018) proposed the Hierarchical Relation Network (HRN) that extends the predictions to systems consisting of deformable objects. (Sanchez-Ciozzlle
et al. 2019)
proposed the Hamiltonian ODE graph network (HOGN), which injects Hamiltonian mechanics into the model as a physically informed inductive bias. Later, to better consider the intrinsic symmetry of the target systems, GNNs with different invariance and equivariance are proposed (Satorras et al.)
2021 Huang et al.) 2022 Han et al., 2022. Brandstetter et al. 2021 Wu et al. 2023 Liu et al.
2024). To better capture the complex system interactions, High-order graph ODE (HOPE) Luo et al.
2023) innovatively incorporates information from high-order spatial neighborhood and high-order derivatives into dynamical system modeling. Considering that the observation of real-world systems may be incomplete and irregular samples, (Huang et al.) 2020) proposed LG-ODE, which is capa- ble of generating continuous system dynamics based on the latent ordinary differential equations. Later, Coupled Graph ODE (CG-ODE) (Huang et al., 2021) was proposed to apply ODE-based modeling to both node features and interactions. Similar ideas are also adopted in other time series research (Rubanova et al. 2019). By separating the commonalities intrinsic to the systems and the environmental factors causing the dynamics shift, Generalized Graph Ordinary Differential Equations (GG-ODE) (Huang al. 2023) improves the generalization across systems different environments. Similarly, Prototypical Graph ODE (PGODE) (Luo et al.)) disentangles object states and system states to independently model their influence and improve the generalization capability. Disentangled Intervention- betdee--nondnnddd Dynamic graph Attention networks (DIDA) (Zhang et al., 2022c) disentangles the invariant and variant patterns dynamic graphs, and leverages the invariant patterns to ensure a stable prediction performance under spatio-temporal distribution shift. Context-attended Graph ODE (CARE) (Luo et al., 2024c) models the continuously varying environmental factors with a context variable, which is leveraged to better predict the system evolution with temporal environmental variation. Online Relational Inference (ORI) (Kang et al. 2024) models the relationship as trainable parameters, which is accompanied by AdaRelation for the online setting. Despite the substantial contributions these methods have made to dynamic system prediction, they have been limited to learning single system with fixed dynamics. As one of the two major components of our MS-GODE,
3

---

Published as a conference paper at ICLR 2025
the backbone model for dynamics system prediction is mainly built upon the LG-ODE framework (Huang et al.] (020). Different from the other approaches that are typically limited to data with regular intervals or complete observation at each time stamp, LG-ODE supports irregular and incomplete observations, which is essential to the applications studied in our work.
2.2 CONTINUAL LEARNING & MASKED NETWORKS
Existing continual learning methods can be categorized into three types (Parisi al. 2019 Van de
Ven & Tolias 2019
De Lange et al. [2021] (Zhang et al.) [222a;] Konishi et al., 2023 Zhang et al.
2024a 2023b;
2022b Fionda et al. 2023). Regularization-base based methods slow down the adaption of
important model parameters via regularization terms, so that the forgetting problem is alleviated (Wu
al. 2024 Goswami et al. (023). For example, Elastic Weight Consolidation (EWC) (Kirkpatrick
al. 2017) and Memory Aware Synapses (MAS) (Aljundi et al. | (0118) estimate the importance of
the model parameters to the learned tasks, and add penalty terms to slow down the update rate of the parameters that are important to the previously learned tasks. Second, experience replay- based methods replay the representative data stored from previous tasks to the model when learning new
tasks to prevent forgetting (Liang & Li 2023 Rolnick al.. 2019 Rebuffi et al. 2020). For example, Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato 2017
2017
Prabhu et al.)
leverages the
gradients computed based on the buffered data modify the gradients for learning the current task and avoid the negative interference between learning different tasks. Finally, parameter isolation- methods gradually introduce new parameters the model for new tasks to prevent the parameters that are important to previous tasks (Qiao et al. 2023] Yoon et al. 2017). For example, Progressive Neural Network (PNN) (Rusu et al., 2016) allocates new network branches for new tasks, such that the learning on new tasks does not modify the parameters encoding knowledge of the old tasks. Our proposed MS-GODE also belongs to the parameter isolation-basee methods, and is related to subnetwork-besd -ased ones (Wortsman et al. 2020 Kang et al., 2022 Zhou et al., 2019) and the adge-popup algorithm (Ramanujan et 2022). SupSup studies continual learning for classification tasks with output entropy- based mask selection, which is not applicable our task. Edge-popup a algorithm provides a simple yet efficient strategy to select a sub-network, and is adopted by us to optimize the binary masks over the model parameters.
3 LEARNING SYSTEM DYNAMICS WITHOUT FORGETTING
3.1 PRELIMINARIES
In CDL, a model is required to sequentially learn on multiple systems. A system is composed of multiple interacting objects, and is naturally represented as a graph 9 = EV. (V is the node set denoting the objects of the system, and E is the edge set containing the information of the relation and interaction between the objects. Based on E. the spatial neighbors of a node v is defined as (v) = {uleu,r E EJ. Each object node is accompanied by observational data containing the observed states at certain time steps X. = {x1 lt T.). i.e. the trajectory of the system evolution. With a system structured as graph, its trajectory is naturally spatial-temporal graph, in which each node is an observed state x In the following, we will refer to x as a 'state', In a multi-body system, the trajectory records the 3D locations of the particles over time. While in other systems, e,g. cellular systems, a state could be the amount of a certain substance instead of locations, and a trajectory records the states at different time stamps. The set To contains the time steps (real numbers) when the states of u are observed and can vary across different objects. For the prediction task, the observations lie within a certain period, i.e. UVev Tr Elto, 4, and the task is predict the system states at future time steps beyond t1. We denote the future time steps to predict for an object v as Tyree, In CDL, different systems in a sequence may exhibit different dynamics and contain different objects (i.e. V).
3.2 FRAMEWORK WORK OVERVIEW
In this subsection, we provide a high-level introduction to the workflow of MS-GODE (Figure while the details of each component provided in the following subsections.
The backbone network; 2) The sub- network learning module; 3) The mode switching module. The backbone network consists of: I) an encoder network Enc(;; 0g) parameterized by the parameters 8E for encoding the trajectories into the latent space: 2) An ODE-based generator Gen(:; 0c) parameterized by 9g for predicting the future trajectories within the latent space; 3) A decoder network Dec(:;0p) parameterized by
Overall, MS-GODE consists of three core components: 1)
Op for transforming the predicted latent states back into the data space. Within standard learning
4

---

Published as a conference paper at ICLR 2025
Simulated Prediction
Training Inference
1 Feed partial observations to model
Mask Pool M
Parii Aiserrvitions
3 Inttaliize
( Generate the past predictions
switching based on objective
Encoder Generator
M gas ELBOO.M)
Decoder
System 12
Feed all observation to model
only)
System!
ELBO
Generats the future prediction
Objective Function
Mask Inference
Future Prediction
Future Prediction
L
Colculate function
Encoder& Generator Decoder
Masked Decoder
Mp
Mask Pool
Mc
M
Masked Interaction
Network
Masked Decoder
Observations
ODE Salver
[x
Prediction
Encoder
Figure 3: The upper part illustrates the workflow MS-GODE during mask selection and inference, which are denoted by dashed and solid lines. During mask selection, the observation is split into two parts, and the first part is fed into the model for selecting the mask that can best predict the second part. During inference, the entire observation is fed into the model for prediction of future states. The lower part demonstrates the structure of the masked encoder, masked generator, and masked decoder. Different components fetch their corresponding mask from the mask pool and apply the mask onto the parameters.
scheme, the model will be trained by updating the parameters (0g, fg, 0p). However, introduced above, when continually training the model on multiple systems with different dynamics, this learning scheme will bias the model towards the most recently observed system, causing the catastrophic forgetting problem. In this work, inspired by the recent advances in sub-network learning (Ramanujan al., 2020; Wortsman [0g, do. pD.. Instead, we train the model by optimizing the connection topology of the backbone model and encode the dynamics of each system into a sub-network. This is equivalent encoding each system-specific dynamics pattern into a binary mask overlaying the shared parameters with fixed values. In this way, the interference between learning on different systems with different dynamies, i.e. the forgetting problem, can be avoided. Moreover, this approach is also memory efficient since the space for storing the binary masks negligible. With the system-specifer masks, the three model components are formulated as:
al., 2020), we propose to avoid direct optimization of the parameters
Enc(; O Mg);;
Gen(:; @ Ma); Dec(:;0p c M²))
where the superscript is the system index. The details of the binary mask optimization during training and mask selection during testing are provided in Section 335| In the following subsections, all parameters are subsets of fgg 6g, or dp and are controlled by the corresponding masks. For example, W mag in Section 3.3 is part of the encoder parameters 0F and under the control of Mi.
3.3 MASKED ENCODER NETWORK
As the first component of the model, the masked encoder network serves to encode the dynamics pattern in the observational data into the latent space. As introduced in Section 3.1 the trajectory of a system is a spatial-temport graph. Therefore, the encoder is constructed as an attention- based spatial-temporal graph neural network framework (ST-GNN) (Huang et al.) 2020) Hu et al. 2022 Huang et al. 2021 Zhang et al., 2020a).. Originally, the graph attention network (GAT) (Velicković et al. 2017) 18 designed to aggregate information over the spatially neighboring nodes. On spatial- temporal graphs, the information aggregation is extended to both spatial and temporal neighboring nodes. Such spatial-temporal neighborhood of state x is defined the states of the spatially connected nodes within a specified temporal window Swindow
(xx()()) leu, and 1g t) < Swindoww.
(2)
5

---

Published as a conference paper at ICLR 2025
Then, iterative message passing is conducted over the spatial-temporal edges to update the represen- tation of each state. The update of the hidden representation of a state x, at the I-the layer of the ST-GNN is formulated as,
by = h +o(
Σ =
a(h .hig t) Wasg msg -t),
(3)
where -(::-) calculates the attention score between a pair of states, and msg(..)) is the message function commonly adopted in GNNs (Gilmer et al.) (017). However, different from the message function in typical GNNs, the temporal relationship between the states is crucial part understand the system dynamics. Therefore, we follow the strategy in LG-ODE (Huang et al. 2020) to incorporate the temporal distance between the states in the message function and attention function,
msg (s .9 t)) a ( tmp concat ( , -2t))) TE(q - t),
(4)
exp(msg(h!_) 9 t))² h²,')
a a(bb, b : t) E Le EN.. exp (h p t))² h,,")"
(5)
where TE(-) is temporal position encoding developed based on the position encoding in Transformer (Vaswani et al. 2017) for incorporating the temporal information into the representation. Finally, for subsequent state prediction, the state representations are averaged over the temporal dimension,
h Fimat
Σ oh? msg(h!.)) msg (h t to), LE.
Te
where to denotes the starting time of all states in the observational data (Section 3.1 and the average term h, of each node is a weighted summation over the representations at all time steps,
h = o (Tl
Σ TET
msg(h!, - to) Waog).
3.4 MASKED ODE-BASED GENERATOR
ODE-based generator (Huang al. 2020; Chen et al., 2018; Rubanova al., 2019) ensures that the model can handle observations with irregular temporal intervals and incomplete states, as well as predict future states at any time denoted by real numbers. Specifically, the trajectory prediction is formulated as solving an ODE initial value problem (IVP), where the initial values of the objects ([= V)) are generated from the final representation of the states (ini | V), Section 3.3). Mathematically, the procedure of predicting the future trajectory of system is formulated as,
2 p(2 (, vev V
(8)
{ e V,r Tgred)= = Gen((z w V). (Tgredd E V}:0G O OM)),
(9)
To estimate the posterior distribution ((<=1)l E V)I(x./v E VJ) based on the observation (i.e. {X,|v V)), the distribution is assumed to be Gaussian. Then the mean μt and standard deviation oy are generated from (hiimall IV e V} with a multi-layer perceptron (MLP),
a(=1) 1%./o= E VJ) (V a.) (mmlphh)mmati Oa @ M V.
(10)
As noted in Section 3.2 the parameters of mlp(-) and Fine(.) are part of 0g and controlled by M
Based on the approximate posterior distribution g((2)) |o E VJI[,, / e VJ), we sample an initial state for each object, upon which the ODE solver will be applied for generating the predicted states in the latent space. The dynamics of each object in the system are governed by its interaction with all the other objects. Therefore, the core part of the ODE-based generator is trainable interaction network that encodes the dynamics in the form of the derivative of each Z..
dz
dt |-- Fine( N.()));; o Mc).
(11)
where the function Fine (-) parameterized by 0 predicts the dynamics (derivative) of each object based on all the other objects interacting with v (i.e. (u defined in Section and denotes any possible future time. Note that (v) only contains the spatial neighbors and is different from

---

Published as a conference paper at ICLR 2025
Na because the object-wiss interaction a certain time t' is not dependent system states other times. For example, in a charged particle system governed by electrostatic force, the force between pair of partieles at t' is solely determined by the relative positions (i.e. the states) of the particles (Figure at 1'. In our work, we adopt the Neural Relational Inference (NRI) (Kipf et al. 2018) as the function F Fint()) Based on Fine(:): the future state of the system at arbitrary future time t2 can be obtained via an integration
J,, b dt
12 dz
122 dt = 2 + 1.¹ Fine((22)) E ENN(u)); @ Mc)dt,
+
(12)
which can be solved numerically by mature ODE solvers, e.g. Runge-Kutta method. After obtaining the latent representations of the states at the future time steps ((z) E future; V E VV), the predictions are generated via a masked decoder network that projects the latent representations back into the data space
y = Dec(zi; MD), I T future, U V.
(13)
In our work, Dec(:;Op) is instantiated as multi-layer perceptron (MIP).
3.5 SUB-NETWORK LEARNING
MS-GODE is trained by maximizing the Evidence Lower Bound (ELBO). Denoting the concatenation of the initial latent states ((z]:l E V)) as Z.y. the ELBO is formulated as
ELBO(M*) = Ezz w (2x [og(((( (x E T fatures v E VV))
(14)
[L 2 (X. |u E ()))llp((?)))
(15)
where p(Z))) denotes the prior distribution of Z which is typically chosen standard Gaus- sian. M denotes the union of all masks over different modules of the framework (i.e. M = (M Mj, M..). In model, the parameters (0g,0,, and 0p) will be fixed, and the ELBO will be maximized by optimizing the binary masks M overlaying the parameters via the Edge-popup algorithm (Ramanujan et al., (020). After learning each system, the obtained mask added into mask pool M to be used in testing.
3.6 MODE-SWITCHING
During testing, MS-GODE will be evaluated on sequence of systems exhibiting diverse dynamics patterns, and will automatically switch to the optimal mode that ensures the highest accuracy. This is achieved by applying the most suitable mask based on the performance of reconstructing part of the given observations. To obtain the most suitable mask, a given observation from [to.t]] | first split it into two periods [4o, 2 J and [ J. Then, the first half is fed into the model and the correct mask can be chosen by selecting the one that can reconstruct the second half with the lowest error.
4 EXPERIMENTS
In this section, we aim to answer the following questions. 1. How to properly configure MS-GODE for optimal performance? 2. How would the configuration of the system sequence influence the performance? 3. How is the performance of the existing CL techniques? 4. Can MS-GODE outperform the baselines?
4.1 EXPERIMENTAL SYSTEMS
In experiments, we adopt physics and biological cellular systems. A detailed introduction to the system sequence construction is provided in Appendix
Simulated physies systems are commonly adopted to evaluate the machine learning models in the
task of learning system dynamics (Liu al. 2024 Battaglia et al. [0166] Huang et al. (020).. The physics systems adopted in this work include spring-comen connected particles and charged particles (Figure
7). We carefully adjust the system configuration and construct system sequences with different
levels of dynamics shift (Appendix A.
Biological cellular systems are innovatively introduced in this work based Virtual Cell plat- form (Schaff et al. 1997; Cowan et al.) (012): Blinov et al. (007)). Currently, Bio-CDL includes rule- based model of EGFR and compartmental rule based model of translocation based the Ran protein
7

---

Published as a conference paper at ICLR 2025
0.40
2.0
1.5
top-k
0:35
0.40
0.35
0.30
0.25
1.0
0.30
top-k
0.30
top-k
0.20
top-k
0.25
fast
0.25
fast
0.15
fast
0.20
0.20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.6 0.7
top-k ratio
top-k ratio (b)
top-k ratio
top-k
(a)
(c)
(d)
Figure 4: Performance comparison among different strategies to binarize the mask values. (a) Comparison over the cellular system sequence EGFR,, - Ranı EGFR2 - Ranz (b)c))d) Comparison over different physics system sequences. Blue line denotes the performance of top-k selection with different thresholds. Red line demonstrates the performance of using fast selection.
(GTPase). experiments, we adjust the coefficients of the models to construct sequence containing 2 EGFR and 2 Ran systems interleaved with each other (EGFR; - Ran, - EGFR,, - Ranz). Full details are provided in Appendix A.1.2
4.2 EXPERIMENTAL SETUPS EVALUATION
Model evaluation in CDL. The models in this work learn sequentially on multiple systems under the continual learning setting, thus the evaluation is significantly different from standard learning settings. After learning each new task, the model is tested on all learned tasks and the results form performance matrix M TR. N where MP, denotes the performance on the j-system after learning from the 1-st to the i-th system, and N is the number of systems in the sequence. In our experiments, each entry of MP a mean square error (MSE) evaluating the performance on a single system. To evaluate the performance over all systems, average performance (AP) can be calculated. For example,
Σ My is the average performance after learning the entire sequence with N tasks. Similarly,
ΣN M My, More details on model evaluation can N-1
average forgetting (AF) can be calculated as
be found in Appendix A.4 The baseline are configured by combining existing continual learning techniques with the LG-ODE (Huang et al. 2020) model. Other dynamics learning models were excluded because they typically don't it support the practical setting with irregular and incomplete observation studied in this work. All experiments are repeated times on a Nvidia Titan Xp GPU, The results are reported with average and standard deviations.
Baselines model settings. We adopt state-of tfite-of-thees -hettrr baselines including the performance upper (joint training) and lower bounds (fine-tune) in experiments. The state-of-the-an baselines adopted in this work include Elastic Weight Consolidation (EWC) (Kirkpatrick et al. (017)) based on regu- larization, Learning without Forgetting (LwF) (Li & Hoiem| 2017) based on knowledge distillation,
Gradient Episodic Memory (GEM) (Lopez-Paz Ranzato (017)) based on both memory replay and regularization, that integrates the idea of data sampling bias correction into memory replay, and Scheduled Data Prior (SDP) (Koh et al.) (023) that adopts data-driven approach to balance the contribution of past and current data. Besides, joint training and fine-tuning are commonly adopted in continual learning works. Joint training trains the model jointly over all systems, which does not follow the continual learning setting. Fine-tune directly trains the model incrementally on new systems without any continual learning technique. As revealed by Ramanujan et al. (2020). for learning subnetworks, pre-trained models are the most suitable for serving as the backbone. However, unlike the Computer Vision (CV) tasks studied by Ramanujan et all (2020), pre-trained models are not readily available in the context of dynamics system modeling. For example, [Seffner et al. (2024) provides promising pre-trained model with valuable insights into pre-training for dynamical systems, but the model will investigate how to adapt the provided model to extrapolation tasks once their code and model are released. Therefore, we adopt the random initialization strategy, which is shown by Ramanujan al. (2020) be less effective but could be comparable to pre-trained models with a proper ratio of remaining parameters, which thoroughly investigated in our experiments reported in Section 4.3 (Figure 4). More details of experimental settings and baselines and experimental settings are provided in Appendix A. A.3
Caas-Correetion (Chrysakis & Moens 2023) based Memory Replay (BCMR)
still limited to interpolation tasks. However,
4.3 MODEL CONFIGURATION AND PERFORMANCE (RQ1)
8

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparisons on physics system sequences (L lower means better).
So Low-level dynamics shift
Sog2: Mid-level dynamics shift
Soq3: High-lovel dynamics shift
Method
AP
AF
AP
AF 7%
AP
AF 1%
Fine tune
0.369:10.027
0.115
0,391:1:.044
0.314:10.00
0.258:10.025
0.086:10.037
BWC2011
0.208.10.015
0.007.10.019
0.227110033
0.008.10.022
0.148.10.011
0.00810.017
GEM 2017
0.251400087
0.079=
0.379=40.023
0.302=000000
0. 0:3--0000377
-0.091 #0.033
LwF 2017
0.258+0.011
0.104=0.042
0,363-40.099
0.312=00.042
0.130-0..025
0.016+000022
BCM
2013
0.284=:0.017
-0.006=10.031
0.298:20.028
0.233--0.017 -0.001+0.0333
0.027:+0.023
SDP
2023
0.352#0.021
0.121
0,303=:00026
0.352=00008
0.213=00005
0.024:000045
Joint
0.194000006
0.186:10.015
| 0.116::0.009
Ours
0.200-1:0003
0.002=00.044
0.204:200055
+0.001
0.113:+0.001
-0.000=000000
When optimizing the system-specific masks using the edge-popup algorithm (Ramanujan et al. (0200) (Appendix A.2), each entry of the mask assigned with continuous value
0.8
0.6
<
0.4
for gradient descent after backpropagation. During inference, different strategies can be adopted to transform the continuous scores
0.4 0.5
0.3
dropout ratio (a)
dropout ratio (b)
into binary values. In our experiments, we tested both 'fast selection' and 'top-k selec- tion' with different thresholds. 'Fast selec- tion' sets all entries with positive score values into '1's and the other entries into 'O's. "Top-k selection' first ranks the score values and sets a specified ratio of entries with the largest values into 'I's. In Figure show the performance of different strategies. Overall, 'top-k selection' is inferior to 'fast selection'. : This is potentially because 'fast selection' does not limit the number of selected entries, therefore allowing more flexibility for optimization. We also observe that the performance of 'top-k' selection is more sensitive on the cellular systems compared to the physics systems, indicating that the cellular systems have higher optimization difficulty for sub-network (binary mask) learning over system sequences.
Figure 5: AP (a) and AF (b) of MS-GODE with different dropout rate.
Second, sub-network learning will deactivate some neurons in the model, which resembles the dropout mechanism widely adopted in machine learning models and may cause the model to be over-sparsified. Therefore, we investigate the influence of dropout rate in MS-GODE. From Figure 5 can see that smaller dropout rate results in lower error (better performance). This corroborates our hypothesis that the mask selection mechanism and dropout complement each other, and the dropout rate should be decreased when the masking strategy is adopted.
4.4 SEQUENCE CONFIGURATION AND PERFORMANCE (RQ2)
In this subsection, we investigate the influence of system sequence configuration on the learning difficulty and performance. Specifically, we construct 3 physies system sequences with increasing level of dynamics shift (Appendix A.I.T). Sequence 1 (low-level dynamics shift) consists of 8 spring connected partiele systems, in which consccutive systems are only different in one system coefficient. Sequence 2 (mid-level dynamics shift) is constructed to have a higher level of dynamics shift by simultaneously varying 2 system coefficients. Finally, sequence (high-level dynamics shift) is constructed by interleaving spring-connecter particle systems and charged particle systems with disparate dynamics. As shown in Table I. in terms of both AP and AF, most methods, including MS-GODE, obtain similar performance over Sequence and 2, and obtain better performance on Sequence 3. For MS-GODE, since the systems with more diverse dynamics are easier to distinguish for selecting the masks during inference. For all the methods
general, two possible cases are: 1. The model is not well trained on any system (it has not encoded much information), therefore has nothing to forget. 2. The model is well trained and can maintain enough information from each system. However, the lower AP (low MSE) obtained on the system sequence with high-level dynamics shift indicates that the model has well adapted to each system, i,c, case not true. When case 2 holds, it indicates that the model's parameters are sufficiently updated to fit each system. When learning on one system, two possible cases are: 1. All parameters are modified uniformly. 2. A subset of parameters parameters are uniformly modified, it would be almost impossible to maintain the performance on previous systems. Therefore, the potential case is that each system relies more a subset of the
modified more than the others. When the baseline is Fine-tune, if all the

---

Published as a conference paper at ICLR 2025
task
task
task
task
(a)
(b)
task
task (f)
task
task (h)
(c)
(d)
(e)
(g)
Figure 6: Visualization of performance matrices. (a), (b), (c), (d) corresponds to MS-GODE, fine-tune, EWC,
and LwF on the cellular system sequence. (e), (f), (g). (h) the performance of MS-GODE, EWC, fine-tune, and GEM, on the physics systems. Lower values indicate better performance.
parameters, and systems with larger difference dynamics may lead to less overlap in the subsets of parameters they rely on.
4.5 COMPARISONS WITH STATE-OF-TIIB-ART (RQ3.4)
In this subsection, we compare MS-GODE with state-off tate-oofthhe-rts methods including the joint training. The experiments conducted both physics systems (Table T and cellular systems (Table 2 Besides, by comparing the results across different sequences in the two tables, we find that the sequences with gradual dynamics shift (Sequence 1,2 of physics systems) are more difficult learn than the sequences with abrupt dynamics shift (Sequence 3 of the physics sys- tems and the cellular system). advantages of MS-MSDDD over the baselines mainly come from two perspectives. First, since the learning of the subnetworks are learned in completely independent manner, the plasticity when learning new systems is guaranteed. Second, the indepen- dency of the subnetworks also eliminates the forgetting issue and protect the performance stability previously learned systems.
Table 2: Performance comparisons on biological cellular systems (1 lower means better).
EGFR, + Ran, + EGFR, + Ranz
Method
AP 4
AF L
Fine tume
1.0099
0,22610.037
EWC 2017
0.3121:0.028
4.013:10.019
GEM 2017
0.316:+00.08 0.003
0.35220.100
LwF
0.330:10.036
0.349=0.046
BCMR 2023
0.149:10.025
0.013:100044
0.197.1..025
Joint
0.055:1-00.00
Ours
0.144::.012
-0.003=000066
4.6 IN-DEPTH INVESTIGATION ON THE LEARNING DYNAMICS (RQ3,4)
Table I and provide the overall performance. However, as introduced in Section to obtain an in-depth understanding of the performance of different methods, we have to seek help from the performance matrix. In Figure 6 we visualize the performance matrices of different methods. The i-th column demonstrates the performance of the i-th task when learning sequentially over the systems. Comparing MS-GODA ((a) and (e)) and the other methods, we find that MS-GODE maintains a much more stable performance. EWC ((c) and (f)) also maintains a relatively stable performance of each system based on its regularization strategy. However, compared MS-GODE, the model becomes less and less adaptive to new systems, because the regularization is applied to more parameters when proceeding each new task. Fine-tune ((b) and (g)) is not limited by regularization, therefore is more adaptive on new systems but less capable of preserving the performance on previous systems compared to EWC. LwF (d), although based on knowledge distillation, does directly limit the adaptation of the parameters. Finally, based on memory and gradient modification. GEM (h) maintains the performance better than fine-tune (g), and more adaptive new tasks than EWC (f). More details on the performance matrices are provided in Appendix A.5 A.5
CONCLUSION
In this paper, we systematically study the problem of continual dynamics learning (CDL) from different perspectives, including investigating the influence of task configuration and evaluating the performance of existing continual learning techniques. Based on the findings, we propose Mode-switching Graph ODE (MS-GODE). Additionally, we also construct Bio-CDL, consisting of biological cellular systems and significantly enriching the research field of machine learning on conduct comprehensive experiments
system dynamics. Finally, system sequences, which not only demonstrate the effectiveness of MS OS-GODE, but also provide insights into the problem of machine leamingg over system sequences with dynamics shift.
both physics and cellular
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project supported by the National Research Foundation, Singapore, under its NRF Professor- ship Award No. NRF-P2024-001.

---

Published as a conference paper at ICLR 2025
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp.
2018.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez. Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
Michael L Blinov, James C Schaff, Dan Vasileseu, Ion I Moraru, Judy E Bloom, and Leslie M Loew. Compartmental and spatial rule-based modeling with virtual cell. Biophysical journal, 1 11(()): 1365-1372, 2017.
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Ge- ometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:21 2021.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pp. 532-547, 2018.
Michael K Chen, Xikun Zhang, and Dacheng Tao. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprini arXiv:2501.1483 2025.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Aristotelis Chrysakis and Marie-Francine Moens. Online bias correction for task-free continual learning. ICLR 2023 at OpenReview, 2023.
Ann E Cowan, Ion I Moraru, James C Schaff, Boris M Slepchenko, and Leslie M Loew. Spatial modeling of cell signaling networks. In Methods in cell biology, volume 110, pp. 195-221. Elsevier, 2012.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, (477):3366-3385, 2021.
Valeria Fionda, Olaf Hartig, Reyhaneh Abdolazimi, Sihem Amer- Yahia, Hongzhi Chen, Xiao Chen, Peng Cui, Jeffrey Dalton, Xin Luna Dong, Lisette Espin- Noboa, et al. Tutorials at the web conference 2023. In Companion Proceedings of the ACM Web Conference 2023, pp. 648-658. 2023.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp. 1263-1272. PMLR, 2017.
Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity class distributions in exemplar-free continual learning. Advances in Neural Information Processing Systems, 36, 2023.
Wei Guo, Hao Wang. Luankang Zhang, Jin Yao Chin, Zhongzhou Liu, Kai Cheng, Qiushi Pan, Yi Quan Lee, Wanqi Xue, Tingjia Shen, et al. Scaling new frontiers: Insights into large recommen- dation models. arXiv preprint Xixv:2412.000711 2024.
Jiaqi Han, Wenbing Huang, Tingyang Xu, and Yu Rong. Equivariant graph hierarchy- based neural networks. Advances in Neural Information Processing Systems, 35:9176-99 79-91877 2022.
Lianyu Hu, Shenglan Liu, and Wei Feng. Spatial temporal graph attention network for skeleton-based action recognition. arxiv preprint arXiv:2208; 08599, 2022.
Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun. and Junzhou Huang. Equivariant graph mechanics networks with constraints. arXiv preprint arXiv:2203. 06444, 2022.
12

---

Published as a conference paper at ICLR 2025
Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly- sampled partial observations. Advances Neural Information Processing Systems, 33:16177- 16187, 2020.
Zijie Huang, Yizhou Sun, and Wei Wang. Coupled graph ode for learning interacting system dynamics. In KDD, pp. 705-715, 2021.
Zijie Huang, Yizhou Sun, and Wei Wang. Generalizing graph ode for learning complex system dy- namics across environments. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 798-809, 2023.
Yushan Jiang. Zijie Pan, Xikun Zhang, Sahil Garg, Anderson Schneider, Yuriy Nevmyvaka, and Dongjin Song. Empowering time series analysis with large language models: A survey. arXiv preprint arXiv:2402.03182, 2024.
Beomseok Kang, Priyabrata Saha, Sudarshan Sharma, Biswadeep Chakraborty, and Saibal Mukhopad- hyay. Online relational inference for evolving multi-agent interacting systems. arXiv preprint arXiv:2411.01442, 2024.
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Josnson Sung Ju Hwang. and Chang D Yoo. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pp. 10734-10750. PMLR, 2022.
Diederik P Kingma and Max Welling. 1311v:1312.61144 2013.
Auto-encoding variational bayes.
arXiv preprint
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Ban Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 (13):3521-3526. 2017.
Hyunseo Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Sculki Park, Jung- Woo Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. The Eleventh International Conference Learning Representations, 2023.
Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter- level soft-masking for continual learning. In International Conference on Machine Learning, pp. 17492 17505. PMLR, 2023.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pallern Analysis and Machine Intelligence, (0(11))2935-2947 2017.
Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning. Advances in Neural Information Processing Systems, 36, 2023.
Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 8653- 8661, 2021.
Yang Liu, Jiashun Cheng, Haihong Zhao, Tingyang Xu, Peilin Zhao Fugee Tsung, Jia Li, and Yu Rong. SEGNO: Generalizing equivariant graph neural networks with physical inductive biases. The Twelfth International Conference on Learning Representations, 2024. URL https: http::////ppeeeeee......m. net forum?id-3oTPsO0.....
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv: 2017.
13

---

Published as a conference paper at ICLR 2025
Rengiang Luo, Huafei Huang, Shuo Yu, Zhuoyang Han, Estrid He, Xiuzhen Zhang, and Feng Xia. FÜGNN: Harmonizing faimess and utility graph neural networks. In Proceedings of the 30th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2024a.
Renqiang Luo, Huafei Huang, Shuo Yu, Xiuzhen Zhang, and Feng Xia. FairGT: fairness-aware graph transformer. In Proceedings of the 33rd International Joint Conference on Artificial Intelli- gence, 2024b.
Renqiang Luo, Huafei Huang. Ivan Lee, Chengpei Xu, Jianzhong Qi, and Xia Feng. Fairgp: A scalable and fair graph transformer using graph partitioning. AAAI, 2025.
Xiao Luo, Yiyang Gu, Huiyu Jiang, Hang Zhou, Jinsheng Huang, Wei Ju. Zhiping Xiao, Ming Zhang, and Yizhou Sun. Pgode: Towards high-quality system dynamics modeling. In Forty-first International Conference Machine Learning.
Xiao Luo, Jingyang Yuan, Zijie Huang, Huiyu Jiang, Yifang Qin, Wei Ju, Ming Zhang, and Yizhou Sun. Hope: High-order graph ode for modeling interacting dynamics. In International Conference on Machine Learning, pp. 23124-23139. PMLR, 2023.
Xiao Luo, Haixin Wang. Zijie Huang, Huiyu Jiang, Abhiject Gangan, Song Jiang, and Yizhou Sun. Care: Modeling interacting dynamics under temporal environmental variation. Advances in Neural Information Processing Systems, 36, 2024c.
Jonathan Moore. In the wrong place at the wrong time: does cyclin mislocalization drive oncogenic transformation? Nature Reviews Cancer, 13(3):201-208, 2013.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems, 31, 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71 71, 2019.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524-540. Springer, 2020.
Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu Yong Peng, Yuan Xie, et al. Prompt gra- dient projection for continual learning. In The Twelfth International Conference on Learning Representations, 2023.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11893-11902, 2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irreuulaly-samppled time series. Advances in neural information processing systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1006::6606604671 2016.
Alvaro Sanchez-Gonzaled Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv: 12790, 2019.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference machine learning, pp. 9323-9332, PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
James Schaff, Charles C Fink, Boris Slepchenko, John H Carson, and Leslie M Loew. A general computational framework for modeling cellular structure and function. Biophysical journal, 73()): 1135-1146, 1997.
Patrick Seifner, Kostadin Cvejoski, Antonia Körner, and Ramsés Sánchez. Foundational inference models for dynamical systems. arXiv preprint ar arXiv:2402.07594 2024.
Tingjia Shen. Hao Wang. Chuhan Wu, Jin Yao Chin, Wei Guo, Yong Liu, Huifeng Guo, Defu Lian, Ruiming Tang, and Enhong Chen. Predictive models in sequential recommendations: Bridging performance laws with data quality insights. arXiv preprint arXiv:2412.00430, 2024.
Gido M Van de Ven and Andreas S Toliia. Three scenarios for continual learning. arXiv preprini UrXivv/904.007340 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint vrrXiv:/706.033 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 109000033. 2017.
Liyun Wang. Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: theory, method and application. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. arXiv preprint XXiv:20006117669 2020.
Liming Wu, Zhichao Hou, Jirui Yuan, Yu Rong, and Wenbing Huang. Equivariant spatio-temporal attentive graph networks to simulate physical dynamics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttp:://ppereet 35nF
net. forum? d=
Xinyu Wu, Ye Yuan, Xikun Zhang, Can Wang, Tiantian Xu, and Dacheng Tao. Gait phase classifica- tion for a lower limb exoskeleton system based on a graph convolutional network model. IEEE Transactions on Industrial Electronics, 69(5):4999 5008, 2021.
Yichen Wu, Long-Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. The Twelfth International Conference Learning Representations, 2024.
Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. Dataset regeneration for sequential recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3954-3965, 2024.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint 1w..::1718.0110000 2017.
Xikun Zhang, Chang Xu, Xinmei Tian, and Dacheng Tao. Graph edge convolutional neural networks for skaleton-based action recognition. IEEE Transactions on Neural Networks and Learning Systems, 31(8):3047-306) 2019.
Xikun Zhang, Chang Xu, and Dacheng Tao. Context aware graph convolution for skeleton-baser action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14333-14342 2020a.
Xikun Zhang. Chang Xu. and Dacheng Tao. On dropping clusters to regularize graph convolutional neural networks. In European Conference on Computer Vision, 2020b.
Xikun Zhang, Dongjin Song, and Dacheng Tao. Cglb: Benchmark tasks for continual graph learning. In Thitty-sxtth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022a.
15

---

Published as a conference paper at ICLR 2025
Xikun Zhang, Dongjin Song, and Dacheng Tao. Sparsified subgraph memory for continual graph representation learning. In 2022 IEEE International Conference on Data Mining (ICDM), pp- 1335-1340. IEEE, 2022b.
Xikun Zhang, Dongjin Song, and Dacheng Tao. Hierarchical prototype networks for continual graph representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4): 4622-4636, 2023a. doi: 10.1009/TPMI.20000
Xikun Zhang, Dongjin Song, and Dacheng Tao. Ricci curvature- graph sparsification for continual graph representation learning. IEEE Transactions on Neural Networks and Learning Systems, 2023b.
Xikun Zhang, Dongjin Song, Yixin Chen, and Dacheng Tao. Topology embedding memory for learning on expanding graphs. 2024a.
Xikun Zhang, Dongjin Song, and Dacheng Tao. Continual learning on graphs: Challenges, solutions, and opportunities. arXiv preprint arXiv:2402.11565, 2024b.
Zeyang Zhang, Xin Wang. Ziwei Zhang, Haoyang Li, Zhou Qin, and Wenwu Zhu. Dynamic graph neural networks under spatio-temporal distribution shift. Advances in neural information processing systems, 35:6074-6089, 2022c.
Fan Zhou and Chengtai Cao. Overcoming catastrophic forgetting in graph neural networks with experience replay. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 4714-4722, 2021.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. Advances in neural information processing systems, 32, 2019.
A APPENDIX/ / SUPPLEMENTAL MATERIAL
A.1 SYSTEM SEQUENCE CONFIGURATION
A.1.1 PHYSICS SYSTEM SEQUENCE
Simulated physics systems are commonly adopted to evaluate the machine learning models in the task of learning system dynamics (Liu et al. 2024 Battaglia et al. 2016 Huang et al,, 2020). The physics systems adopted in this work include spring connected particles and charged particles (Figure (7) with disparate dynamics, therefore are ideal for constructing system sequences to evaluate a model's continual learning capability under severe significant dynamics shift. Besides, the configuration of each system type is also adjustable. For the spring connected particles, the number of particles, strength of the springs, and the size of the box containing the partieles are adjustable. For the charged particles, the number of particles, charge sign, and the size of box are adjustable. In our experiments, we constructed multiple systems with different configurations, which are aligned into sequences for the model to learn.
The physics system sequences are constructed to have different types of dynamics changes. System sequence 1 is composed of 8 spring connected particle systems. Each system contains 5 particles, and some pairs of particles are connected by springs. An illustration is given Figure 7 For each system, besides the number of particles, the size of the box containing the particles and the strength of spring are adjustable. In Sequence I, the first 4 systems have constant spring strength and decreasing box size. From the 5-th system, the box size is fixed, and the spring strength is gradually increased. Sequence also contains 8 systems of spring connected particles and is designed to posses
severe dynamics shift. Specifically, both the box size and spring strength vary from the first to the last system, and the values are randomly aligned instead of monotonically increasing or decreasing. Sequence is designed to posses more significant dynamics shift than Sequence by incorporating the charged particle systems in the sequence. Specifically, Sequence 3 contains 4 spring connected particle systems and 4 charged particle systems, which are aligned alternatively. The box size gradually decreases and the interaction strength (spring strength or amount of charge on the particles) gradually increases. In a charged particle system, the particles could carry either positive or negative charge, and the system dynamics is governed by electrostatic force, which is significantly different from the spring connected particle system.
16

---

Published as a conference paper at ICLR 2025
System
S2
S3
SS
S7
S8
S10
Type
Spring
Spring
Spring
Spring
Spring
Spring
Spring
Spring
Spring
Spring
particles
5
5
5
5
5
5
5
5
5
Box size
10.0
5.0
3.0
1.0
0.5
0.5
0.5
0.5
3.0
1.0
Interaction strength
0.01
0.01
0.01
0.01
0.01
0.1
0.5
1.0
0.1
0.5
Table Spring connected system configurations.
System
C1
C2
C3
C4
Type
Charge
Charge
Charge
Charge
# partieles
5
5
Box size
10.0
3.0
1.0
0.5
Interaction strength
0.01
0.1
0.5
1.0
Table 4: Charged particle system configurations.
Specifically, we list the configurations of the systems in Table 3 and Then the three sequences can be precisely represented as:
System Configurations
Postively Charged
Negatively Charged Particle
Neutral Particle System
-WW- Spring
Particle
System
System
1. Sequence 1: S + S²
S3 + S4
+ Ss + S6 7
S7 Sz
Particle Trajectories
2. Sequence 2: S + Sg
S2 + S7 + Ss + S6 7
S5 S3
3. Sequence 3: S Ci
Sg + C2 S10 C3
Figure 7: Illustration of the continual learning over different physics systems with different dynamics. The factors deter- mining the dynamics shown in this figure include the type and strength of the interactions.
Sg - C4
For each system, the simulation runs for 6,000 steps, and the observation is sampled every 100 steps, resulting in 60-step series. During training, the first 60% part of the trajectory each system is fed to the model to generate prediction for the remaining 40%. For each system sequence, 1,000 sequences are used for training, and another 1,000 sequences are used for testing.
A.1.2 BIOLOGICAL CELLULAR SYSTEM
In this paper, we build a novel benchmark, Bio-CDL, containing biological cellular dynamic systems based Virtual Cell (Schaff al., 1997 Cowan et al. 2012 Blinov al.) 2017) with different system configurations and variable selection. Currently, Bio-CDL is built based on two types of cellular models. The first one is rule-based model of EGFR receptor interaction with adapter proteins Grb2 and Shc. The second is compartmental rule based model of translocation through the nuclear pore of cargo protein based on the Ran protein (GTPase). illustration of the Ran system is provided in Figure 8
Detailed description of these two systems can be found via httpp::/vcel.e org http://cee166.11_leeeeeeeeee https: http:://oealll.ong//ebbbttttt www.lllllooorrlleeeellllllleeel
and Ran httpooort__utooriall.gm pdf Based on these 2 types of models, we construct multiple systems, which are aligned into different system sequences. Details are provided below.
Rule-Based
17

---

Published as a conference paper at ICLR 2025
bill-bbasedeeff- snffr_tutorie
Rule-based Ran ansport
Figure 9
Applications (3)
/at netrork_deten
Geometry
Spacifications
Protecols
Simulations
Parmmeter Estimation
Figure 10
Our new benchmark is constructed based on the EGFR receptor interaction model and Ran transportation model. The length of the observe trajectories ranges from 250 550 time steps for EGFR systems and 40 to 150 time steps for Ran systems.
CDK2
RAN-GDP
Cyclln
CDK1
RAN-GDP
NLS
RAN- GDP
Cyclin NES
BAANGAI
Importin-a
Exportin
RAN-GTP
Importin-8
We provide below the details of data gen- eration, and will release our generated data later. The code for generating the configu- ration files for the simulations is contained
Importin
Noclear Membrane
CDK1 Cyclin B1
CDK2
Cyclin
RAN-GTP
NES
Importin-8
Exportin
MMNAMM ROC1
MAXXXX
VCe VCell_config__eee
Nucieus
RAN-GDP
RAN-GTP
We have restricted the model coefficients to specific range to ensure the simulations remain stable. After generating the con- figuration files, they should be uploaded into the VCell platform for generating the simulations. The specific steps are listed below.
Figure 8: Illustration of the RAN-regulated nucleocy- toplasmic transport (Moore 2013). Briefly speaking, this model depicts the translocation of cargo proteins (Exportin 1) via nuclear pores with the assistance of RAN proteins. RAN-GDP first activated into RAN- GTPP and then binds cargo molecules (Exportin 1) forming a complex containing RAN-GTP and Exportin 1. Next, the complex translocated across the nuclear membrane into the cytoplasm with the assistance of RAN. Finally, RAN and Exportin are dissociated after the translocation.
through Download the VCell client ptttp::/vveel.orrt euu-vcell-softwaree
create a free account and log in. 2. Select the 'BioModel' tab, in the 'Search' box, locate and load the models 'Rule-based_ees _enssedeeff__aee tutorial' based Ran Ran_Rrannaaaaaaaaaa transport' under the 'Tutorials' directory for the EGFR model or Ran model, respectively (Figure 9..
or 'rule-
4. In the upper left part of the window, click the 'Applications' then click the 'Simulations' under the 'network determ' tab (Figure IO).
4. In the main window on the upper right, select the 'Simulations' tab and click the first icon under the 'Simulations' tab create a new simulation (Figure IT.
5. Click to select the created simulation, then click the icon with blue arrow and a gear to load the previously generated simulations. 6. After the simulations are done, select all the simulations and click the icon with green arrow and a gear to export all the simulation results into specified folder. Specify the
data_store_path
18

---

Published as a conference paper at ICLR 2025
Gemeetry
catioon
Protorels
Pereetee Intimation
Stnul inti
Ousorated
b Variable Redoction
Oetput mpti
Status
Canbined
Sinnl enO bat .00 60.0
beep suppl
Cebiined IDA/T1OE
compltted
Sinulati
keap exary tnaple
Coobined TDA/CTI0R
coppltee
Sianletien0_bor
60.
keep
Cabiined
Sinclatiend hat 203
60 0
keep srery cemple
Combined IDA/CT008
remplated
keep ersry sumple
Conbined
complated
Simulati bat, 005
keep comple
Cankined TDA/CT00E
romplated
Stnclationn, Saetiien__ee00000 500_000
60.0
keep arery sumple
Cenbined I1A/C1016
cempleted
Sin 508
60.0
Leep smpll
Cubiined IDA/C100
cumpleted
Sisulati
keep exery comple
Caskised
implated
keep
Cabiine
Sinclatient hat 011
60
leep srery cemple
Cubiise 1DA/CT00
rmplated
_02
keep erary sample Cebbined
comlated
Figure 11
and run
generate_datasett
obtain the data files that can be loaded and used by the MS-GODE model.
This implementation of MS-GODE is based on [Pytorch Geometric httmmttriiattts://giiiia com/ ric API.
Example command for running the experiments with MS-GODE:
python sun_models, --cut_num 20 \
--nepos 20 \
--device
--n_iters_to_vi> 10
--thresholding fast'"
-mask True -dropout_mask '0.0"
-batch-size 10
--mode "ex"
-noomaliieVCellfettt 'universal' \
--system VCell''
--repeats 5
--verwrite_resulee 'False'
--fi__random_seed 'False"
--savereeuutts True
Example scripts for data generation of cellular/physies systems:
python /.tatta/generate_ddtttssttt --s --s --simlaation VCell --num-train 3000
--num-test 3000 \
--n-balls
The generation of cellular simulation data requires first obtaining simulation data from VCell platform, which is introduced in the Appendix A.1.2 of the submission.
python didata/generate_dataaaae
py --simulation simulation
--num-train 3000 \
--num-test 3000 \
--n-balls
19

---

Published as a conference paper at ICLR 2025
Setup
The mdoel implementation is based on the following packages:
[Python 3.6. 10000//ttts://ww.yyyhooooor//
[Pytorch 1.4.f(httpss/pyyoocc..or/)
torch-cluster==115.33- :
[numpy 1.16. 1.16. )
In our experiments, we adjust the parameter configurations of these two types of models to construct multiple systems with different dynamics, which are aligned into a sequence for the experiments. We generate simulated data using the Virtual Cell platform (VCell) and pre-process the data. In experiments, we adopt a 4-system sequence using the simulated data by alternatively align EGFR systems and 2 Ran systems. EGFR system is rule-based model of EGFR receptor interaction with two adapter proteins Grb2 and She. Ran system a compartmental rule based model of translocation through the nuclear pore of cargo protein based the Ran protein (GTPase). For constructing systems with different dynamics, different EGFR and Ran systems are constructed by selecting a subset of the observed variables from the complete systems. This not only enrich the dynamics diversity across different systems in the learning sequence, but also enhance the learning difficulty the systems. The IDs (inherited from VCell platform) of the selected variables in EGFR11 EGFR2, Ranj, Rang are ss15,s16.221 211.1112,222222 s.ss2.222.s33,... $26, s27, s30, s36, s43, s44, s58, s68), J. (Ran C_ nuc, RCCI, s10, s11, s12, s13, s14, s15, s16, s17, C_nuc, s18, s19, s2, $20, $21, s22, s23, s24, s25, s26, s27, s28, s29, s3, s30, s31, s32, s33, s34, s35, s4, s5, s6, s7, s8, ). (s14, s15, s16, s23, s24, s25, s31, s5 ), respectively.
s38, s45, s47, s59, s74, s9, s96), {s24, s25,
Using the code for generating cellular system data requires the following steps.
1. Using the script contained in our provided code MMS-GDDE/VCell _onfil_oocncggelll gen' to generate the configuration files for VCell simulation. Users can freely adjust the configurations the script.
2. Download the VCell application
3. Load the generated configuration files into the VCell platform.
4. Generate simulation results. Instruction of using this script also provided in the ReadMe file of our code.
5. Using the script 'MS-GGDD//attlgeeeeaat patpr//d for processing the simulation results. After this step, the data can used for experiments.
Since the entire dataset much larger than the 100 MB limit, we cannot provide the data in the supplementary materials. But we have provided all code and instruction for generating the data, and will release the complete Bio-CDL benchmark to the public later. Besides the sequence EGFR, + Ranj EGFR2 + Rang adopted in the experiments, the complete Bio-CDL also contain more different system configurations and sequence configurations. Moreover, we are also working enriching Bio-CDL with more diverse system types and system sequences.
For each cellular system, we generate 100 iterations of simulation using VCell. Different from the physics systems, the temporal interval between two time steps of cellular system is not constant. Similar to the physics systems, the first 60% part of the observation of each system simulation is fed into the model to reconstruct the remaining 40% during training. For each system, we use 20 trajectories for training and 20 trajectories for testing.
A.1.3 ADDITIONAL DETAILS OF EXPERIMENTAL SETTINGS
For the encoder network, the number of layers is 2, and the hidden dimension is 64. 1 head is used for the attention mechanism. The interaction network of the generator is configured as I-layer network, and the number of hidden dimensions is 128. Finally, the decoder network is fully-connected
20

---

Published as a conference paper at ICLR 2025
layer. The model is trained for 20 epochs each system in the given sequence. We adopt the AdamW optimizer (Loshchilov & Hutter 2017) and set the learning rate as 0.0005. In our work, task boundaries provided the models during training.
A.1.4 ADDITIONAL DISCUSSION ON PERFORMANCE AND LOW-LEVEL DYNAMICS SIIIF
The performance of MS-GODE is mainly determined on two factors, including the performance on each system and the forgetting issue, Therefore, to further improve the performance under gradual dynamics shift, there are two promising approaches:
Enhancing the performance on each single system. As revealed in Ramanujan et al. (020), the larger the backbone network, the more probable the masked sub-network can reach the capacity of the full backbone network. In other words, increasing the size of the model can improve the performance.
2. Reducing the performance decrease after learning new systems. Since MS-GODE completely separate the masks for different dynamics, the forgetting issue has been eliminated. Therefore, the performance decrease after learning systems mainly comes from the incorrect mask selection. The strategy above to increase model size can help improve the mask selection, because better fitting each system increases the difference between the masks. Furthermore, the mask selection can also be improved by incorporating a mixture of selection criteria. Specifically, in our experiments, the mode-switching module of MS-GODE selects the mask based on the error of reconstructing the second 50% of the observation. This can be enhanced by incorporating different splitting ratios and mixing the result. Since the correct mask tend to exhibit lower error with most splitting ratios, selecting the one that succeeds in more splitting ratios could increase the possibility of finding the correct one.
A.2 EDGE-POPUP ALGORITHM
In this work, adopt the edge-popup algorithm (Ramanujan al. 2020) for optimizing the binary masks. The main idea is to optimize a continuous score value for each entry of the mask during the backpropagation, and binarize the values into discrete binary values during forward propagation (Appendix (A.2). Accordingly, the strategy for binarizing the mask entry values is crucial factor influencing the performance. For convenience of the readers, provide the details about this algorithm in this subsection.
Given fully connected layer, the input to neuron in the /-th layer can formulated weighted summation of the output of the neurons in the previous later.
I
WunFu:
(16)
upt-1
where 1- denotes the nodes in the previous layer and 2, refers to the output of neuron u.
With the edge-popup strategy, the output is reformulated as
Ie
Σ aP--1
Wauzyh(Suu)
(17)
where h (swww) is the binary value over the weight way denoting whether this weight selected and Suo the continuous score used in gradient descent based optimization. During backpropagation, the gradient will ignore h(.)) and goes through it, therefore the gradient over Sao is
OL ŌI aT, Osuv
OL or,
Jo..
Way Fus
(18)
where C denotes the loss function.
During forward propagation, lo(-) can take different options as we mentioned and studied in Section In our experiments [4.3] we tested both 'fast selection' and 'top-k selection' with different thresholds. 'Fast selection' set all entries with positive values into 'I's and the other entries into 'O's. 'Top-k selection' will rank the entry values and set a specified ratio of entries with the largest values into 'I's.
4.3
21

---

Published as a conference paper at ICLR 2025
A.3 BASELINES
1. Fine-tune denotes using the backbone model without any continual learning technique.
2. Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) applies quadratic penalty over the parameters of a model based on their importance maintain its performance on previous tasks.
3. Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) selects and stores representative data in episodic memory buffer. During training, GEM will modify the gradients calculated based on the current task with the gradient calculated based on the stored data avoid updating the model into direction that is detrimental to the performance on previous tasks.
4. Learning without Forgetting (LwF) (Li & Hoiem 2017) is a knowledge distillation based method, which minimizes the discrepancy between the the old model output and the new model output to preserve the knowledge learned from the old tasks.
5. Bias Correction based Memory Replay (BCMR) Chrysakis & Moens 2023). This baseline constructed by integrating the navie memory replay with the data sampling bias correction strategy (Chrysakis & Moens] (223)) In other words, the method does not train the data immediately after observing the data. Instead, stores the observed data into memory buffer. Whenever testing is required, the model will be trained over all buffered data.
6. Scheduled Data Prior (SDP) (Koh al., 2023) considers that the importance of new and old data is dependent on the specific characteristics of the given data, therefore balance the contribution of new and old data based on a data-driven approach. Since the condition assumed for c² in SDP (Koh et al. 2023) does not align with the actual situation in our experiments, the parameter a and B is Eq. (9) of SDP) (Koh et al. (023) are manually selected in experiments.
7. Joint Training (Joint) jointly trains given model on all data instead of following the sequential continual learning setting.
A.4 MODEL EVALUATION
Different from standard learning setting with only one task to learn and evaluate, our setting, the model will continually learn sequence of systems, therefore the setting and evaluation are significantly different. In the model training stage, the model trained over a system sequence. In the testing stage, the model will be tested on all learned tasks. Therefore the model will have multiple performance corresponding to different tasks, and the most thorough evaluation metric is the performance matrix M NxN where Mr,, denotes the performance on the j-system after learning from the 1-st to the i-th system, and N the number of systems in the sequence. In our experiments, each entry of MP is mean square error (MSE) value. To evaluate the overall performance on sequence, the average performance (AP) over all learnt tasks after learning multiple tasks could
be calculated. For example, i.e., Σ) M%r, corresponds to the average model performance after learning the entire sequence with tasks. Similarly, the average forgetting (AF) after N tasks can be formulated as works (Chaudhry et al.
N
ΣN M2 My These metrics are widely adopted in continual learning
2018 Lopez-Paz & Ranzato 20177] Liu et al., 2021 Zhang et al. 2023a
Zhou & Cao 2021). although the names are different in different works.
For convenience, the performance matrix can be visualized as a color map (Figure 6 For example, they are named as Average Accuracy (ACC) and Backwarde Transfer (BWT) in (Chaudhry et al. 2018 Lopez-Paz & Ranzato, 2017), Average Performance (AP) and Average Forgetting (AF) in
(Liu et al.) 2021), Accuracy Mean (AM) and Forgetting Mean (FM) in (Zhang et al., 2023a), and performance mean (PM) and forgetting mean (FM) in (Zhou & Cao 2021)
A.5 PERFORMANCE MATRIX ANALYSIS
Given a visualized performance matrix, we should approach from two different dimensions. First, the i-th row of the matrix denotes the performance each previously learned system after the
22

---

Published as a conference paper at ICLR 2025
model has learned from the 1-st system to the i-th system. Second, check the performance of specific system over the entire learning process, we check the corresponding column. For example, in Figure [ (e), each column maintains stable color from the top to the bottom. This indicates that the performance of each system is perfectly maintained with little forgetting. But if the color becomes darker and darker from top bottom (increasing MSE), it indicates that the corresponding method is exhibiting obvious forgetting problem.
23


Reviews:
[{"review_id": "Yq6Cr9Clo2", "reviewer": "Reviewer_Pzyn", "summary": "The paper proposed a continual learning framework based on GraphODE over dynamical systems. The key motivation is to address the catastrophic forgetting problem if multiple systems of different configurations are within a sequence. The proposed Mode-switching Graph ODE (MS-GODE) is able to select the best sub-network mask for a given observation sequence, following the parameter-isolation category in continual learning. A new dataset on biological cellular systems is created in the experiment.", "questions": "1. For the experiment table 1, what does the results fine-tune and joint mean respectively? In line 400-402 the authors mention that for each baseline there are two options, so I overall do not get the meaning of the two rows in Table 1,which model are they based on ?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper studies an interesting problem in dynamical system modeling under distribution shifts. Though I feel the motivation example can be changed to more realistic one (see below). \n\n2. The paper proposes a useful benchmark for evaluating the distribution shifts in dynamical system modeling, on biological cellular systems.\n\n3. The proposed method is able to achieve good performance over selected baselines.", "weaknesses": "1. While I in general get the motivation of this paper, I feel the examples used in the introduction section need to be further improved. First for Figure 1, there are so many contents/fonts in the figure that are not well-explained in the caption or the main text. It is suggested to use for example legend to denote different kinetic factors, using some boxes to denote the overall system consists of multiple objects. Here for me, it is hard to read from this figure along with its current explanations. For the second example, I feel it is not very natural to combine spring systems and charged particle systems, as there is no reasonable transition among them. A more proper example may be that some water particles will froze into solid when the temperature decreases, and the interaction among liquid and solid particles can be very different. I encourage the authors to further improve the motivation example so the audience from various backgrounds can understand them easily.\n\n2. For the baselines, the authors chose different continual learning framework as comparison. However for learning over multiple systems, there are some work [1,2] that directly learn a generalized neural simulators. It is suggested to also discuss their performance and show if continual learning is the better way to learning over different systems.\n\n3. For majority of the citation, it should be \\citep (reference with a brackets) instead of \\cite. For example, line 40-50.\n\n[1] Generalizing Graph ODE for Learning Complex System Dynamics across Environments.\n\n[2] HOPE: High-order Graph ODE For Modeling Interacting Dynamics\n[2]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UlE7VKjJDS", "reviewer": "Reviewer_VBaq", "summary": "The main motivation of this paper is that, system dynamics are not static, rather subject to change over time in practice. Accordingly, the authors developed a mode-switching graph ODE (MS-GODE) to continually learn evolving system dynamics. Unlike most of the existing works on dynamics modeling, which have been focused on a single dynamics, MS-GODE can process multiple system dynamics by automatically switching binary masks on shared model weights (i.e., different weights for different system dynamics). Also, the authors introduces bio-CDL, a novel dynamic system benchmark to evaluate the proposed methods in addition to physics-based particle systems.", "questions": "provided with weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Continual learning setup on interacting system dynamics is a novel problem. Since very recent works have explored evolving system dynamics, continual learning can provide a new insight on the related research community.\n\n- As the authors mentioned in L131, neural ODE has been focused on a single dynamical systems. Combination of neural ODE with masked networks is a new approach. \n\n- The paper introduces a novel benchmark using biological cellular systems. Considering many benchmarks on interacting dynamical systems are based on physical systems (e.g., springs, charges), the new benchmark could provide another insight whether models still perform well when predicting dynamic states, not locations of particles.\n\n- The details about the experimental settings and background are well presented.", "weaknesses": "- While I still consider the combination of neural ODE with continual learning interesting, technical novelties seem somewhat limited. For example, both the network design (e.g., LG-ODE, NRI) and learning method (e.g., masked-based CL, edge-popup algorithm) are already well known. I think, unique technical contributions this paper had made should have been clearly presented.\n\n- Also, the motivation of CDL needs to be better clarified. I don’t think continual learning is always required for all evolving dynamical systems. For example, what if a system never repeats the past dynamics once it evolves? what if the prediction model can quickly enough adapt to new dynamics? We don’t need to make the models avoid catastrophic forgetting in these cases. In other words, unless the system dynamics repeat again and again, which might narrow the scope of target applications, mitigating catastrophic forgetting won't be useful.\n\n- The results from Table 1 (performance of all methods is better in high-level dynamics shift) are interesting. The authors provide a brief discussion in L471-L475, but is there any experimental or theoretical evidence to believe this? For example, how do the authors believe that “e.g. Fine-tune, the results indicate that diverse systems may guide the model to exploit different parameters for different systems”? In other words, the discussion and detailed analysis on the performance is not enough while comparison with other methods in terms of AP and AF are well studied.\n\n- There have been many recent works to address evolving system dynamics [1,2,3] though they are not continual learning-based works. Since this paper specifically address such systems, I think related works should have included them to further clarify the novelty in the problem setup.\n\n[1] Dynamic Graph Neural Networks Under Spatio-Temporal Distribution Shift, NeurIPS 2022\n\n[2] CARE: Modeling Interacting Dynamics Under Temporal Environmental Variation, NeurIPS 2023\n\n[3] Online Relational Inference for Evolving Multi-agent Interacting Systems, NeurIPS 2024\n\n- In equation (3), do the authors assume the spatial-temporal edges (or graph structures) are known?\n\n- Can the authors specify if task boundaries should be known for training? As there are many different continual learning setups, it would be nice to specify it.\n\n- There are some typos (e.g., L208: “to x) as a state” -> “to x as a state”, L965: “backpropagatioon” -> “backpropagation”)\n\n- It would be better if a code implementation was provided in the review stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGJuFZI8vm", "reviewer": "Reviewer_ZNCe", "summary": "This paper tackles a significant gap in dynamics learning by introducing Continual Dynamics Learning (CDL), addressing systems whose dynamics evolve over time. The core contribution, Mode-switching Graph ODE (MS-GODE), combines sub-network learning with mode-switching to handle varying dynamics while preventing catastrophic forgetting. Their approach marks the first systematic treatment of this problem, supported by extensive empirical validation.", "questions": "Contrary to the author's claim, there have been pretrained models for dynamical systems lately. Can they be leveraged to improve MS-GODE's performance, especially in sub-network learning?\n\nSeifner, Patrick, et al. \"Foundational inference models for dynamical systems.\" arXiv preprint arXiv:2402.07594 (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper's primary innovation lies in recognizing and formalizing the CDL problem, which has been overlooked despite its practical importance. The MS-GODE architecture effectively combines proven techniques (Graph ODEs, sub-network learning) with a novel mode-switching mechanism, demonstrating strong performance across different system configurations. The introduction of Bio-CDL, a benchmark featuring biological cellular systems, significantly enriches the field beyond traditional physics simulations.", "weaknesses": "- Several important baselines are notably absent from the comparison, including CG-ODE, GG-ODE, PG-ODE, and HOPE. \n\n- The experimental validation would be more convincing if it included tests on human motion and molecular dynamics (MD17) datasets, which could reveal how the approach handles different types of dynamic patterns. The model shows sensitivity to hyperparameters, particularly dropout rates and mask selection strategies, but lacks clear guidelines for parameter selection in practical applications. \n\n- The paper also needs stronger theoretical foundations - while the empirical results are promising, there's no formal analysis of why mode-switching works better than traditional continual learning approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j76kvd9ws2", "reviewer": "Reviewer_Cscj", "summary": "The paper introduces Mode-switching Graph Ordinary Differential Equations (MS-GODE), a model designed to address Continual Dynamics Learning (CDL) for predicting time-dependent systems with evolving dynamics. Traditional approaches struggle with continual learning when dynamics vary, leading to issues like catastrophic forgetting, where previous dynamics are “forgotten” once new patterns are introduced. In this work, MS-GODE incorporates sub-network learning with mode-switching capabilities to prevent forgetting, allowing the model to dynamically adapt to different systems without overwriting learned dynamics. Additionally, the paper presents Bio-CDL, a benchmark dataset focusing on biological systems with varied dynamics, to further assess CDL models.", "questions": "1. How robust is the model to variations in initial mask configurations? Would pre-training the masks on related dynamics (e.g., similar system behaviors) improve or stabilize performance?\n2. With multiple modes and sub-networks, how does the model handle scaling to larger or more complex networks (e.g., with thousands of nodes)? Would the approach require significant modification for larger systems, or is it inherently scalable?\n3. The paper notes that abrupt dynamics shifts yield better performance than gradual shifts. Could the authors clarify whether any architectural modifications could improve MS-GODE’s performance under gradual shifts, which are common in many real-world applications?\n4. What if some interactions require longer time to have an effect? Will the method still work?\n5. What if the switching of the dynamics is not observed in the trajectories? Will the performance of the proposed method drop?\n6. Minor: Please check the citation form in the paper, especially in introduction. They are not properly cited and break the sentences.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The methodological foundation of MS-GODE is robust, integrating sub-network learning and mode-switching mechanisms to handle evolving system dynamics. The model architecture is well-constructed, and the choice to use fixed backbone weights with adaptive binary masks for each dynamic mode is theoretically sound and practically effective.\n- Besides, The paper is generally well-organized, with clear problem framing and concise descriptions of the core components of MS-GODE.", "weaknesses": "- The model’s reliance on multiple components—such as sub-networks, binary masks, and a mode-switching module—adds to its complexity. While effective, this multi-faceted design may limit accessibility, especially for researchers less familiar with CDL or advanced dynamic modeling techniques.\n- Although the Bio-CDL benchmark dataset represents a practical biological context, additional real-world datasets across broader domains (e.g., climate science, economics) could have bolstered the model’s credibility and practical relevance. The biological focus, while useful, limits insights into how the model would handle other types of complex, evolving systems.\n- The MS-GODE model’s performance is sensitive to the initialization of binary masks and dropout rates, as shown in the experiments.\n- The content in the main body about the benchmark introduced in this work is too short. As it is claimed to be a main contribution of this work, it would be better to describe more on the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)