ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence
Authors: Wenjie Mei
Dongzhe Zheng
Shihua Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural ODEs
Differential Equations
Dynamical Systems
Deep Learning
Primary area: deep_learning_architectures

Abstract: Neural ODEs (NODEs) are continuous-time neural networks (NNs) that can process data without the limitation of time intervals. They have advantages in learning and understanding the evolution of complex real dynamics. Many previous works have focused on NODEs in concise forms, while numerous physical systems taking straightforward forms in fact belong to their more complex quasi-classes, thus appealing to a class of general NODEs with high scalability and flexibility to model those systems. This however may result in intricate nonlinear properties. In this paper, we introduce ControlSynth Neural ODEs (CSODEs). We show that despite their highly nonlinear nature, convergence can be guaranteed via tractable linear inequalities. In the composition of CSODEs, we introduce an extra control term for learning the potential simultaneous capture of dynamics at different scales, which could be particularly useful for partial differential equation-formulated systems. Finally, we compare several representative NNs with CSODEs on important physical dynamics under the inductive biases of CSODEs, and illustrate that CSODEs have better learning and predictive abilities in these settings.

ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence
Wenjie Mei*t
Dongzhe Zheng*t
Shihua Li
Abstract
Neural ODEs (NODEs) are continuous-tinn neural networks (NNs) that can pro- cess data without the limitation of time intervals. They have advantages in learning and understanding the evolution of complex real dynamics. Many previous works have focused on NODEs in concise forms, while numerous physical systems taking straightforward forms, in fact, belong to their more complex quasi-classes, thus appealing to class of general NODEs with high scalability and flexibility to model those systems. This, however, may result in intricate nonlinear properties. In this paper, introduce ControlSynth Neural ODEs (CSODEs). We show that despite their highly nonlinear nature, convergence can be guaranteed via tractable linear inequalities. In the composition of CSODEs, we introduce extra control term for learning the potential simultaneous capture of dynamics at different scales, which could be particularly useful for partial differential gquation-formllati nally, we compare several representative NNs with CSODEs on important physical dynamics under the inductive biases of CSODEs, and illustrate that CSODEs have better learning and predictive abilities in these settings.
systems. Fi-
Introduction
Neural ODEs (NODEs) were developed from the limiting cases of continuous recurrent networks and residual networks and exhibit non-negligible advantages in data incorporation and modeling unknown dynamics of complex systems, for instance. Their continuous nature makes them particularly beneficial to learning and predicting the dynamical behavior of complex physical systems, which are often difficult to realize due to sophisticated internal and external factors for the systems.
Starting from the introduction of NODEs, many types of variants of NODEs have been studied (see, egg., 66 19 [.).. Nevertheless, there are rare studies concerning highly scalable and flexible dynamics that also present complex nonlinear natures, bringing difficulties in their modeling and analyses. No in-detail research attention has been paid to the scalability of depth and structure in NODEs despite numerous physical systems in the real world having inscrutable dynamics and whose
compositions. To fill in this gap, we propose ControlSynth Neural ODEs (CSODEs), structure another sub-network is also incorporated for enlarging the dexterity of the composition and controlling the evolution of the state. Different from most of the existing methods and experiments, we focus widely investigated physical models, with known state evolution under necessary conditions. Subsequently, we will show that the proposed NODEs are effective learning and understanding those models.
Our contributions are mainly composed of the novel structure of CSODEs, their convergence guar- antees, and the comparative experiments among CSODEs, several representative NODEs, and their
Equal contribution. Correspondence wenjie. mee.mmmeiiee Aseu. edu, and 1sh@seu. edu. Wenjie Mei and Shihua are with School Automation and the Key Laboratory of MCCSE of the Ministry of Education, Southeast University, Nanjing, China. Dongzhe Zheng is with the Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

divisions, illustrating the beneficiality of CSODEs in the setting of prediction. The convergence conditions provide tractable solutions for constraining the learned model to a convergent one. The preliminary experiment demonstrates that our CSODEs can learn the evolution of the dynamics faster and more precisely. Also, we show that introducing sub-networks into CSODE does not impact the overall computational performance more than the other comparable NNs. Finally, we com- pare NODE, Augmented Neural ODE (ANODE), Second Order Neural ODE (SONODE), CSODE, and its variant in real dynamical systems. The experimental results indicate that our CSODE is beneficial to more accurate time series prediction in the systems. Our code is available online at waa.a/ooo/ooonnuuuooeeeeee
2 ControlSynth Neural Ordinary Differential Equations
We begin by introducing the form of CSODEs as follows:
M
=(1) - Aoo(1)+ Σ (,), (I) j=1
where I2= == x(t) E R" is the state vector; the ma- trices A. are with approximate dimensions; W. are weight matrices; the input == u(t) U C R" E LL (refer Appendix
M()
AufulWar)
/
A11
At
At
Benndixc..I:.. C. (ff (Jg : RR + RR5) and g U - R" are the func- tions ensuring the existence of the solutions of the neural network (NN) (1 at least locally in time, and
Figure 1: Schematic of the CSODEs solver, showing integration via NNs at one time step. Using the forward Euler method as an example, it shows how u, and æ, evolve through the update neural function h(-) and NNs g()), A1fi (W1.), As f (War) ) to yield de and T2+ the next variables.
g [91 -01---9h 9n] w.l.o.g., the time I is set as 2 O.
CSODEs extend the concept of Neural ODEs, which are typically expressed as &tt)= f(((t)), where f is a neural network. CSODEs incorporate control inputs u(t) and create combination of subnetworks Σn-1 (Wfs(Waa(tt) This formulation enhances ex-
pressiveness and adaptability to complex systems with external inputs. CSODEs provide a more universal framework and improve upon Neural ODEs by offering greater flexibility, interpretability through separated linear and nonlinear terms, and natural integration with techniques in control theory.
For simplicity, in the experiments (see Section 6p, we select common NNs as the specific examples of the function g: Traditional Multilayer Perceptron (MLP) and two-layer Convolutional Neural Network (CNN), and tanh as the used activation functions: particular subclass of the functions fz. Note that in the case that g(u) represents an MLP, there are many different types of neurons, for example, functional neurons (21], and in CSODEs T there may exist f} E L (X, A, H)), where (X,A, A, 14) denotes a o-finite measure space.
3 Related Work
SONODES, ANODES, and NCDEs vs CSODES SONODES 117 are particularly suitable for learning and predicting the second-order models, and ANODEs 6 are useful for cases where the state evolution does not play an important role. In contrast, many real models of second-order or even higher orders can be transformed into the common first-order but emerge numerous nonlincarities that are difficult accurately treat by NODEs and their variants. Despite the intricate structure of CSODEs. they can be equipped with a significant convergence attribute. The experiments show that our CSODES are more appropriate for model scaling and natural dynamics that exhibit complex nonlinear properties. Although both Neural CDEs (NCDEs) [12] and CSODEs extend NODEs through control mechanisms, their architectural philosophies differ. NCDEs introduce control through path-valued data driving the vector field, following a(!)= = (xx(t)) In contrast, CSODES propose a more sophisticated structure combining multiple subnetworks with a dedicated control term g(u(t)). This design not only provides theoretical convergence guarantees but also enables CSODEs

---

to capture intricate physical dynamics through its hierarchical structure, particularly beneficial for systems exhibiting complex nonlinear behaviors that elude simpler controlled formulations.
Convergence and Stability Relevant to our work, 133 studies the asymptotic stability of a specific class of NODEs part of its fundamental property investigations on NODEs. In contrast, study focuses on the scalability aspects related to both the number of subnetworks and the width experiments primarily toward learning and predicting the dynamic behaviors of physical models rather than image classifications. Stability analyses have also been performed NODE variants, such as SODEF [[]
of NNs, extending beyond the scope of the NNs in [33 [5] [6]. Furthermore,
SNDES 26. and Stable Neural Flows 14
Physical Information Based Dynamics Similar to the way CSODEs embody physical dynamics, various other models have been developed to incorporate physical information, ensuring the derivation of physically plausible results, both in discrete and continuous time. For example, [3] employs physics- informed neural networks (PINNs) (In contrast to NODEs and their variants, which mainly focus on implicitly learning dynamical systems, PINNs explicitly incorporate physical laws into the loss function. These two types of methods tackle the problem integrating physical knowledge into deep learning models from different perspectives and complementary research directions) to solve the Shallow Water Equations on the sphere for meteorological use, and Raissi et al. [20] propose a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations using PINNs. Shi et al. (33 and O'Leary et al. [8] utilize enhanced NODES for learning complex behaviors in physical models, while Greydanus et al. O and Cranmer et al. [5] propose
Hamiltonian and Lagrangian NNs, respectively, to learn physically consistent dynamics.
4 Theoretical Results: Convergence Analysis
Consider the CSODEs given in Equation @ where fj : R + R& are nonlinear activation functions, For them, an imposed condition on fj (the i-th element of the vector-valued fz) is presented as follows:
Assumption For any E (k. and j E M....,,M, sf}(s) > for all s RM(0).
Remark 1 Assumption ① applies to many activation functions, such tanh and parametric ReLU. picks up the activation functions passing through the origin and the quadrants I and III. For more explanations, the reader is referred to Appendix B.]
In this study, analyze the convergence property of the NN ( we first define the concept of convergence:
Definition 1 The model ( is convergent if it admits unique bounded solution for t E I. that is globally asymptotically stable (GAS).
In order to investigate the convergence, two properties have to be satisfied, that is, the boundedness and the GAS guarantees of the solution x* for (T). In this respect, two assumptions are given as follows.
Assumption 2 Assume that the functions f are continuous and strictly increasing for any i kl. and (L....,M,
Assumption 2 aligns with CSODE's structure, reflecting continuity and monotonicity of activation functions. This relates to model dynamics and is satisfied by most common activations.
In the analysis of convergence, one needs to study two models in the same form but with different initial conditions and their contracting properties. To that end, along with ( we consider the model (t) = Aoy(t) + ΣM Ayf, (W + g g(u(t)) with the same input but different initial conditions (0) E R Let { == y xr. Then the corresponding error system is
M
E = A
+
Σ (z, 1=1
(2)
where - - Note that for any fixed x E R2, the functions P5 in
the variable { TR" satisfy the properties formulated Assumptions I 2 is imposed for analyzing the contracting property of (2).
The following assumption
3

---

Assumption 3 Assume that there exist Sy S3" Ho. Hy. H2, Hg" (j.r (1....,,,,, M)) with that
positive
semidefiiite appropriate
diagonal
matrices
dimensions
such
(, 6) (x, s 5 s' s + 2 W (2,, 2 w (W
M
22
E) WWW.
and
fs(WE) (WWGE) ((W35)
s E W H&W, + 25 w. H2ps(a, + 25 WT H3f(WG))
M
+2 Σ (,( W, w,igg w. W.W.W..WW.....
for all v, y E R." and 6= y-
is at least more relaxed than Lipschitz continuity (see Appendix B.2 for an intuitive example of activation functions satisfying Assumption 3
Notice that Assumption S
Convergence Conditions We are now ready to show the convergence conditions for the
CSODES (T):
Theorem Let Assumptions 7 be satisfied. If there exist positive semidefinite symmetric ma- trices P. P; positive semidefinite diagonal matrices {A diag(A) 1........ AB)) (Ar JoseersMM {f j3'""> {T tive definite symmetric matrix 4; and positive scalars 7,, 0 such that the linear matrix inequalities (LMIs) in Appendix B.4 hold true. Then, a forward complete system [ is convergent.
{T {3")
{2, { Eo; posi-
Proof in Appendix C.3 Note that the used conditions on f in Assumption can be relaxed to
"non-decreasing" which enlarges the scope of activation functions, including non-smooth functions like ReLU, then the resulting modifications for the formulations of Theorem can be readily obtained, highlighting the CSODE framework's k's adaptability.
Those LMI conditions ensure system convergence. From an energy perspective, this indicates the error system's 's generalized energy (represented by the energy (or Lyapunov) function) is monotonically non-increasing, leading to convergence towards the equilibrium point: origin. These conditions can be easily verified, thanks CSODE's structural characteristics and LMIs' highly adjustable elements.
The matrices, such as - and P "..' in the LMIs act as compensation terms balancing the effects of linear and nonlinear terms, ensuring the derivative of the energy function V remains non-positive. Properties of fj (Assumptions[[andd] provide facilitation constructing these matrices. Assumption 3 allows for non-restrictive conditions on activation functions, avoiding strong global Lipschitz continuity assumptions and providing precise local asymptotic stability characterization.
Preliminary Experiments
Convergence and Stability Convergence, important attribute showcasing a model's learning ability, refers to its capability to consistently approach the theoretical optimal solution throughout the learning process over iterations. To validate the convergence and stability of CSODES, design an experiment that involves learning simple spiral trajectories, chosen for their fundamental complexity which all models should ideally handle. In this experiment, we compare CSODEs and NODES, both based on the Latent ODE structure [4. This setup provides a fundamental baseline for assessing convergence and ensures fair comparison, enabling each model demonstrate its learning capabilities under comparable conditions without bias toward specific structural advantages. The Mean Absolute Error (MAE) loss function, which measures the average of the absolute differences between the estimated trajectories and the true trajectories, is used as the indicator. We train the model on 100 noisy spiral data observation points to learn this trajectory. The MAE loss values collected over training epochs consistently, plotted in Figure 2 show that the CSODE model not only converges faster but also maintains a lower error rate compared to the traditional NODE model, particularly under the constraints of limited and noisy observational data points. Our model's 's ability to accurately predict the target trajectory despite the noisy data illustrates its noise tolerance, a critical aspect of its robust stability.

---

Generalization and Extrapolation Figure 3 presents a visual comparison of the prediction results from the CSODE model and the traditional NODE model with the actual ground truth trajectory after various training epochs in trajectories learning experiment in the previous paragraph. It is evident from the figure that the CSODE not only learns the trajectory within the observation period more precisely in fewer training epochs but also aligns more accurately with the original trajectories beyond the observation period, which accounts for 25% of the total duration. This demonstrates the robust generalization and extrapolation capabilities of CSODEs for predicting future states, illustrating its better understanding of the underlying time-dependent structures and dynamics the data.
ControlSyath Neural ODE:
30
CSODE
NODE
20
ODE:
MAE
15-
5-
Epach 400
Epoch
Epoch 800
250
1000
1250
1500
Original Trjjectry - Prodicted Trjoctory
Sampled Trjictory with Noise
Epoch
Comparison of Mean Absolute Error (MAE) loss reduction curves across 1500 train- ing epochs between CSODE and NODE models until convergence.
Figure 2:
Figure 3: Qualitative comparison of the CSODE (top) and the NODE (bottom) predictions against ground truth trajectories at 400, 600, and 800 training epochs.
Computational Performance To eval- uate whether introducing subnetworks CSODEs impacts the overall computa- tional performance, we conduct experimen- tal validations on image classification, fo- cusing on aspects such as operational effi- CSODE 0.39%
Table 1: Performance Comparison on the MNIST
Model
Test Err
Params
FLOPS
Time
Mem
0.42%
9c-3s
ANODE
0.41%
0.22M
2.12B
9e
372MB
SONODE
9c-3s
0.22M
2.11B
9c-3s
372MB
ciency and system load. These experiments are based on the ODE-Nets framework 4 using the forward Buler method, and span 100 epochs on the MNIST dataset to compare the performance of NODEs (4, Augmented Neural ODEs (ANODEs) 66. Second Order Neural ODEs (SONODEs) (17), and CSODEs. All experiments in this study are conducted on a system equipped with an NVIDIA GeForee RTX 3080 GPU and CUDA, ensuring consistent computational environment across all tests. Performance metrics include test error rate, number of parameters (# Params), floating-poin op- erations per second (FLOPS), average batch processing time (Time), and peak memory usage (Mem). The results (see Table [ show that although CSODES theoretically involves iterative computation of g(u), it does not significantly reduce computational efficiency in practice. While MNIST is relatively simple dataset, the experimental outcomes preliminarily confirm the benefits of incorporat- ing a learnable function in enhancing the performance of implicit networks, effectively maintaining efficiency, and alleviating concerns over the potential trade-off between expressive capability and computational efficiency the networks.
6 Complex Systems Time Series Prediction Experiments
In this section, we experimentally analyze the performance of CSODEs, based on NODEs, compared to other models based on NODEs and traditional time series forecasting methods in extrapolative time series prediction for complex systems. We select challenging physical dynamical systems from neuroscience (Hindmarsh-RRoe model [00). chemistry and biology (Reaction-Diffusse model (8).
and geophysics and fluid dynamics (Shallow Water Equations 255 which exhibit rich spatiotemporal variations over long time scales.
5

---

6.1 Model Architectures for Experiments
To comprchensively evaluate the performance of different model architectures in dynamic system modeling, this study compares several representative NODEs-based models, including the traditional NODEs 4) ANODEs [6], SONODEs 17,, and our proposed CSODEs. Additionally, these models are compared with traditional MLPs 22 and Recurrent Neural Networks (RNNs) D
Notable that while CSODE can be integrated with the Latent ODE framework, as demonstrated in our preliminary experiments (in Section (5). we opted to conduct our main experiments without this integration. This decision was driven by our observation that NODE itself effectively models long-sequence dynamical systems, allowing us to evaluate CSODE's performance more directly by climinating potential influences from additional Latent ODE components like encoders and decoders.
All models within our experiment that are based on the NODE framework utilize a uniform MLP to parameterize the core dynamical system ÿ = fyy. These models employ the forward Euler method for integration, though other numerical solvers like Dopris are also viable options (detailed performance comparisons of different solvers can be found in Appendix [.I). The MLP serves as the update function, iterating to act on the evolution of dynamic system states. Specifically, the MLP receives the current state input and outputs the next state, thereby modeling the change in state over time. Meanwhile, the RNN employs the same structure as the MLP, using current input and the previous timestep's 's hidden state to update the current timestep's hidden state. To ensure a fair comparison, we finely align the number of parameters across all models, including the parameterization of the core dynamical system and other components (such as the subnetwork in the CSODE model and the augmented dimensions in the ANODE model). All designs ensure that the total number of parameters remains consistent within 11%
The unique feature of CSODEs lies in the introduction of an additional subnetwork to model the control term g(u(t)), extending the original dynamical system. We employ a unified auxiliary NN to model the changing rate of g(w(()), with the initial value g(un) set to be the same as y/- These subnetwork structures are similar to the MLP or use simplified single-layer networks. We also introduce the -dap variant, replacing the function representing the changing rate of the control term with a network consisting of two convolutional layers, explore the scalability and flexibility of CSODEs. Notably, for fair comparison of fundamental architectures, we used the standard Adam optimizer in our main experiments, though we note that alternative optimizers like L-BFGS could further enhance CSODE's performance (see Appendix 1.2 for more details).
At the current research forefront, some researchers have proposed integrating NODEs with Trans- former layers (TLODEs) 27 TLODEs can be considered a special case of CSODEs, where the Transformer layers implement the function f(-). Building on TLODEs, we introduce the control term g(.), creating a more complete CSODEs structure with Transformer layers (CSTLODDEs). Consider- ing the widespread application of Transformers in time-series prediction, we conduct comparative experiments between the Transformer, TLODE, and CSTLODE models. However, due to significant architectural differences between Transformers and MLPs and RNNs, directly incorporating them into the main experiment might introduce additional confounding factors, deviating from the theoretical discussion of general NODEs. Therefore, maintain the focus and clarity of the main text, we have placed the experimental results and discussions related to TLODEs and CSTLODES in Appendix
6.2 Experimental Tasks and Datasets Description
In this subsection, we detail the experimental tasks and datasets used to explore the application of NNs in simulating various dynamic systems. For more comprehensive simulation and experimental details, see Appendix D
Modeling Hindmarsh-Ronn Neuron Dynamics In this task, we explore the application of NNs in simulating the Hindmarsh- Rose neuron model [10]. We validate their potential in simulating complex neuronal dynamics and assess their prospects in broader neuroscience and biophysical research. The Hindmarsh-Ross model is widely used describe neuronal dynamic behavior, particularly suitable for studying neuronal firing behavior and chaotic phenomena. This model describes the evolution of the neuron's 's voltage and recovery variables over time with the following three coupled nonlinear differential equations:

---

dr dt = ax + bx - 2 +
dy
= dx² dz dt
= r(s(2 mo) - 2),
(3)
dt
where z represents the membrane potential, y represents the recovery variable associated with the membrane potential, and is the variable for adaptive current. The parameters b, d. T, 8, To, and determine the neuron's 's nonlinear firing behavior, voltage response curve, recovery process rate, and adaptive current, simulating the voltage changes and behavior of the neuron.
We conduct 1000 simulations, each lasting 20 seconds. In the first 10 seconds of each simulation, collect 1500 data points reflecting the changes in the neuron's 's three-dimensional coordinates (vr, z). We divide these data points into 50 sequences, each containing 30 time points. Our goal is to these sequences to train an NN to predict the dynamic changes over the next 10 seconds, specifically the sequences of the next 30 time points.
Modeling Reaction-Diffusian System In this task, we employ NN models to simulate the Gray- Scott equations 8 a Reaction-Diffusion system widely used to describe chemical reactions and diffusion processes. This system is particularly significant in pattern formation, biological tissue, and self-organizing chemical processes, holding important theoretical and practical implications. This task allows us to explore the potential applications of these models in environmental and bioengineering contexts. The model describes the interactions and spatial diffusion of two chemical substances, U and V, through set of partial differential equations (PDEs):
OU DUV UV2 + j((1 OV
Ōt VVV +
Øt
where Dy and Dy are the diffusion coefficients, and j and l are reaction rate parameters.
We conducted 1000 simulations with periodic boundary conditions [T] for 150 seconds, each ran- domizing initial conditions and diffusion coefficients Dy and Dv. and used data from the first 100 seconds. These 100 seconds of data were divided into 100 sequences, each spanning 25 seconds, to train the NNs. Our goal is to utilize the training results predict chemical dynamics over the next 50 seconds.
Modeling Shallow Water Equations In this task, explore the application potential of NODE systems in simulating the Shallow Water Equations [25]. which describe the horizontal flow and surface behavior of liquids in confined spaces. These equations have significant physical relevance in fields such as hydrology and environmental engineering, especially in flood simulations and wave dynamics analysis. Mathematically, the Shallow Water Equations nonlinear PDEs that represent changes in water depth (h) and flow velocity (u), incorporating the principles of continuity and conservation of momentum, with gravity g, follows:
expressed through set of
Oh (huu = 0,
a(hu) a1
Ø1 + v
+ (hu 2 u) vo
(5)
We conduct 1000 periodic boundary condition wave propagation simulations [ lasting 7 seconds, generating 1 5500 data points in the first 4.7 seconds to describe water wave depth variations. After normalizing the data using sliding window, we segment it into 100 sequences, each containing 15 points, to train a global water surface dynamics model to predict water wave depth changes during the remaining 2.3 seconds of the simulation.
6.3 Metrics for Assessing Prediction Accuracy
We employ the following metrics to compare time series predictions and ground truth:
Mean Squared Error (MSE): Calculates the mean of squared differences between predicted and actual values, sensitive to large errors. Used in all three tasks.
Mean Absolute Error (MAE): Measures the average absolute difference between predicted and actual values. robust to noise. Used in all three tasks.
R² Score: Quantifies the proportion of variance predictions relative to actual data, reflecting explanatory power and accuracy. Used in the Hindmarsh-Ron model's 3D time series prediction.
7

---

Chamfer Distance (CD); Measures the average shortest distance between points in two sets, empha- sizing spatial structure matching accuracy 2 Used to compare predicted and ground truth physical field sequences in Reaction-Diffusion systems and Shallow Water Equations tasks.
6.4 Experimental Results and Analysis
Experiments are conducted ten times for each task, and the average metrics are presented in Table 2 the reader is referred Appendix [ for more statistical details. NNs in Group B. based NODE and its variants, outperform traditional models in Group A, such as MLP and RNN, in predicting complex physical dynamic systems. Traditional models struggle with complex time dependencies and nonlinear dynamics, while models based on differential equations demonstrate greater adaptability and accuracy.
Table 2: The table categorizes NN models for time-series prediction in complex dynamical systems into three groups: Group A includes traditional models like MLP and RNN; Group features ODE- based models including the NODE, ANODE, SONODE, and CSODE models; Group C presents Adapt with convolutional layers. The best-performing models are highlighted in blue and
the second-best in brown.
Hindmarsh-Rose
Sraction-Diffusion
Shallow Water
Group
Model
MSE
MAE
R
MSE
MAE
CD
MSE
MAE
CD
MLP
2,394
1.033
0.241
7.43le-2
0.398
7,745
0.135
0.35
3.17
A
1.975
0.871
0.356
2.376e-2
0.143
3.115
0.103
0.26
3.06
NODE
1.551
0.682
0.515
1.134-22
0.081
1.311
0.071
0.22
2.89
ANODE
0.745
0.586
0.637
1.09e-22
0.076
1.392
0.067
0.19
2.88
B
SONODE
0.739
0.561
0.611
8.145e-3
0.056
1.315
0.065
0.19
2.83
CSODE (Ours)
0.783
0.470
0.758
6.365e-3
0.058
0.939
0.041
0.16
2.27
C
CSODE-Adapl (Ours)
0.408
0.370
0.887
5.635e-3
0.035
0.837
0.031
0.15
1.89
Within Group B, the CSODE model surpasses other models due to its control elements that enhance precision and adaptability to changes in initial conditions and system parameters. Figures Figures and show that CSODEs accurately reflect complex dynamic system details through qualitative results, with more results available in Appendix G
Moreover, the ESODE-Adapl model (Group C) integrates convolutional layers, enhancing its appli- cability and effectiveness, particularly in dynamic systems with significant spatial features, such as Slaction-Diffusion systems. This model performs better than all others, highlighting the flexibility and highly customizable structure of the CSODEs and its advantages and potential in predicting complex physical dynamic systems.
Gray-Scott Model of Reaction Reaction-Diflusion U (dimerssioless)
NODE
ANODE CSODE
Initial Condition
Ground Truth for Prediction
CSODE-Adpp Gromd Truth
06
64
Xariis
Y Yanis
Figure 4: Qualitative results of Hindmarsh Hinddarrs--Roe
Qualitative results of Reaction- Diffusion system prediction
Figure 5:
model prediction
6.5 Comparison with Observation-aware Baselines
We conducted supplementary experiments (CharacterTraaectorii and PhysioNet Sepsis Prediction) with CdapE-Adapt following the experimental setup in [12], maintaining similar network structures,

---

parameter counts, and optimization methods. Overall, CSODE performs slightly worse than Neural CDE in irregular observation experiments but better in other time-series-related tasks. We have also performed our main experiments for neural CDE. The corresponding results are shown as follows:
Table 3:
Performance comparison with observation aware neural networks. Group A includes observation-awar baselines like ODE-RNN and Neural CDE; Group B contains our base model CSODE; Group C showcases CSODE-Adapt with enhanced observation mechanisms. Best perform- ing models are marked in blue, second- best in brown.
Metric/ / Model
Neural CDE
ODE-RNN
CSODE
CSODE-Adapt
Group
CharacterTrajectories (Test Acc.)
30% Missing
97.8%
96.8%
97.3%
97. 1%
50% Missing
98.2%
96.5%
97.8%
97.6%
70% Missing
97.2%
95.9%
96.8%
96.59
PhysioNet Sepsis Prediction (AUC)
w/o OI
0.365
0.855
0.868
0.871
w/ OI
0.885
0.870
0.881
Reaction- tiffi---fffuoonn
MSE
7.1e-3
7.5e-3
7.0e-3
6.8e-3
MAE
0.62
0.59
0.58
CD
0.945
0.950
0.942
0.940
As shown in Table 3, for the Character task with irregular observations, Neural CDE achieves the best performance across all missing data ratios, followed by CSODE and then ODE- RNN. In the PhysioNet Sepsis Prediction task, without considering observation intensity (OI), CSODE-Adapt achieves the highest AUC value of 0.871, while with Ol consideration, Neural CDE performs best with an AUC of 0.885, followed closely by CSODE-Adapt and CSODE. For the Reaction-Diffusin modeling task, CSODE- Adapt demonstrates superior performance across all metrics (MSE, MAE, CD), while CSODE and Neural CDE show comparable performance, both outperforming ODE-RNN. Overall, while Neural CDE exhibits advantages in handling irregular observations, GOODE-Adape shows competitive or superior performance tasks requiring complex dynamic system modeling and clinical prediction, demonstrating its effectiveness as general-purposs time series modeling tool.
7 Model Scaling Experiment
In the experiments above, CSODEs demonstrate significant superiority over traditional NODEs and their variants, under the maintenance of the same number of parameters and architectural configuration. Based on these findings, our scaling experiments focus primarily on exploring the scalability and architectural robustness of CSODEs, without further comparison to other models.
We also observe changes in system performance after scaling CSODEs. To maintain consistency in the experiments, each sub-network is configured with two dense layers. We select the Deaction-Diffusiie task in Section as an example to explore the impact of increasing the number of sub-networks and the width of each sub-network on system performance. Specifically, the network widths, which refer to the number of hidden dimensions in the dense layers, are set at 128, 256, 512, 1024, and 2048. The number of net-neeworks, equivalent to M in NNs (T. is set at 1, 2, 3, 4, and 5.
The experimental design varies network width and number of sub-networks. We We employ learning rate formula: learning rate W where k is a constant, W is the network width, and N is the number of sub-networks. This adjusts the learning rate based on width and moderates it for sub-networt count to handle complexity. For instance, with a width of 1024 and three sub-networks, We use the Adam optimizer for training.
the learning rate is 1024x
In terms of overall performance, the heatmap in Figure shows that under the CSODEs, increasing the network width and number of subnetworks results in stable and enhanced overall performance. Additionally, the scatter plot demonstrates that increasing the number of subnetworks significantly improves the model's generalization ability, with training and validation performance showing stronger correlation. For further details on comparative experiments, the model's stability, and convergence despite the increase in the number of subnetworks and width, refer to Appendix
H

---

Configuration I Heatmap
Comparison Training Validation Lass for Different Subnets
Network Width
128
0,1150
0.0600
0,0350
0,0338
0.0328
0.11
0,025
0,10
Subrat
Subret
256
0.0041
0.09
0/030
Walths
0.06
Timl Valkaaii
tos
Subret
512
0.0879
0,01/1
0.0344
0.0254
0.0300
0.07
0015
0.06
1024
0.0704
0.0065
0.0255
0.0349
0.0327
00.0
2048
0.0637
0,0300
0.0335
0.0344
0,0343
0.04
0.03
0.005 0.010
0,015
0.000
0.605
Traiting Loss
Figure 6: Performance comparison of CSODE models with varying numbers of sub-networks. The scatter plot visualizes the performance trajectory of CSODE models during training, where each point represents the training loss (X-axis) and validation loss (y-axis) at a specific epoch. Models with 1-5 sub-networks are compared, each depicted in distinct color while maintaining fixed network width of 512 neurons. Points clustering near the bottom-left corner indicate superior model performance, while their distribution relative to the diagonal reveals the balance between training and validation performance.
8 Conclusion
In this work, we analyzed the Icarning and predicting abilities Neural ODEs (NODEs). A class of new NODEs: ControlSynth Neural ODEs (CSODEs) was proposed, which has complex structure but high scalability and dexterity. We started by investigating the convergence property CSODEs and comparing them with traditional NODEs in the context of generalization, extrapolation, and computational performance. We also used a variety of representative NODE models and CSODEs to model several important real physical dynamics and compared their prediction accuracy.
We presented that although the control subjects (NODEs, Augmented Neural ODEs (ANODEs), and Second Order NODEs (SONODEs)) do not have the physics information-bast -aaedd inductive biases specifically owned by our CSODEs, they can learn and understand complex dynamics in practice. In particular, SONODES own inductive biases for second-order ODE-formulated dynamies, while the ones of CSODEs mainly are for first-order models with high nonlinear natures, scalability, and flexibility that belong broad class of real systems. The experimental results dynamical systems governed by the Hindmarsh- Rose Model, Reaction-Diffussee preliminarily demonstrate the superiority of our ControlSynth ODEs (CSODEs) in learning and predicting highly nonlinear dynamics, even when represented as partial differential equations.
Model, and Shallow Water Equations
Limitations and Future Work The effectiveness of inductive biases of CSODEs varies which, depending on the specific application scenarios, may not be preferable; in the evolution of partial differential equations, there often exists mutual interference between different scales (e.g., spatial and temporal scales), which, however, is approximately learned by CSODEs. We believe this work could provide promising avenues for future studies, including: For enlarging the use scope of inductive biases of CSODEs in complex dynamics and reflecting the mutual intervention between scales, one can consider more general NODE: ă (z, u). with guaranteed stability and convergence. This allows greater scope of dynamics and thus may prompt the improvement of the accuracy of modeling and predicting systems with more complex structures and behaviors. Furthermore, in practice, CSODEs are more sensitive to learning rate selections due to their more complex architectures. Our preliminary investigation in Appendix L.3 reveals the significant impact of hyperparameter adjustments on model performance. Building upon these initial findings, future research will examine this sensitivity more thoroughly and consider methods like adaptive learning rate adjustment and model simplification to address it. Finally, this work maintained standard optimization settings for fair comparison, future research could explore specialized training algorithms that leverage CSODE's structural properties and theoretical foundations.
10

---

Acknowledgments
This work supported by the National Natural Science Foundation of China (NSFC) under grant 62403125, the Natural Science Foundation of Jiangsu Province, and the Fundamental Research Funds for the Central Universities under grants 2242024k30037 and 2242024k30038.
References
[1] M. P. Allen and D. Tildesley. Computer simulation of liquids. Oxford University Press, 2017.
[2] H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Proceedings: Image Understanding Workshop, pages 21-27. Science Applications, Inc, 1977.
[3] A. Bihlo and O. Popovych. Physics-informed neural networks for the shallow- equations on the sphere. Journal of Computational Physics, 456: 1024, 2022.
[4] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
[5] M. Cranmer, S. Greydanus, Hoyer, Battaglia, Spergel, and S. Ho. Lagrangian neural networks. arXiv preprint urXiv:2003.046344 2020.
[6] E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
[7] J. L. Elman. Finding structure in time. Cognitive Science, 14(2):17922) 1990.
[8] Gray and S. Scott. Autocatalytic reactions in the isothermal, continuous stirred tank reactor: isolas and other forms of multistability. Chemical Engineering Science, 38(1):29-433 1983.
[9] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[10] J. L. Hindmarsh and R. Rose. A model of neuronal bursting using three coupled first order differential equations. Proceedings of the Royal Society of London. Series B. Biological Sciences, 221 (1122))):77-102, 1984.
[11] Q. Kang. Y. Song, Q. Ding, and W. P. Tay. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Advances Neural Information Processing Systems, 34: 34:14925-14999 14937, 2021.
[12] P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
[13] D. Llorente-Vidie M. Ballesteros, I. Salgado, and I. Chairez. Deep learning adapted to differential neural networks used pattern classification of electrophyssological signals. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4807-4188 2021.
[14] S. Massaroli, M. Poli, M. Bin, J. Park, A. Yamashita, and H. Asama. Stable neural flows. arXiv preprint arXiv:2003.088 08063, 2020.
[15] S. Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama. Dissecting neural ODEs. Advances in Neural Information Processing Systems, 33:3952-3963, 2020.
[16] W. Mei, D. Efimov, and R. Ushirobira. On annular short-time stability conditions for generalised Persidskii systems. International Journal of Control, 97(9):2084 2094, 2024.
[17] A. Norcliffe, Bodnar, B. Day, N. Simidjievski, and Liò. On second order behaviour in augmented neural odes. Advances in Neural Information Processing Systems, 33:591 1-5921, 2020.
[18] O'Leary, A. Paulson, and A. Mesbah. Stochastic physics-informed neural ordinary differ- ential equations. Journal of Computational Physics, 468: 1 111466, 2022.
11

---

[19] M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532. 2019.
[20] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informedd neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[21] Rossi and B. Conan-Guez. Functional multi-layer perceptron: a non-linear tool for functional data analysis. Neural Networks, 18(1):45-60) 2005.
[22] D. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by pack-propagating errors. Nature, 323(6889):533-536, 1986.
[23] G. Shi, D. Zhang, M. Jin, S. Pan, and S. Y. Philip. Towards complex dynamic physics system simulation with graph neural ordinary equations. Neural Networks, page 106341, 2024.
[24] E. D. Sontag and Y. Wang. On characterizations of the input-to-state stability property. Systems & Control Letters, 24(5):351-355 1995.
[25] J. J. Stoker. Water waves: The mathematical theory with applications, volume 36. John Wiley Sons, 1992.
[26] A. White, N. Kilbertus, M. Gelbrecht, and N. Boers. Stabilized neural differential equations for learning dynamics with explicit constraints. Advances in Neural Information Processing Systems, 36, 2024.
[27] Y. D. Zhong, T. Zhang, A. Chakraborty, and B. Dey. A neural ODE interpretation of transformer layers. arXiv preprint arXiv:2212.06011 2022.
12

---

A Broader Impact
The introduction of ControlSynth Neural ODEs (CSODEs) represents an advancement in the field of deep learning and its applications in physics, engincering, and robotics. CSODEs offer highly scalable and flcxible framework for modeling complex nonlinear dynamical systems, enabling the learning and prediction of intricate behaviors from data, which has implications across various domains.
the realm of physics, CSODEs provide a powerful tool for studying complex physical processes areas such as meteorology, fluid dynamics, and materials science. By leveraging their highly expandable network structure and control term, CSODEs may capture the dynamics of systems with clusive internal and external factors, overcoming the limitations of traditional mathematical modeling approaches. This opens up new avenues for understanding and predicting the behavior these complex systems.
Morcover, the theoretical convergence guarantees of CSODEs lay solid foundation for applications of neural networks in physical modeling. Despite their high nonlinearity, the convergence of CSODEs can be ensured through tractable linear inequalities. This enhances the interpretability and reliability of neural network models, facilitating their deployment in engineering and safety-critical domains where provable convergence is crucial.
The control term introduced in CSODEs also enables the learning of multi-scale dynamics. which is particularly relevant for systems described by partial differential equations. By simultaneously capturing dynamics different spatial and temporal scales, CSODEs expand the practicality of neural networks to multi- selli--caaleee physical problems.
The performance of CSODEs learning and predicting the dynamics of physical systems, as demonstrated through the comparative experiments, highlights their alignment with physical laws and their ability to effectively uncover underlying patterns in data. This motivates the incorporation of appropriate inductive biases when designing neural network architectures for scientific and engineering applications.
In the field of robotics, CSODEs have the potential to change the design and control of robotic systems. They can be applied to disturbance observation, pose estimation, and the design control strategies. By learning the dynamics of disturbances, CSODEs enable robots to accurately estimate their states and make appropriate compensations in the face of environmental perturbations. For pose estimation, CSODEs can learn the pose dynamies directly from sensor data, establishing adaptive pose estimators that enhance accuracy and robustness complex environments.
B Further Technical Details for Section
B.1 Further Implications of Assumption 1
With reordering of nonlinearities and their decomposition, there exists index w E (0....,M. .M such that for all E (w. and E (1....,ke), lim
fi(v) = co. Also, there exists E [s..... M) such +-
that for all E {l,..., .). E k.). we have lim j f((rdrr- +oo Here, w implies that all 1-+100 0
nonlinearities are bounded. The sets with the upper bound smaller than the lower bound are regarded as empty sets, e.g., E (1,0} = 0 the case that w = O.
B.2 An Example of Activation Functions Satisfying Assumption 3
In practice, Assumption 3 can be verified based chosen fa (e.g., RcLU, Sigmoid, tanh). For example, for the tanh function,
e" f,(x) e" ee-=
15,()) <I 1f(()) f((()l 1= vl.
is Lipschitz continuous with constant and satisfics:
p)(x,{) p5(,, <) < E WT W3E. ((W5)) 5(W53) S.
this case, we can choose:
= If S4 St HH. H3, H > 0. the restrictions are relaxed, verifying that Assumption is less strict than Lipschitz continuity.
S = Hld T
= - = -|
B.3 Verification Experiment on Assumption 3
To verify the actual existence of matrices in Assumption 3
we designed supplementary experiment to
validate the trained CSODE models, involving three dynamical system time series prediction tasks in the main
13

---

experiments. Retrained CSODE models (3 fully-connected layers. 128 units, Softplus: Adam optimizer. 500 epochs, learning rate =10- batch=64, MSE loss). We randomly selected 500 sample pairs (a' .y from each task's test set, calculated the difference e = s' f(=') where f()) is the trained CSODE model, and used YALMIP to define optimization variables and constraints.
constructed 500 matrix inequalities (5+) PC' <
and added the semi-definite constraint P P 2 0. Using the Sedumi solver (precision 10 60-6, max 5000 iterations), we solved the optimization problem and recorded the solving time and iteration count. We then
performed individual validation for each sample pair to calculate the satisfiability ratio of Assumption 3
Experimental Results (average of 3 independent experiments): For the Hindmarsh- Ross model, solving time was 15.3 minutes with 2684 iterations and 99.8% (499/500) satistiability. The Deaction-Diffusiin System task took 18.7 minutes with 3,217 itcrations and 99.6% (498/500) satisfiability. The Shallow WaterEquations required 24.2 minutes with 40505 iterations and 99.4% (497/500) satisfiability. The solver quickly found solutions satisfying Assumption B with >99% satisfiability across tasks, supporting our theoretical assumptions for trained models.
B.4 Formulations of Matrix Inequalities in Theorem I
The lincar matrix inequalities in Theorem D are formulated as follows.
M
P+
A 0; Q= = Q 0; =
T Uu
Σ
Ysr O.
(6)
f=1
1=1
a=1
am1 Pmp+1
M
P +
N > 0; Q = σ-
~0;
_ -
Σ
5
+
> Hb
W, 2 0,
T3 7S
0HL 2 0:
1=1
j=1
j=1
2 7
0HL 2 0:
f 7 OHS 2 0:
M
M
M
E" W
7 > S8 + Σ Hb
W₂+ + Σ
r 1 0Hl + S2 -83 -
j=1
j=1
M Σ
Yúr -
-
7S3""
OHE > 0,
1=
where
Q1,1 A. P +E; QALAI
A, &'WW,, +3; = A.W,JA A" W.W. Wo: Q1941 PA, + A W N + Togi -
QI.M+2 = P: QM+2.M+2 = - -42
Q1, AT P- + PAo =5: Q22= Q1 PA+I; Q1 A Ω
22 A T; Qa.3 -01; A= [A ... Au ]: r= WTT,, ... W.TM ]:
A WTT A WTAM 1: 1 -00000000 Wr WT 1 f (W, WyT,,, W.)
C Notation, Definitions, and Proof
C.1 Related Notation
The symbol R represents the of real numbers, RR+ E 22 0}, and RD denotes the vector space of n-tuple of real numbers. The transpose of a matrix A E 12"x" is denoted by AT. Let I stand for the identity matrix. The symbol ||-I] refers to the Euclidean norm on R².
For a Lebesgue measurable function R + R. detine the norm (lul(()) .2) sup [[t]] (((()) for (f1.12) C R. We denote by LP3 the space of functions with || ==|uul(-ox,, < +
A continuous function a : R+ + R2 belongs to class X if it is strictly increasing and a<0)) = 0, and X means that a is also unbounded. A continuous function 3 : RR x R4 + R4 belongs to class X L if
8(,r) E K and A(r, -) is decreasing to zero function for any fixed > O.
C.2 Related Definitions
Definition 2 The system ① is called forward complete for all zo E R" and u E Lr, the solution 2(t, 2o, u) is uniquely defined for all 2 0.
14

---

Definition 3 A forwward complete system ① with the output v(() = E(L) is said to be input-to-state stable (ISS) if there exist 3 H L and y H such that ||=((, no, u)ll S 8 (1]=o)). 1) ((llullo)) VL 1
for any =00) un E R" and u E L
iateeindepnndent toputtt-output stable (SI1OS) there exist B X L, r H such that
llu(t, zo, u)ll & ((1E(0))), ((lull)))) A1 220 0
for any E R and u E Lx
C.3
Proof of the Convergence Theorem
Proof of Theorem I The proof developments are divided into two main parts: ISS and Global Asymptotic Stability (GAS) guarantees, as shown in the sequel.
ISS analysis of ①T for boundedness
Consider Lyapumov function
Wiz
V(s) = = +2
fj(d)ds,
(8)
where the vector W the row of the matrix Wj. is positive definite and radially unbounded due to Then, taking the derivative of V (z), one has
Finsler's 's Lemma under the condition (6) and Assumption 7
V. Px +x Px
P$ + 2
x W.
JVJA7 A (WWF)
==1
M
= x (AT (A0 P PAu)
Σ (() AT
Px+x PP+ AW) +2x Pg(u)
x
M
+2
puu) W
f. Ao
W. (3
Therefore, under the condition
6 we obtain
2
x
(Wi)))
fi (Wiz)
V.
: Jar (w))))) g(u)
(w)) s(u)
M
- Σ ('W
Σ x TT w, (W5z)
2 M-1
Σ
S.W..)) W. (.W.).T.Tn W.T., W.f.(W,)) g()) @g(())
r=st1
M
< T
Σ
x T I To,s.,
M
2
Σ fe(W.)) w. W.T., w.f.(wz)) g(u) (g(())
r=st1
S -o(VV g(u)) @g(u):
for function Hoc. Under Theorem of (24), one can verify the first condition the ISS property due to the form of V, and the second relation can be recovered via V 2 a- (2g(u) 4g(u)) + V < -})((V)) This means that the ISS property of the NN is guaranteed, and the boundedness its solution.
15

---

Global attracting solution of the error dynamics unbounded (for the variable {) function
2
Consider another positive definite and radial
kg Σ
Wis f)()dd..
V(E) = =
PE
N
==1
Similarly, taking the time derivative of V::
(1(
(i(z,))
(ar(a,E) S((Wi)
p S((WE)
(W
(W
M
Σ ()( (F, 0 Σ S (W5E))(GG)))
E = - -
$ [TW.TTPpsx.8) - =
E (2
M M
Σ Σ py (x, W W W. (WWE).
5=1rm1
Then, under the condition
and Assumption 3 it can be deduced that
V $
Σ p,(c,t) (,c,
- - - SWW (2s((()
j=1
M
M
2
E WW, (2
2
(s( E) W, W, W.F.(W.)
==1
1=1
M
E
E W, 7
Σ
S
Σ
Hb
W/ E
1=1
f=1
E WW, (T5 - 0HT) (,(
2 E WW, (0, (0,- -HH3 f((W
2 P)(eet) w, Σ -0H3" W,J W.J.(W.EE.
- OHH"')
Therefore, with the conditions we substantiate that the system 1 SHOS with respect to &, meaning that the solution is GAS. This completes the proof.
D Experiment Setting Details
This section provides a detailed overview of the parameters used in our experiments to ensure reproducibility. Detailed settings for each modeling task are specified in the corresponding tables: Table for the Hindmarsh- Rose model, Table for the Reaction-Diffusioo system, and for the Shallow Water Equations.
Each table includes configurations such physics parameters, the ML. network structure for function f(-). optimizer, loss function, learning rate, and training epochs, among other parameters.
E Experimental Statistical Analysis
This section shows on the statistical methods employed to ensure the reliability and significance of our experi- mental results, in accordance with the required scientific standards. We focus on the computation of standard deviation percentages measure of variability and consistency across experimental runs in Experiments.
16

---

Table 4: Experimental Parameters for Modeling the Hindmarsh- -ose System
Parameter
Description
Parameter Values (a, C, To,,
1.0, 3.0, 1.0, 5.0, 0.5, 0.5,, Three fully connected layers, 1024 hidden dimensions each Adam
Network Structure
Optimizer
Loss Function
Mean Squared Error
Learning Rate
0.001
Training Epochs
1000
Table 5: Experiment Parameters for Modeling the Reaction Reaction-Diffusion System
Parameter
Description
Diffusion Coefficients (Dv, Dv)
0.15 0.17, 0.05 0.10
Reaction Rate Parameters (j, k)
0.035, 0.065
Spatial Domain
2.5 meters
Grid Resolution
50x50
Main Network Structure
Two fully connected layers, 2048 hidden dimensions each
Optimizer
Adam
Loss Function
Mean Squared Error
Learning Rate
0.0005
Training Epochs
1000
Table 6: Experiment Parameters for Modeling Shallow Water Equations
Parameter
Description
Gravitational Acceleration
9.8 m/s²
Water Depth
meter
Spatial Domain Length
meters
Grid Resolution
50x50
Main Network Structure
Two fully connected layers, 2048 hidden dimensions each
Optimizer
Adam
Loss Function
Mean Squared Error
Learning Rate
0.001
Training Epochs
1000
E.1
Statistical Methods for Reliability Analysis
To assess the reliability of experimental setups, the standard deviation for each performance metric was computed based on multiple runs. The formula for standard deviation is:
24
2)2
(9)
N
In the formula, represents the standard deviation, denotes individual observations, is the of these observations, and the total number observations.
The percentage standard deviation calculated to provide a relative measure variability:
Ø
Percentage Standard Deviation
x 100%
(10)
This measure serves our error bar, which reflects the extent of variability due to experimental conditions such model initialization and parameter settings.
17

---

E.2 Statistical Measures in Experiments
As shown in Table the vast majority of the standard deviation percentages are below 10% for experiments in Section indicating a high degree of statistical robustness across most metrics. This consistency underscores the reliability of our experimental design and the effectiveness of our model implementations.
Table 7: Standard Deviation Percentages for each metric across models. All values represent the standard deviation as a percentage (%) of the mean for each metric.
Group
Model
Hindmarsh-Ron
Reaction-Diffusion
Shallow Water
MSE
MAE
R
MSE
MAE
CD
MSE
MAE
CD
MLP
8.86
9.51
7.45
6.86
4.20
6.94
5.44
8.54
5.57
A
RNN
7.63
9.26
9.70
8.79
13.59
9.58
7.52
8.45
8.06
NODE
5,16
4.02
8.14
4.43
5,32
4.76
4.73
4.38
3.42
ANODE
3.48
4.06
4.46
3.75
4.57
4.43
3.79
4.68
4.29
SONODE
4.46
5.12
4.99
4.55
5.60
4.91
4.84
5.82
4.97
CSODE
4.41
3.24
4.28
5.81
5.26
4.75
4.84
7.25
4,98
C
AdapE-Adapt
3.60
4.64
4.24
4.65
6.53
4.55
2.61
5.51
4.74
The same statistical methods employed in Section are also utilized to analyze the Model Scaling experiments, outlined in Section 7 The heatmap in Figure displays the standard deviation percentages for each metric, all of which remain below 25%, indicating reliability in scaling experiments well. Additionally, in Section | the shaded areas around the loss curves further illustrate the variability of our results. These shadows in the plots provide visual representation of the variability, akin to error hars, showing fluctuations within a confined range. In Table the test error rate variability is maintained within tight range of 0.005%.
E.3 Statistical Considerations in Models Performance Comparison
In study, we have chosen straightforward approach presenting model performances directly through metrics in SableWiithee deploying traditional statistical tests such ANOVA t-tests. This decision was based on the clarity and immediacy with which the performance metrics convey the effectiveness of each model. By doing so, aim to keep the main article succinct and focused on substantive evaluations, thereby avoiding the potential complication of statistical tests that might obscure the main findings. This method ensures that the paper remains accessible readers, emphasizing practical implications.
F Experimental Result of Integrating CSODEs with Transformer Layers
To compare the performance of Transformer, TLODE, and CSTLODE models on time series prediction tasks, we conduct experiments on three datasets: Hindmarsh-Roon Model, Saaction-Diffusion Model, and Shallow Water Equations. The experimental results shown in Table It can be observed that TLODE and CSTLODE overall outperform the original Transformer model, indicating that combining the Transformer with Neural ODE can enhance model performance on time series prediction tasks.
Table 8: Performance comparison of Transformer, TLODE, and CSTLODE on Hindmarsh- Model, Reaction- Diffuusinn Model, and Shallow Water Equations datasets. The best and second best results are highlighted in blue and brown, respectively.
Model
Hindmarsh-Ron
Deaction-Diffusion
Shallow Water
MSE
MAE
R²
MSE
MAE
CD
MSE
MAE
CD
Transformer
1.673
0.673
0.417
1.573e-2
0.115
3,003
0.081
0.23
3.01
TODE
0.731
0.488
0.623
7.346e-3
0.051
0.723
0.031
0.16
1.90
CSTODE (Ours)
0.411
0.363
0.867
4.341e-3
0.033
0.655
0.033
0.15
1.91
Further comparing TLODE and CSTLODE, we find that CSTLODE achieves better results on most evaluation metrics. This suggests that by introducing the control term 9()), the ControlSynth framework can better guide the model in learning the dynamics of time series. Specifically, on the Hindmarsh Rose and Reaction- Diffusion datasets, CSTLODE's MSE, MAE, and CD indicators significantly better than those of TLODE. On the
18

---

Shallow Water Equations dataset, the performance of the two models is more comparable, with TLODE slightly better in MSE and CSTLODE slightly better in MAE.
Overall, the experimental results validate the effectiveness of combining the ControlSynth framework with Transformer. The introduction of the control term g(-) can further enhance model performance. This provides new approach to the field time series prodiction. In the future, we plan to apply and validate the combination of yontrolSynth and Transformer in more practical scenarios. We will also explore the possibilities of combining ControlSynth with other advanced time series prediction models.
G Additional Qualitative Prediction Results
ahal Noaaa Anyreree
This plot compares the time series prediction of ground truth, NODE, ANODE, and CSODE models for the Hindmarsh-Roo Rose neuron dynamics.
Sati
(a) Hindmarsh- Rose neuron dynamics
Gny-Sea Mode of Reaeton Dettuoon, U (dmermzttzo))
Inttal Londiton Phodetion Leornd Iruhh
Initial conditions and later system states for the dynamics, showing predictions from NODE, ANODE, and CSODE alongside ground truth.
Neuni ODE
Angnented Neunl ODE
ConnelSynt Nerral ODE
(b) Reaction- Diffusion system dynamics
2D Shallow Equatoas, (Metss)
Displays initial conditions and predictions by NODE, ANODE, and CSODE for Shallow Water Equations, compared with the ground truth.
Neadl ODE
Auginentad Neara ODE
Comollyyrh Neard ODE
M
(c) Shallow Water Equations
Figure 7: Visual comparison of neural network predictions on different dynamic systems.
19

---

In this section, we present additional qualitative results designed to enhance the reader's s understanding of the complex lasks modeled and the performance of various models, specifically focusing NODEs, ANODEs, and CSODEs. These visual comparisons, as illustrated Figure 7 provide a clear and intuitive view of each model's capability to predict and simulate the dynamics of complex systems.
The visual results are instrumental in demonstrating the capabilities of the CSODEs, particularly its adeptness in capturing the intricate behaviors of dynamic systems more effectively than traditional models. This is evident from the accurate time series predictions and the fidelity in spatial-tempore dynamics shown across different scenarios- from neuronal activity in the complex nonlinear Hindmarsh- Rasse model to fluid dynamics Shallow Water Equations, and the chemical kinetics in Drfftion-iiffusionn processes.
H
Further Results on Model Scaling
H.1
Comparison with Neural ODE and Augmented Neural ODE
We conducted comprehensive experiments comparing CSODE, original NODE, and ANODE under equivalent parameter counts achieved through width scaling. Taking the Reaction-Diffusin task as an example, with CSODE configured with subnetworks, we systematically evaluated model performance across increasing network widths (128, 256, 512, 1024, 2048). TablesFandd 10 present the MAE and MSE metrics respectively.
The results demonstrate consistent pattern: while all models show improved performance with increased width, CSODE maintains a clear advantage across all scales. At smaller widths (128-256), the performance gap particularly notable, with CSODE achieving an MAE of 0.060 compared to NODE's 0.083 and ANODE'S 0.078 at width 128.
As network capacity increases, CSODE's advantage becomes even more pronounced. At larger widths (1024- 2048), while NODE and ANODE show signs of performance saturation (MAF stabilizing around 0.06), CSODE continues to improve, reaching MAE of 0.030 width 2048. This suggests that CSODE makes more effective use of additional parameters, likely due to its structured approach to handling dynamical systems.
Table 9: MAE loss comparison for CSODE, NODE, and ANODE models with different widths the Reaction- Difffusin task. Lower values indicate better performance.
Width
CSODE
NODE
ANODE
128
0.060
0.083
0.078
256
0.056
0.063
0.061
512
0.045
0.058
0.055
1024
0.037
0.061
0.057
2048
0.030
0.063
0.057
Table 10: MSE loss comparison for CSODE, NODE, and ANODE models with different widths in the DifetiionDiifusion task. Lower values indicate better performance.
Width
CSODE
NODE
ANODE
128
0.0036
0.0069
0.0061
256
0.0031
0.0040
0.0037
512
0.0020
0.0034
0.0030
1024
0.0014
0.0037
0.0033
2048
0.0009
0.0040
0.0033
The MSE metries further corroborate these findings, showing that CSODE achieves consistently lower error rates across all width configurations. The improvement in MSE is particularly notable at larger widths, where CSODE achieves an MSE of 0.0009 at width 2048, significantly outperforming both NODE (0.0040) and ANODE (0.0033). This suggests that CSODE not only performs better in terms of absolute errors but also shows superior stability in handling larger variations in the prediction task.
These results highlight CSODE's superior scalability and efficient parameter utilization, demonstrating that its architectural advantages persist across different model capacities. The consistent performance improvements with increased width also suggest that CSODE may benefit from even larger model scales in more complex applications. Furthermore, the results indicate that the improvement in the learning ability of these two models with increasing width is not significant CSODE.
20

---

H.2 Convergence Study in ControlSynth Neural ODE Model Scaling
the context of scaling the CSODE framework, the impact of network configurations such as subnetwork count and width on model performance is critical. Several key observations are performed:
Loss for Width 512 with Different Sub-network Numbers
Loss for Sub-network with Different Widths
0,025
Submet Train 0.025
Widih 128 Train
Suboct Valid
Width 128 Valid
Suhmer
Submet
0.020
256
0.020
Submet
Train
Widhh 512 Valid
0.015
Submct
Subner Valid
Lask
0.015
Width 1024 Tminn
Width 1024 Valid
Submer Train
Width 2048. Train
Submet 5 Valid
Width 2048 Valid
0.010
0.010
0.005
0.005
200
1000
200
1000
Fipcchs
Epochs
Figure 8: The left figure shows the loss curves for different sub-network numbers at a fixed width (512), and the right figure displays the loss curves for different widths with a fixed sub-network number (3), reflecting how changes in configuration impact model performance.
When the width is held constant, for instance at 512, an increase in the number of sub-networks initially raises the loss due the augmented complexity of the network as the left figure Sigwre However, as training advances, the model effectively adapts to this increased complexity. ultimately achieving lower level of loss than configurations with fewer sub-networks. This indicates that higher number of sub-networks can enhance the model's 's depth of learning and capability for feature extraction, leading improved performance after extensive training.
Increasing Sub-network Count at Fixed Width:
With the the width shows marked improvement in learning capabilities, the right figure in Figure [ shows. The additional parameters afforded by a broader network aid in capturing more complex features of the data, which accelerates the reduction in both training and validation losses. This setup not only speeds up the learning process but also tends to improve generalization across different tasks.
Increasing Width Fixed Sub-network Count:
count fixed, such as three, expanding
Overall Convergence Trends: Whether increasing the sub-network count the width, all configurations eventually exhibit trend toward convergence in losses. This suggests that, with suitable adjustments training duration and parameter settings, various network configuration designs of CSODEs are capable uncovering and effectively learning the intrinsic patterns hidden in the data.
Summary: These observations align well with the goals of the CSODE framework, which is designed to enhance NODE system scalability while ensuring strong convergence and high performance.
I Solver, Optimizers, and Hyperparameters
In the training and evaluation of neural differential equation models, the selection of solvers, optimizers, and hyperparameters plays pivotal role in determining both the performance and efficiency of the models. This section delves into the theoretical underpinnings of these components and provides an in-depth analysis of their impacts through comprehensive experimental studies.
1.1 Impact of Solver Choice
Numerical solvers are essential for approximating solutions to differential equations. which the backbone of Neural Ordinary Differential Equations (NODEs) and their variants. The choice of solver affects both the accuracy and computational efficiency of the model. Solvers can be broadly categorized into explicit and implicit methods, with further subdivisions based on their order of accuracy and adaptive step-size capabilities.
21

---

The Euler method is first-order explicit solver known for its simplicity and computational efficiency. Despite its low accuracy, serves a baseline for comparison due its straightforward implementation and ease of understanding.
Dopri5, an explicit Runge-Kutte method of order a higher-order adaptive solver that adjusts step sizes based on the estimated error, providing a balance between accuracy and computation time, Adaptive solvers like DopriS can handle stiff equations more effectively by modifying the step size dynamically, thereby improving numerical stability.
L1.1 Experimental Setup
To evaluate the impact of solver choice, we conducted experiments on multiple tasks, including the Reaction- Diffusion model, which is known for its complex dynamics and sensitivity to solver accuracy. The models compared include:
NODE: The standard Neural Ordinary Differential Equation model.
ANODE: Augmented Neural ODE, which adds additional dimensions to improve model expressive- ness.
CSODE:
Our proposed ODE model with composite sub-network, is designed enhance stability and performance.
We implemented both Euler method and the Dopris solver within each model architecture.
1.1.2
Results and Discussion
Table 11: Mean Absolute Error (MAE) Comparison between Euler and Dopris Solvers on
Reaction-Diffusine Task
Model
Euler MAE
Dopri5 MAE
NODE
0.073
0,071
ANODE
0.063
0.062
CSODE
0.033
0.032
Table IT presents MAE for each model using both Euler and DopriS solvers. results indicate that while DopriS offers slight improvement in MAE across all models, the relative performance ranking remains consistent. Specifically, CSODE maintains its superior performance regardless of the solver used, underscoring its inherent robustness.
Computational Overhead: As expected, Dopri5 incurs higher computational cost compared the Euler method. In our experiments, Dopriš increased training time by approximately 35% relative to Euler. However, the trade-off between computational cost and marginal accuracy gains must be carefully considered, especially in scenarios where real-time inference is critical.
Stability Considerations: Higher-order solvers like Dopriš exhibit better stability properties, particularly in handling stiff differential equations. Although our primary tasks did not involve highly stiff systems, the use of Dopri5 beneficial in extending the applicability of these models more complex dynamical systems.
I.2 Impact of Optimizer Selection
Optimizers are algorithms or methods used to adjust the weights of neural networks to minimize the loss function. The choice of optimizer can significantly influence the convergence speed, the quality of the solution, and the model's ability escape local minima.
Adam is widely used optimizer that combines the benefits of both AdaGrad and RMSProp. It maintains per-parameter learning rates adapted based on the first and second moments the gradients, making it suitable for problems with sparse gradients and noisy data.
is a quasi-Newton optimizer that approximates the inverse Hessian matrix to guide the search direction. It is particularly effective for problems where high precision in convergence is desired, albeit the cost of higher memory usage and computational overhead.
L-BFGS (Limited-memory
Goldfaer-Gollfarb--
I.2.1 Experimental Setup
To assess the impact of optimizer choice, we extended our experiments by incorporating the L-BFGS optimizer alongside Adam. The models evaluated include NODE, ANODE, and CSODE across all primary tasks, with the
22

---

Reaction- Disifon--ifffooon model serving the exemplar case. The hyperparameters for each optimizer were carefully tuned ensure fair comparison:
Adam: Learning rate 0.001, 31 = 0.9, A2 0.999, 10
L--BGG: Maximum iterations = 1000, tolerance = 10-9.
1.2.2 Results and Discussion
Table 12:
Mean Squared Error (MSE) Comparison between Adam and L-BFGS Optimizers on the
Reaction- Diffusion Task
Reaction-Diffusion
Model
Adam MSE
L-BFGS MSE
Improvement
NODE
1.134e-2 1.095c-2
9,536e-3
15.9%
ANODE
9.153c-3
16.4%
CSODE
6.365e-3
5.321c-3
16.4%
Table 12 illustrates the performance gains achieved by switching from Adam L-BFGS across different models. On average, L-BFGS improves the MSE by upproximately 16%, with CSODE benefiting similarly to NODE and ANODE. Notably, while all models experience performance enhancements, CSODE consistently outperforms the other variants, reinforcing its superior architecture.
The quasi-Newton nature of L-BFGS allows it navigate the loss landscape more effectively, reducing likelihood getling trapped in shallow local minima. This property is particularly advantageous for complex models like CSODE, which possess intricate loss surfaces due to their composite structure.
Robustness Local Minima:
Memory and Computational Trade-offs: While L- L-BFGS offers superior convergence properties, it demands significantly memory and computational resources, especially for large-scale models. In experiments, L-BFGS required approximately twice the memory footprint compared to Adam increased the per-iteration computation time by 40%. Therefore, the choice hetween Adam and L.BBFG should consider the available computational resources and the specific requirements of the task at hand.
1.3 Impact of Hyperparameters Adjustment
Hyperparameters such learning rate batch size are critical in training neural networks. The learning rate determines the step size during the optimization process, directly intluencing the convergence speed and stability. A learning rate that is too high can cause the optimization to overshoot minima, leading to divergence, while rate that too low can result in excessively slow convergence or getting stuck in local minima.
Batch size affects the gradient estimation's is variance and the computational efficiency. Smaller batch sizes introduce more noise into the gradient estimates, potentially aiding in escaping local minima but making convergence noisier. Larger batch sizes provide more accurate gradient estimates but may lead poorer generalization and require more memory.
I.3.1 Experimental Setup
We conducted a systematic hyperparameter search to examine the sensitivity of CSODE to changes in learning rate and batch size. The ranges explored were:
Learning Rate: [0.0001, 0.0005, 0.001, 0.005, 0.01]
Batch Size: [32, 64, 128, 256)
Each combination of learning rate and batch size was evaluated on the CSODE model using the Reaction- Diffusion task. Performance was measured in terms of MAE on the validation set, and each setting replicated three times account for stochasticity.
Additionally, investigated the impact of network architecture parameters, specifically network depth and width, understand their influence on CSODE's performance main text.
by CSODE across various learning rates batch sizes. The results reveal that CSODE demonstrates strong robustness to hyperparameter variations within certain ranges.
Learning Rate Sensitivity: Within the learning rate range of (0.0005, 0.005], CSODE's MAE remained within a 5% fluctuation band, indicating stable performance. Specifically, at fixed batch size of 128, varying the learning rate within (0.0005,0 0.005] resulted in standard deviation of only 1.2% in MAE. However, at learning rate of
23

---

0.01, the model exhibited signs of instability, likely due gradient explosion, while at 0.0001, convergence was significantly slower, potentially leading to gradient vanishing issues.
Batch Size Influence: CSODE maintained consistent performance batch sizes introduced minor varianees but did adversely affect the overall performance, suggesting that CSODE can be effectively trained under different memory constraints.
batch sizes from 32 256. Smaller
The increased network complexity necessitates careful hyperparameter tuning to maintain gradient stability. Ar higher depths and widths, the model becomes more susceptible to gradient vanishing or explosion. Therefore, selecting an appropriate learning rate crucial to ensure that gradients remain within reasonable range, facilitating effective training.
Gradient Stability:
I.3.2 Recommendations for Hyperparameter Tuning
Based experimental findings, propose the following guidelines for tuning hyperparameters in CSODE:
Learning Rate: Utilize learning rate within [0.0005, 0.005] range to achieve stable efficient convergence. Employ learning rate sehedulers or adaptive methods to fine-tune within this range dynamically.
Batch Size: A batch between and 256 is recommended. Smaller batch sizes can be used memory constraints are present, without significant degradation in performance.
Optimizer Selection: For scenarios requiring high precision and better convergence properties, L.- BFGS is preferable. However, for large-scale real-time applications, Adam remains viable choice due to its efficiency.
J Ability to Adapt to Different Spatial Scales
practice, CSODE demonstrates superior robustness when handling dynamical system datasets across different spatial scales, as evidenced by the stability of performance metrics. Table [T] preseentts a comprehensive comparison of the model's stable ability across different spatial scales in the Reaction-Diffusioo task. better stability. Best performing models are marked in blue, second- brown.
Table 13: Performance stability comparison across different spatial scales in the
model. The percentages show the standard deviation of model performance. Lower values indicate
Spatial Scale
NODE
ANODE
CSODE
Original Scale
5.3%
4.6%
5.3%
5x Scale
7.8%
7.1 %
5.5%
10x Scale
8.8%
7.3%
5,5%
While all models show comparable performance in the original scale setting (NODE: 5.3%. ANODE: 4.6%. CSODE: 5.3%), significant differences emerge when spatial domain is expanded. times the original scale, NODE and ANODE show notable degradation in stability (7.8% and 7.1% respectively), while CSODE maintains relatively stable performance (5.5%). This trend becomes even more pronounced at 10 times the original scale, where NODE and ANODE further deteriorate to 8.8% and 7.3%, while CSODE maintains its stability 5.5%.
Particularly noteworthy is stability drop from the original to 10x scale: CSODE shows only a 0.2% increase standard deviation, compared to significantly larger increases NODE (3.5%) and ANODE (2.7%). This remarkable stability across different spatial scales suggests that CSODE's architecture, particularly its control term component, provides inherent advantages in handling spatial scale variations in dynamical systems.
These results indicate that CSODE not only performs well in standard settings but also maintains consistent performance across varying spatial scales, making it particularly suitable for applications where spatial scale invariance is crucial.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have clarified the paper's contributions and scope in both abstract and introduction.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of this work at the end of conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should refiect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions and proof can be found in Section 4 and Appendix C
respectively.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
25

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In this study, we provide detailed descriptions of model architectures. experiment- specific configurations, simulation parameters, and hyperparameters Sections [. and D along with comprchensive dataset information Section 66.2 Additionally, the scaling of model structures and hyperparamcters is thoroughly elaborated in Section 7 ensuring that the reader can accurately replicate our experimental results and verify the statements.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language madel), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that repreducibility may he tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-soure models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Our Gitllub repository URL., provided in Section houses all necessary source code, detailed installation instructions, and step-by-ster guides for setting up the environment running experiments, ensuring replicability. We provide comprchensive data generation algorithms for synthctic datasets, which collectively facilitate the reproduction of experimental results detailed in this study.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines http:://nissccccpull//
quuiees/fooeeSuuniesssooooooooo for more details.
26

---

While we encourage the release of code and data, we understand that this might not be possible. "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (httpe: htap:://nips.. cc wttpp/ccccppbbllccc p
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs data and code permitted.
To preserve anonymity, the source code has heen hosted on dedicated GitHub account created specifically this submission, which does not contain any other repositories or identifiable information.
6. Experimental Setting/Detann
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The detailed information training and testing is thoroughly presented in Scction and in Appendix [ including data splitting methods, the basis for hyperparameter selection, types of optimizers, and other experimental configuration details, ensuring a complete understanding of the results and the reproducibility of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our research assesses the reliability of experimental results using standard deviation percentages as error bars, clearly explaining the calculation formulas and statistical significance. Appendix lists the standard deviation percentages for all performance metrics from experiments, comprehensively demonstrating the statistical significance of the experiments.
E
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap. etc.)
The assumptions made should be given (c.g., Normally distributed errors).
should be clear whether the error is the standard deviation or the standard error of the mean.
is OK to report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have a 96% CI, if the hypothesis of Normality of crrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates).
27

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding ligures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Details on the GPU type and CUDA platform used across all experiments docu- mented in Section and Table highlighting the uniform computational environment. Due to the consistency in resource use across different Neural ODE models tested under similar conditions, specific compute requirements for individual experiments are not listed separately, ensuring that all necessary information for replication is efficiently conveyed without redundancy.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with the NeurIPS Code of Ethics httph///p///eeerr....//bbbb///ba httt:///ww..ccccccccc......a.aa.
Answer: [Yes]
Justification: Our research adheres to NeurIPS ethical guidelines through use of synthetically generated data, ensuring no risk to privacy, and by sharing all methodologies and code to promote reproducibility and transparency. We ensure fairness in algorithmic design, respect for intellectual property, consider the societal impacts avoid harm, and maintain anonymity
submissions.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix A for the details of broader impacts of this work.
Guidelines:
The answer means that there is no societal impact of work performed.
the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation. generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out a generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
28

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: have not used data models that have high risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The related assets have been suitably credited and mentioned.
Guidelines:
The answer NA means that the paper does not existing
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppppppaapittatee www.reewthhhc..... cww. datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
existing datasets that are re-packaged, both original license and license of derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: assets utilized and created are synthetic or derived from open-source resources, requiring no consent from third parties.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/modde as part of their sub- missions via structured templates. This includes details about training. license. limitations.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
29

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our experiments are not crowdsourcing this work irrelevant to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our experiments are irrelevant to such risks.
Guidelines:
answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
30


Reviews:
[{"review_id": "mSbVc33sYi", "reviewer": "Reviewer_JYd3", "summary": "In this paper, a method called ControlSynth Neural ODEs is proposed. This method is essentially defined as a neural ordinary differential equation with a control input. In particular, the authors focus on the convergence, of which definition requires the existence of a solution and also the asymptotic stability of the solution. In this paper, theoretical conditions to satisfy the convergence are presented. Some numerical experiments are also provided.", "questions": "As it is stated that the convergence is guaranteed in this paper, I imagine that there must be some tricks in designing the control input and the learning algorithm so that the conditions for the convergence are always satisfied. How are they designed?", "limitations": "There seems to be no problem.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In my opinion, the main contribution is the theoretical analysis of convergence. In fact, stability is very important for the use of neural ordinary differential equations, which are often unstable when used for modeling.", "weaknesses": "In this paper, conditions for convergence are presented. Thus, it is expected that the control input and the neural ordinary differential equations should be trained so that these conditions are satisfied. However, I could not find any description of how to achieve this in the paper. In fact, the conditions for the convergence assume the existence of several matrices, and it seems difficult to design an optimization algorithm for training while satisfying these conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful reviews of our work and for recognizing our study on convergence analysis. We sincerely appreciate your review comments and have carefully analyzed and responded to them, conducting a relevant experiment:\n\n1. **Regarding the existence of special optimization algorithms for training:**\n\nWe acknowledge your concerns about satisfying convergence conditions in training algorithms. Our paper's main focus is proposing the CSODE models based on theoretical convergence analysis, rather than designing new learning algorithms. For fair comparisons, we used the widely adopted Adam optimizer with MSE loss. This uniform approach highlights the strengths and weaknesses of different model structures, avoiding bias from varied optimization algorithms. We believe this comparison method is fair and reasonable.\n\nWe agree designing new learning algorithms is a promising future research direction, but it is beyond our current paper's scope. Our focus is on model theory and structure, instead of algorithm improvement. In the Conclusion, we suggest this as a future research topic.\n\n2. **Regarding the design of control inputs:**\n\nIn our work, we used the state variable $x_0$ as a placeholder for $u_0$ to ensure fair comparison with baselines, as original neural ODEs and variants lack control inputs. We believe this is a reasonable approach. In future works, more refined methods can be considered under the existence of control inputs in the CSODE framework.\n\nFrom a theoretical perspective, the design or the form of control input $u$ does not affect convergence property. In the error dynamics equation (2), the term $u$ is eliminated. Theorem 1 also does not restrict $u$. The only restriction for $u$ is given after Equation (1), not in Theorem 1. In the current version, we do not include the design of inputs as our main attention is on the convergence guarantees under the most basic form.\n\n3. **Regarding the way of applying the convergence conditions:**\n\nFirst, Theorem 1 directly relates model parameters to convergence through matrix inequalities. Inserting only the trained parameters into those inequalities verifies convergence without checking assumptions or matrix inequalities in the training processes. This provides a more concrete, operational convergence criterion. Note that the three assumptions only have a claim on the activation functions of CSODE, but convergence is assured as long as the matrix inequalities hold (solutions of the matrix inequalities are only relevant to the weight matrices $A_0, A_1, \\dots, A_M, W_1, \\dots, W_M$). \n\nSecondly, we discuss how each assumption naturally holds:\n\n**Assumption 1:**\nAssumption 1 selects the activation functions passing through the origin and the quadrants I and III. Remark 1 actually states all cases of the boundedness of the activation functions: none of, part of, or all of the nonlinearities $f_s^i$ are bounded. Also, both the unboundedness and boundedness of $\\lim_{\\nu\\rightarrow\\pm\\infty} f_s^i(\\nu)$ may lead to the unboundedness of $\\lim_{\\nu\\rightarrow\\pm\\infty}\\smallint_{0}^{\\nu}f^{i}_{s}(r)dr$.\n\n**Assumption 2:**\nAssumption 2 aligns with CSODE's structure, reflecting continuity and monotonicity of activation functions. This relates to model dynamics and is satisfied by most common activations.\n\n**Assumption 3:**\nWe describe the rationality of Assumption 3 from three perspectives:\n\n- The positive semi-definite diagonal matrices in Assumption 3 are mathematical solutions used to derive system stability conditions and do not need to appear explicitly in the model design.\n\n- The inequalities in Assumption 3 resemble local Lipschitz continuity or local quadratic boundedness. In practice, they can be verified based on chosen $f_j$ (e.g., ReLU, Sigmoid, $\\tanh$). For the $\\tanh$ function,\n  $$\n  f_j(x) = \\frac{e^x - e^{-x}}{e^x + e^{-x}}, \\quad |f_j(x)| < 1, \\quad |f_j(x)-f_j(y)| \\leq |x-y|.\n  $$\nIt is Lipschitz continuous with constant $1$ and satisfies:\n  $$\n  p_j(x,\\xi)^\\top p_j(x,\\xi) \\leq \\xi^\\top W_j^\\top W_j \\xi, \\quad f_j(W_j\\xi)^\\top f_j(W_j\\xi) \\leq \\xi^\\top W_j^\\top W_j \\xi\n  $$\nIn this case, we can choose:\n$$\nS_0^j = H_0^j = I, \\quad S_1^j = S_2^j = S_3^{j,r} = H_1^j = H_2^j = H_3^{j,r} = 0.\n$$\nIf $S_1^j, S_2^j, S_3^{j,r}, H_1^j, H_2^j, H_3^{j,r} >0$, the restrictions are relaxed, making Assumption 3 less strict than Lipschitz continuity.\n\n- To address your concerns about the actual existence of matrices in **Assumption 3**, we designed **a supplementary experiment** to validate the trained CSODE models, involving three dynamical system time series prediction tasks in the main experiments.\n    \n    Experimental Steps: Retrained CSODE models (3 FC layers, 128 units, Softplus; Adam optimizer, lr=1e-3, batch=64, 500 epochs, MSE loss). We randomly selected 500 sample pairs $(x^i, y^i)$ from each task's test set, calculated the difference $\\xi^i = y^i - f(x^i)$ where $f(\\cdot)$ is the trained CSODE model, and used YALMIP to define optimization variables and constraints. We constructed 500 matrix inequalities $\\xi^{iT} P \\xi^i \\preceq 0$ and added the semi-definite constraint $P \\succeq 0$. Using the Sedumi solver (precision 1e-6, max 5000 iterations), we solved the optimization problem and recorded the solving time and iteration count. We then performed individual validation for each sample pair to calculate the satisfiability ratio of Assumption 3.\n    \n\n    Experimental Results (average of 3 independent experiments): For the Hindmarsh-Rose model, solving time was 15.3 minutes with 2,684 iterations and 99.8\\% (499/500) satisfiability. The Reaction-Diffusion System task took 18.7 minutes with 3,217 iterations and 99.6\\% (498/500) satisfiability. The Shallow Water Equations required 24.2 minutes with 4,105 iterations and 99.4\\% (497/500) satisfiability. The solver found solutions satisfying Assumption 3 quickly, with >99\\% satisfiability across tasks, supporting our theoretical assumptions for trained models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CWJpW1Npe4", "reviewer": "Reviewer_tUtW", "summary": "The paper introduces ControlSynth Neural ODEs (CSODEs), a novel approach to modeling dynamical systems with neural ordinary differential equations (NODEs). The proposed models constraint the system to a convergent once. The CSODE framework incorporates an additional control term to ensure the existence of the solutions. The authors present theoretical guarantees for convergence and demonstrate the superior performance of CSODEs compared to other NODE variants in their experiments.", "questions": "In the preliminary experiments section, neural odes seems not to converge properly. Was an alternate optimizer like LBFGS or BFGS tried? In most cases where vanilla neural ODEs tend to not converge, it is more often than not an optimizer issue rather than a model issue.", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Experiments**: The authors validate their model on learning dynamical systems. This is a welcome change from several papers testing neural odes on problems like image classifications and such, where neural odes tend not to be the best solution. Hopefully more papers do the same!\n* **Technical Rigor**: The paper provides detailed theoretical analysis and proofs for the convergence of CSODEs.", "weaknesses": "Overall the paper looks solid. 1 questions regarding the training pointed out in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your positive evaluation and valuable feedback on our work. Your insights are crucial in enhancing the quality and rigor of our research. We are particularly grateful for your observation regarding the convergence issues of Neural ODEs, which prompted us to conduct in-depth supplementary studies.\n\nWe agree with your assessment that applying Neural ODEs to understand physical models is indeed a significant and promising direction. As you pointed out, while previous works often focused on tasks like image classification where Neural ODEs may not be the optimal solution, we believe that leveraging Neural ODEs for modeling dynamic systems is not only more natural but also holds profound implications. The continuous-time representation of Neural ODEs aligns well with many physical processes, and the ODE formulation often allows for better interpretability of learned dynamics, bridging the gap between data-driven and physics-based modeling.\n\n## Addressing the Optimizer Issue\n\nOur initial choice of the Adam optimizer was based on its versatility, aiming to provide a universally applicable baseline. In response to your suggestion about the optimizer, we have conducted comprehensive additional experiments:\n\n### 1. Optimizer Comparison\n\nFollowing your recommendation, we expanded our experiments to include the L-BFGS optimizer in addition to the original Adam optimizer. The results indeed show that L-BFGS significantly improves the convergence performance of Neural ODEs and their variants.\n\n### 2. Experimental Setup\n\nWe performed comparative experiments on Neural ODE, Augmented ODE, and our proposed CSODE across all main experimental tasks.\n\n### 3. Key Findings\n\nUsing the Reaction-Diffusion model task as an example, the MSE results after employing the L-BFGS optimizer are given as follows:\n\n| Model | Original MSE | L-BFGS MSE | Improvement |\n| --- | --- | --- | --- |\n| Neural ODE | 1.134e-2 | 9.536e-3 | 15.9% |\n| Augmented ODE | 1.095e-2 | 9.153e-3 | 16.4% |\n| CSODE | 6.365e-3 | 5.321e-3 | 16.4% |\n\nSimilar improvements were observed in other tasks.\n\n### 4. Result Analysis\n\nAlthough all models benefited from the use of L-BFGS, CSODE maintained its advantage over other methods. This not only validates your insight but also further confirms the effectiveness and robustness of CSODE.\n\nThe results suggest that the quasi-Newton characteristics of L-BFGS may be more suitable for addressing the gradient flow issues in Neural ODEs, and L-BFGS might be more effective in avoiding local optima compared to Adam in training Neural ODEs.\n\n### 5. Revision Details\n\nBased on these new findings, we did:\n\n- Briefly discuss the impact of optimizer choice on model performance in the main text;\n- Provide complete supplementary experimental results and detailed analysis in the Appendix;\n- Add a discussion about the importance of optimization strategies for Neural ODE-type models.\n\n## Regarding the Concern that Neural ODEs Seem Not to Converge Properly\n\nWe acknowledge that the curves presented in our paper indeed show some minor fluctuations towards the end, which may be due to insufficient epoch settings in our original setup. This observation reminds us to pay more attention to the stability of experiments and the presentation of results in future work.\n\nTo validate our results, we conducted additional experiments by extending the training epochs to 1.5 times the original number while continuing to use the original Adam optimizer. The results show that the Neural ODE model ultimately achieved training and test losses within 10% of those obtained under the original epoch settings. Moreover, the performance fluctuations in the additional epochs were contained within 5%.\n\nThese new experimental results indicate that our original experimental results are representative and the model is in a relatively converged state. Although the fluctuations at the end of the original curves might give the impression of incomplete convergence, the extended training time experiments confirm that the model indeed reached a stable performance level.\n\n## Conclusion\n\nWe sincerely thank you for your suggestion, which has not only helped us improve our current research but also provided valuable insights for future Neural ODE research. We believe these additions will significantly enhance the depth and impact of the paper.\n\nIf you have any further questions, or suggestions, or need clarification, we welcome your feedback and look forward to an in-depth discussion.\n\nThank you again for your valuable time and insightful opinion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B8mxbsP9UU", "reviewer": "Reviewer_Wqu1", "summary": "The authors present a new class of continuous-time neural networks, ControlSynth ODEs. This new class of ODEs are able to learn the dynamics of physical systems faster and more precisely. In addition the authors provide theoretical convergence guarantees for these new models, and demonstrate their effectiveness compared to traditional Neural ODEs.", "questions": "It would be beneficial to outline what architectural changes are made from standard NODEs to CSODEs.\nIn addition, to what hypothesis motivated these changes. \nWhy are they termed ControlSynth ODEs? The name is uninformative.\nWhat are the inductive biases of these models?", "limitations": "The limitations of the method are unclear. \nWhat are the inductive biases of these models?\nHow does sensitivity to hyperparameters emanate during training?\nWhy is this specific architecture more difficult to train than the alternatives?", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Although the analysis is difficult to follow, the theoretical convergence guarantees make these an attractive model.\nIn the benchmark experiments, adaptive CSODEs achieve are superior compared to the other models.", "weaknesses": "The explanation of the model is far too condensed and unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your time and effort. We highly value your invaluable suggestions and sincerely apologize for the lack of clarity in certain parts of the manuscript. Please allow us to further elaborate and analyze the key issues you have raised.\n\n**Regarding the Concept and Motivation for Introducing Control Terms in CSODE:**\n\nIn the field of Control Theory, control terms are typically used to regulate system states to achieve desired performance. Inspired by this, we introduced an additional control term to the traditional NODE to enhance the model’s ability to handle complex multi-scale dynamics. Specifically, the control term is generated by an independent sub-network, which can adaptively regulate the evolution of the system state. This design endows CSODE with greater flexibility and expressive power when learning and predicting the behavior of complex physical systems, such as those described by partial differential equations. Therefore, we named the model CSODE to highlight this inspiration. We have provided a more in-depth explanation of the motivation and mechanism for introducing the control term in the revised manuscript's introduction and methods sections to underscore its importance and novelty.\n\n**Regarding the Inductive Bias Analysis of CSODE:**\n\nInductive bias reflects the constraints or preferences of a model in function space. NODE, based on a simple but general differential equation form, has a strong inductive bias, which limits its ability to fit complex nonlinear dynamics to some extent. CSODE, by introducing control terms, expands the function space, enabling the model to learn a broader range of dynamics. This smaller inductive bias gives CSODE stronger expressive power, making it advantageous in learning and predicting complex system behaviors. By introducing Lyapunov functions and providing verifiable linear matrix inequality conditions, we established sufficient conditions for the convergence of CSODE states. This stability theory-based inductive bias ensures that the solutions generated by CSODE are more in line with the characteristics of physical systems. We have added a new section in the Appendix of the revised manuscript to specifically discuss the inductive bias of CSODE and highlight its advantages through a comparison with NODE.\n\n**Regarding the Architectural Evolution from NODE to CSODE:**\n\nCSODE made two main improvements on the basis of NODE: first, the introduction of an independent control term, and second, the allowance for different more complex nonlinear terms. Specifically, we used a parallel neural network to generate control inputs, creating two pathways for information transmission in CSODE: one responsible for the active evolution of states and the other for the adaptive regulation of states. This dual-path structure enables CSODE to handle multi-scale effects. Additionally, CSODE expanded the form of nonlinear functions, allowing for more general nonlinear combinations, creates **a more generalized and extensible structure** for Neural ODE, allowing it to flexibly combine various different models to parameterize internal dynamic models, thereby enhancing the model’s capacity and flexibility. In the revised manuscript, we have added a schematic diagram to visually compare NODE and CSODE and provide a detailed explanation of the motivation and significance of each improvement in the text to highlight the novelty and superiority of CSODE.\n\n**Regarding Hyperparameter Sensitivity:**\n\nWe have conducted supplementary experiments and have included more hyperparameter sensitivity analysis results in the Appendix of the revised version.\n\nOur analysis focused on the impact of learning rate and batch size on CSODE performance. Results show that CSODE exhibits strong robustness to changes in these two hyperparameters. Specifically, within the learning rate range of [0.0005, 0.005] and batch size range of [32, 256], CSODE's performance fluctuations were controlled within 5\\%.\n\nIt is noteworthy that we observed some interesting phenomena. For instance, when the batch size is fixed at 128, CSODE's performance is most stable when the learning rate varies within [0.0005, 0.005] with a standard deviation of only 1.2\\%.\n\nHowever, due to CSODE's deeper structure compared to the original neural ODE, appropriate learning rate selection is crucial for maintaining gradients within a reasonable range. In our main experiments, learning rates at the 1e-2 level were more prone to gradient issues, which may face problems of gradient vanishing or explosion.\n\nRegarding the impact of network depth and width, we conducted a systematic exploration in Chapter 6: Model Scaling Experiment. Results show that increasing CSODE's network width and the number of sub-networks steadily improves model performance. Specifically, when network width increased from 128 to 2048 and the number of sub-networks increased from 1 to 5, CSODE's performance on the test set steadily improved, with an average increase of 23\\%.\n\nWe believe that by supplementing these detailed experimental results and discussions in the revised manuscript, we can help readers gain a more comprehensive understanding of CSODE's hyperparameter characteristics, providing valuable insights for future applications and improvements.\n\nOnce again, we sincerely thank you for your valuable comments. These suggestions not only help improve the quality of the paper but also provide excellent inspiration for our subsequent research. In the revised manuscript, we carefully address every detail to improve the clarity of our work and accurately convey the innovations and contributions to the readers. If you have any further suggestions or questions, we are more than willing to listen and respond positively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yoBFICt4E", "reviewer": "Reviewer_x1sG", "summary": "The paper introduces an extension to Neural ODEs. At it's core this is an architectural change to Neural ODEs, adding more important structure to the dynamics function through a control signal $u(t)$. The paper shows that this can lead to rich non-linear dynamics with convergence guarantees. Extensive experiments on various synthetic and real systems show that structuring the dynamics in this way leads to faster training, better performance and reliable scaling.", "questions": "- Why is the Euler method used for ODE integration and not Dopri5? This is the standard ODE solver used in Neural ODE works.\n- Is it possible to include results for Neural CDE/ODE-RNN/Latent ODE?", "limitations": "- Limitations are well addressed in the conclusion.\n- A thorough broader impact statement is provided in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This is a really solid paper in my opinion. The paper identifies a problem with Neural ODEs, proposes a theoretically grounded solution. As far as I can tell the theory is correct. The evaluation is extensive, generally exploring all important aspects of the work.\n\nThe appendix is rich with important details, showing further results.", "weaknesses": "The paper would be improved most by restructuring the writing. I believe the related work section would be better after the introduction rather than before the conclusion. Section 2 needs to be slightly improved. The description jumps straight into the mathematical detail without very much explanation of $u(t)$ or the subnetworks. I believe the paper would be improved significantly by reducing Section 3 by moving details to the appendix and giving a more intuitive explanation of the main result. Additionally use the space to give a more accessible explanation of CSODE in Section 2.\n\nThere should be some sort of comparison to Neural CDE, since it also uses control signals and is considered SOTA for Neural ODE methods. Other methods that could be compared to to improve the evaluation is Latent ODE and ODE-RNN. \n\nSection 6 on scaling is good but should involve comparisons to other models. Since a claim of the paper is that CSODE has guarantees especially for larger models then this should be shown against NODE and NODE variants. Additionally for the scatter plot in Figure 6, as far as I can tell this is about including more subnetworks, this is not clear from the plot legend or figure caption.\n\nAnother claim is that this method inherently works at different spatial scales. There does not appear to be an experiment testing this, is this possible?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely appreciate your thorough review and valuable feedback on our paper. Your insights are crucial for improving the quality of our work. We have dealt with each of your comments or suggestions carefully.\n\n## 1. Suggestions on the Organization of Paper\n\nWe are grateful for your constructive suggestions. We restructured the paper to enhance its readability as follows:\n\n- Move the Related Work section to follow the Introduction.\n- Strengthen the explanation in Section 2, providing a more intuitive and comprehensible introduction to CSODE.\n- Simplify the content in Section 3, moving some technical details to the Appendix.\n- Present Figure 6 more clearly to explicitly show the changes in the number of subnetworks.\n\n## 2. Comparison with Neural CDE and ODE-RNN\n\nWe acknowledge the importance of Neural CDE and its excellent performance in learning and prediction. \n\nIn the revision, we discuss Neural CDE in more detail in the Related Work section. Additionally, we have conducted supplementary experiments (CharacterTrajectories and PhysioNet Sepsis Prediction) with CSODE-Adapt following the experimental setup in the Neural CDE paper, maintaining similar network structures, parameter counts, and optimization methods. Overall, CSODE performs slightly worse than Neural CDE in irregular observation experiments but better in other time-series-related tasks. And we have also performed our main experiments for neural CDE. The corresponding results are shown as follows:\n\n### CharacterTrajectories (Irregular Observations):\n\nWith 30%, 50%, and 70% data loss, CSODE-Adapt achieved Test Accuracies of 97.3%, 97.8%, and 96.8% respectively, outperforming ODE-RNN but slightly underperforming Neural CDE.\n\n### PhysioNet Sepsis Prediction:\n\nWithout considering Observational Intensity (OI), CSODE achieved a test AUC of 0.868, surpassing both ODE-RNN and Neural CDE. When considering OI, we emulated the Neural CDE approach by training the control variable u in CSODE using OI observations, the test AUC reached 0.881, very close to Neural CDE but superior to Neural CDE and ODE-RNN.\n\n### Dynamic System Modeling:\n\nTaking the Reaction-Diffusion model task as an example, Neural CDE performed with MSE 7.1e-3, MAE 0.60, and Chamfer Distance 0.945, very close to CSODE but slightly inferior to CSODE-Adapt.\n\n## 3. Comparison with Latent ODE\n\nCSODE, as a generalized extension of Neural ODE, can indeed be combined with the Latent ODE framework. In Preliminary Experiments of our original manuscript, inspired by the Neural ODE paper, we compared the original Neural ODE and CSODE for dynamical systems based on the Latent ODE framework.\n\nHowever, in our main experiments, we found that Neural ODE itself can model long-sequence dynamical systems. To obtain more general conclusions, we chose to directly compare Neural ODE and its variants with CSODE, eliminating the influence of how well other components in Latent ODE (such as encoders and decoders).\n\n## 4. Model Scalability in Section 6\n\nCSODE enhances the scalability of Neural ODE, theoretically guaranteeing convergence while increasing parallel subnetworks. Section 6 was originally intended to observe whether increasing subnetworks could effectively maintain high performance.\n\nFollowing your suggestion, we conducted supplementary experiments comparing CSODE, original Neural ODE, and Augmented Neural ODE when achieving the same parameter count by increasing width. Taking the Reaction-Diffusion task as an example, when the number of CSODE subnetworks is 2, increasing the width (128, 256, 512, 1024, 2048):\n\n- CSODE losses (as shown in Figure 6 of the initial submission): 0.060, 0.056, 0.045, 0.037, 0.030\n- Neural ODE losses: 0.083, 0.063, 0.058, 0.061, 0.063\n- Augmented Neural ODE losses: 0.078, 0.061, 0.055, 0.057, 0.057\n\nThe results indicate that the improvement in the learning ability of these two models with increasing width is not as significant as CSODE. We added detailed experimental data and more details to the Appendix of the revised version.\n\n## 5. Ability to Adapt to Different Spatial Scales\n\nWe apologize for any confusion caused by our previous explanation. We intended to provide an intuitive explanation. In practice, we found that CSODE learns more robustly when dealing with dynamical system datasets with different parameter settings, with a smaller standard deviation percentage in experimental results, compared to Neural ODE and Augmented Neural ODE.\n\nFor example, when we alter the spatial scale of the Reaction-Diffusion model task:\n\n- In the original setting: The standard deviation percentages for Neural ODE, Augmented Neural ODE, and CSODE were approximately 5.3, 4.6, and 5.3 respectively.\n- Expanding the spatial domain to 5 times of the original: The standard deviation percentages for Neural ODE, Augmented Neural ODE, and CSODE increased to 7.8, 7.1, and 5.5 respectively. At 10 times the original, the percentages were 8.8, 7.3, and 5.5.\n\nThis indicates that CSODE has advantages in adapting to different spatial scales. We included more complete information in the Appendix of the revised version.\n\n## 6. Choice of Solver\n\nWe chose the Euler method mainly based on the following considerations:\n\n- In practice, we found that more accurate solvers can indeed improve network performance to some extent, but significantly increase training and inference time.\n- The Euler method is the most basic and intuitive integration method, which can reduce the reader's understanding cost and provide the most essential comparison.\n\nIn response to your suggestion, we have supplemented experiments using the Dopri5 solver. The results show that changing the solver does not affect the relative performance between models. For example, in the Reaction-Diffusion model task, NODE, ANODE, and CSODE-Adapt performed with MAE of 0.073, 0.063, and 0.033 respectively, consistent with the original conclusions. These supplementary results were added to the Appendix of the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chairs and Reviewers,

We sincerely thank you for your thorough review of our paper. We have carefully considered each comment and conducted extensive supplementary experiments and analyses. Below are our responses to the main concerns raised:

## 1. CSODE as a Generalized Extension of Neural ODE

CSODE is not only an improvement on Neural ODE but also takes a more general form. Our model has the following characteristics:

1. **Universality**: CSODE can be viewed as a superset of Neural ODE. When the control term is zero, CSODE degenerates to the standard Neural ODE. This design allows CSODE to encompass all functionalities of Neural ODE while providing greater modeling flexibility.
2. **Flexibility**: By adjusting the structure and strength of the control term, CSODE can smoothly transition between Neural ODE and more complex dynamical systems. This flexibility enables CSODE to adapt to a wider range of important domains.
3. **Compatibility**: CSODE maintains the same basic structure as Neural ODE, allowing it to be seamlessly integrated into existing Neural ODE-based frameworks, such as Latent ODE.

## 2. Innovations and Contributions of CSODE

Based on the above generality, the main innovations of CSODE include:

1. Introduction of an independent control term, enhancing the model's ability to handle complex multiscale dynamics. This design allows CSODE to adaptively regulate the evolution process of system states.
2. Extension of nonlinear function forms, improving the model's expressiveness and flexibility. CSODE allows for more general nonlinear combinations, enabling it to capture more complex dynamical features.
3. Establishment of a convergence analysis framework based on Lyapunov stability theory, providing theoretical guarantees for model convergence. This analysis not only applies to CSODE but also provides a methodological basis for studying more general forms of Neural ODE.

## 3. A New Paradigm for Understanding Physical Models

CSODE provides a powerful tool for understanding and modeling complex physical systems:

1. The continuous-time representation naturally aligns with physical processes, allowing CSODE to more accurately capture the dynamic characteristics of systems.
2. The introduction of the control term enables CSODE to simulate physical systems with external inputs or internal regulation mechanisms, which is challenging to achieve in standard Neural ODEs.
3. The ODE form of CSODE facilitates the interpretation of learned dynamics, bridging the gap between data-driven modeling and physical modeling.

## 4. Generality and Scalability

CSODE provides a highly general and scalable Neural ODE framework:

1. By adjusting the complexity of the control term, CSODE can smoothly transition between simple and complex models, adapting to problems of varying complexity.
2. CSODE's parallel subnetwork structure allows for increasing model capacity by adding subnetworks while maintaining convergence.
3. Our experiments show that as the number of subnetworks and network width increase, CSODE's performance steadily improves, demonstrating excellent scalability.

## 5. Natural Integration of Convergence Conditions with Model Structure

CSODE's convergence conditions are mathematically closely connected to the model structure:

1. Our theoretical analysis draws on Lyapunov stability theory, constructing sufficient conditions for convergence.
2. These conditions not only provide theoretical guarantees for the model but also guide model design, ensuring that the solution generated by CSODE better conforms to the state of physical systems.
3. The establishment of the convergence analysis framework provides a theoretical foundation for studying more general forms of Neural ODE.

## 6. Summary of Supplementary Experiments

1. **Optimizer Experiments**: We compared the performance of Adam and L-BFGS optimizers. Results show that L-BFGS can significantly improve the convergence performance of Neural ODE and its variants, but CSODE maintains its advantage.
2. **Comparison with Neural CDE and ODE-RNN**: In irregular observation experiments, CSODE performed slightly worse than Neural CDE but better in other time-series-related tasks.
3. **Model Scalability Experiments**: By increasing network width, we compared the performance of CSODE, original Neural ODE, and Augmented Neural ODE with the same parameter count. Results show that CSODE's learning ability improves more significantly with increasing width.
4. **Ability to Adapt to Different Spatial Scales**: Experiments show that when changing the spatial scale of dynamical systems, CSODE demonstrates stronger robustness, with a smaller standard deviation percentage in experimental results.
5. **Solver Selection**: We supplemented experiments using the Dopri5 solver. Results indicate that changing the solver does not affect the relative performance between models.

These supplementary experiments further validate the superiority of CSODE and the correctness of our theoretical analysis.

Through these innovations and improvements, CSODE not only extends the capabilities of Neural ODE but also provides new insights into the design and analysis of continuous-time deep learning models. We believe that CSODE, as a more general and flexible framework, will bring new opportunities and challenges to fields such as dynamical system modeling and time series analysis.

Once again, we appreciate your valuable comments. We have comprehensively revised and improved the paper based on your feedback. If you have any further questions, we welcome continued discussion and exchange.

Meta-review: In this paper, authors present a Neural ODEs with a control term that provides better expressivity and flexibility. They provide theoretical guarantees and also demonstrate the superior performance on three examples datasets, namely, Hindmarsh-Rose, Reaction-Diffusion, Shallow Water. Reviewers have appreciated the work and acknowledged the meaningful contribution that the present work makes in modeling dynamical systems. They have also suggested some modifications and additional experiments, which the authors have performed and reported. It is recommended that these results and the modifications in terms of writing are incorporated in the final version of the paper.
Decision: Accept (poster)