Title: ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural ODEs

Differential Equations

Dynamical Systems

Deep Learning
Abstract: Neural ODEs (NODEs) are continuous-time neural networks (NNs) that can process data without the limitation of time intervals. They have advantages in learning and understanding the evolution of complex real dynamics. Many previous works have focused on NODEs in concise forms, while numerous physical systems taking straightforward forms in fact belong to their more complex quasi-classes, thus appealing to a class of general NODEs with high scalability and flexibility to model those systems. This however may result in intricate nonlinear properties. In this paper, we introduce ControlSynth Neural ODEs (CSODEs). We show that despite their highly nonlinear nature, convergence can be guaranteed via tractable linear inequalities. In the composition of CSODEs, we introduce an extra control term for learning the potential simultaneous capture of dynamics at different scales, which could be particularly useful for partial differential equation-formulated systems. Finally, we compare several representative NNs with CSODEs on important physical dynamics under the inductive biases of CSODEs, and illustrate that CSODEs have better learning and predictive abilities in these settings.

ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence

Abstract

Neural ODEs (NODEs) are continuous-time neural networks (NNs) that can process data without the limitation of time intervals. They have advantages in learning and understanding the evolution of complex real dynamics. Many previous works have focused on NODEs in concise forms, while numerous physical systems taking straightforward forms, in fact, belong to their more complex quasi-classes, thus appealing to a class of general NODEs with high scalability and flexibility to model those systems. This, however, may result in intricate nonlinear properties. In this paper, we introduce ControlSynth Neural ODEs (CSODEs). We show that despite their highly nonlinear nature, convergence can be guaranteed via tractable linear inequalities. In the composition of CSODEs, we introduce an extra control term for learning the potential simultaneous capture of dynamics at different scales, which could be particularly useful for partial differential equation-formulated systems. Finally, we compare several representative NNs with CSODEs on important physical dynamics under the inductive biases of CSODEs, and illustrate that CSODEs have better learning and predictive abilities in these settings.

1 Introduction

Neural ODEs (NODEs) [4] were developed from the limiting cases of continuous recurrent networks and residual networks and exhibit non-negligible advantages in data incorporation and modeling unknown dynamics of complex systems, for instance. Their continuous nature makes them particularly beneficial to learning and predicting the dynamical behavior of complex physical systems, which are often difficult to realize due to sophisticated internal and external factors for the systems.

Starting from the introduction of NODEs, many types of variants of NODEs have been studied (see, e.g. , [6, 19, 17, 15]). Nevertheless, there are rare studies concerning highly scalable and flexible dynamics that also present complex nonlinear natures, bringing difficulties in their modeling and analyses. No in-detail research attention has been paid to the scalability of depth and structure in NODEs despite numerous physical systems in the real world having inscrutable dynamics and compositions. To fill in this gap, we propose ControlSynth Neural ODEs (CSODEs), in whose structure another sub-network is also incorporated for enlarging the dexterity of the composition and controlling the evolution of the state. Different from most of the existing methods and experiments, we focus on widely investigated physical models, with known state evolution under necessary conditions. Subsequently, we will show that the proposed NODEs are effective in learning and understanding those models.

Our contributions are mainly composed of the novel structure of CSODEs, their convergence guarantees, and the comparative experiments among CSODEs, several representative NODEs, and their

divisions, illustrating the beneficiality of CSODEs in the setting of prediction. The convergence conditions provide tractable solutions for constraining the learned model to a convergent one. The preliminary experiment demonstrates that our CSODEs can learn the evolution of the dynamics faster and more precisely. Also, we show that introducing sub-networks into CSODE does not impact the overall computational performance more than the other comparable NNs. Finally, we compare NODE, Augmented Neural ODE (ANODE), Second Order Neural ODE (SONODE), CSODE, and its variant in real dynamical systems. The experimental results indicate that our CSODE is beneficial to more accurate time series prediction in the systems. Our code is available online at https://github.com/ContinuumCoder/ControlSynth-Neural-ODE.

2 ControlSynth Neural Ordinary Differential Equations

We begin by introducing the form of CSODEs as follows:

where x t := x ( t ) ∈ R n is the state vector; the matrices A · are with approximate dimensions; W · are weight matrices; the input u t := u ( t ) ∈ U ⊂ R m , u ∈ L m ∞ (refer to Appendix C.1); f j = [ f 1 j . . . f k j j ] ⊤ ( f j : R k j → R k j ) and g : U → R n are the functions ensuring the existence of the solutions of the neural network (NN) (1) at least locally in time, and g = [ g 1 . . . g n ] ⊤ ; w.l.o.g. , the time t is set as t ≥ 0 .

CSODEs extend the concept of Neural ODEs, which are typically expressed as ˙ x ( t ) = f ( x ( t )) , where f is a neural network. CSODEs incorporate control inputs u ( t ) and create a combination of subnetworks ∑ M j =1 A j f j ( W j x ( t )) . This formulation enhances ex-

ut

ct

Ao'

h()

A1f1(W-)

AMfM(WM)

△t

△t

At

△t

△t

△t

+

ut+△t

t+△t

pressiveness and adaptability to complex systems with external inputs. CSODEs provide a more universal framework and improve upon Neural ODEs by offering greater flexibility, interpretability through separated linear and nonlinear terms, and natural integration with techniques in control theory.

For simplicity, in the experiments (see Section 6), we select two common NNs as the specific examples of the function g : Traditional Multilayer Perceptron (MLP) and two-layer Convolutional Neural Network (CNN), and tanh as the used activation functions: a particular subclass of the functions f j . Note that in the case that g ( u ) represents an MLP, there are many different types of neurons, for example, functional neurons [21], and in CSODEs (1) there may exist f i j ∈ L 1 ( X, A , µ ) , where ( X, A , µ ) denotes a σ -finite measure space.

3 Related Work

SONODEs, ANODEs, and NCDEs vs CSODEs SONODEs [17] are particularly suitable for learning and predicting the second-order models, and ANODEs [6] are useful for cases where the state evolution does not play an important role. In contrast, many real models of second-order or even higher orders can be transformed into the common first-order but emerge numerous nonlinearities that are difficult to accurately treat by NODEs and their variants. Despite the intricate structure of CSODEs, they can be equipped with a significant convergence attribute. The experiments show that our CSODEs are more appropriate for model scaling and natural dynamics that exhibit complex nonlinear properties. Although both Neural CDEs (NCDEs) [12] and CSODEs extend NODEs through control mechanisms, their architectural philosophies differ. NCDEs introduce control through path-valued data driving the vector field, following ˙ x ( t ) = f ( x ( t )) . In contrast, CSODEs propose a more sophisticated structure combining multiple subnetworks with a dedicated control term g ( u ( t )) . This design not only provides theoretical convergence guarantees but also enables CSODEs

to capture intricate physical dynamics through its hierarchical structure, particularly beneficial for systems exhibiting complex nonlinear behaviors that elude simpler controlled formulations.

Convergence and Stability Relevant to our work, [13] studies the asymptotic stability of a specific class of NODEs as part of its fundamental property investigations on NODEs. In contrast, our study focuses on the scalability aspects related to both the number of subnetworks and the width of NNs, extending beyond the scope of the NNs in [13, 15, 16]. Furthermore, our experiments are primarily toward learning and predicting the dynamic behaviors of physical models rather than image classifications. Stability analyses have also been performed on NODE variants, such as SODEF [11] SNDEs [26], and Stable Neural Flows[14].

Physical Information Based Dynamics Similar to the way CSODEs embody physical dynamics, various other models have been developed to incorporate physical information, ensuring the derivation of physically plausible results, both in discrete and continuous time. For example, [3] employs physicsinformed neural networks (PINNs) (In contrast to NODEs and their variants, which mainly focus on implicitly learning dynamical systems, PINNs explicitly incorporate physical laws into the loss function. These two types of methods tackle the problem of integrating physical knowledge into deep learning models from different perspectives and are complementary research directions) to solve the Shallow Water Equations on the sphere for meteorological use, and Raissi et al. [20] propose a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations using PINNs. Shi et al. [23] and O'Leary et al. [18] utilize enhanced NODEs for learning complex behaviors in physical models, while Greydanus et al. [9] and Cranmer et al. [5] propose Hamiltonian and Lagrangian NNs, respectively, to learn physically consistent dynamics.

4 Theoretical Results: Convergence Analysis

Consider the CSODEs given in Equation (1), where f j : R k j → R k j are nonlinear activation functions. For them, an imposed condition on f i j (the i -th element of the vector-valued f j ) is presented as follows:

Assumption 1 For any i ∈ { 1 , . . . , k j } and j ∈ { 1 , . . . , M } , sf i j ( s ) > 0 for all s ∈ R \{ 0 } .

Remark 1 Assumption 1 applies to many activation functions, such as tanh and parametric ReLU. It picks up the activation functions passing through the origin and the quadrants I and III. For more explanations, the reader is referred to Appendix B.1.

In this study, to analyze the convergence property of the NN (1), we first define the concept of convergence :

Definition 1 The model (1) is convergent if it admits a unique bounded solution for t ∈ R that is globally asymptotically stable (GAS).

In order to investigate the convergence, two properties have to be satisfied, that is, the boundedness and the GAS guarantees of the solution x ∗ for (1). In this respect, two assumptions are given as follows.

Assumption 2 Assume that the functions f i j are continuous and strictly increasing for any i ∈ { 1 , . . . , k j } and j ∈ { 1 , . . . , M } .

Assumption 2 aligns with CSODE's structure, reflecting continuity and monotonicity of activation functions. This relates to model dynamics and is satisfied by most common activations.

In the analysis of convergence, one needs to study two models in the same form but with different initial conditions and their contracting properties. To that end, along with (1), we consider the model ˙ y ( t ) = A 0 y ( t ) + ∑ M j =1 A j f j ( W j y ( t )) + g ( u ( t )) with the same input but different initial conditions y (0) ∈ R n . Let ξ := y -x . Then the corresponding error system is

where p j ( x, ξ ) = f j ( W j ( ξ + x )) -f j ( W j x ) . Note that for any fixed x ∈ R n , the functions p j in the variable ξ ∈ R n satisfy the properties formulated in Assumptions 1, 2. The following assumption is imposed for analyzing the contracting property of (2).

Assumption 3 Assume that there exist positive semidefinite diagonal matrices S j 0 , S j 1 , S j 2 , S j,r 3 , H j 0 , H j 1 , H j 2 , H j,r 3 ( j, r ∈ { 1 , . . . , M } ) with appropriate dimensions such that

Notice that Assumption 3 is at least more relaxed than Lipschitz continuity (see Appendix B.2 for an intuitive example of activation functions satisfying Assumption 3).

Convergence Conditions We are now ready to show the convergence conditions for the CSODEs (1):

Theorem 1 Let Assumptions 1-3 be satisfied. If there exist positive semidefinite symmetric matrices P, ˜ P ; positive semidefinite diagonal matrices { Λ j = diag(Λ j 1 , . . . , Λ j n ) } j M =1 , { ˜ Λ j = diag( ˜ Λ j 1 , . . . , ˜ Λ j n ) } j M =1 , { Ξ s } s M =0 , { Υ s,r } 0 ≤ s<r ≤ M , { ˜ Υ j,j ′ } M j,j ′ =1 , { Γ j } j M =1 , { Ω j } j M =1 , ˜ Ξ 0 ; positive definite symmetric matrix Φ ; and positive scalars γ, θ such that the linear matrix inequalities (LMIs) in Appendix B.4 hold true. Then, a forward complete system (1) is convergent.

Proof in Appendix C.3. Note that the used conditions on f i j in Assumption 3 can be relaxed to "non-decreasing", which enlarges the scope of activation functions, including non-smooth functions like ReLU, then the resulting modifications for the formulations of Theorem 1 can be readily obtained, highlighting the CSODE framework's adaptability.

Those LMI conditions ensure system convergence. From an energy perspective, this indicates the error system's generalized energy (represented by the energy (or Lyapunov) function) is monotonically non-increasing, leading to convergence towards the equilibrium point: origin. These conditions can be easily verified, thanks to CSODE's structural characteristics and LMIs' highly adjustable elements.

The matrices, such as ˜ Ξ 0 and ˜ Υ j,j ′ , in the LMIs act as compensation terms balancing the effects of linear and nonlinear terms, ensuring the derivative of the energy function ˜ V remains non-positive. Properties of f j (Assumptions 1 and 2) provide facilitation in constructing these matrices. Assumption 3 allows for non-restrictive conditions on activation functions, avoiding strong global Lipschitz continuity assumptions and providing precise local asymptotic stability characterization.

5 Preliminary Experiments

Convergence and Stability Convergence, an important attribute showcasing a model's learning ability, refers to its capability to consistently approach the theoretical optimal solution throughout the learning process over iterations. To validate the convergence and stability of CSODEs, we design an experiment that involves learning simple spiral trajectories, chosen for their fundamental complexity which all models should ideally handle. In this experiment, we compare CSODEs and NODEs, both based on the Latent ODE structure [4]. This setup provides a fundamental baseline for assessing convergence and ensures a fair comparison, enabling each model to demonstrate its learning capabilities under comparable conditions without bias toward specific structural advantages. The Mean Absolute Error (MAE) loss function, which measures the average of the absolute differences between the estimated trajectories and the true trajectories, is used as the indicator. We train the model on 100 noisy spiral data observation points to learn this trajectory. The MAE loss values collected over training epochs consistently, plotted in Figure 2, show that the CSODE model not only converges faster but also maintains a lower error rate compared to the traditional NODE model, particularly under the constraints of limited and noisy observational data points. Our model's ability to accurately predict the target trajectory despite the noisy data illustrates its noise tolerance, a critical aspect of its robust stability.

Generalization and Extrapolation Figure 3 presents a visual comparison of the prediction results from the CSODE model and the traditional NODE model with the actual ground truth trajectory after various training epochs in trajectories learning experiment in the previous paragraph. It is evident from the figure that the CSODE not only learns the trajectory within the observation period more precisely in fewer training epochs but also aligns more accurately with the original trajectories beyond the observation period, which accounts for 25% of the total duration. This demonstrates the robust generalization and extrapolation capabilities of CSODEs for predicting future states, illustrating its better understanding of the underlying time-dependent structures and dynamics in the data.

30

CSODE

25

NODE

20

MAE

15

10

5

0

250

500

750

1000

1250

1500

Epoch

Computational Performance To evaluate whether introducing subnetworks in CSODEs impacts the overall computational performance, we conduct experimental validations on image classification, focusing on aspects such as operational effi-

ControlSynth Neural ODE:

Neural ODE:

Epoch 400

Epoch 600

Epoch 800

—Original Trajectory

—PredictedTrajectory

●Sampled Trajectory with Noise

ciency and system load. These experiments are based on the ODE-Nets framework [4], using the forward Euler method, and span 100 epochs on the MNIST dataset to compare the performance of NODEs [4], Augmented Neural ODEs (ANODEs) [6], Second Order Neural ODEs (SONODEs) [17], and CSODEs. All experiments in this study are conducted on a system equipped with an NVIDIA GeForce RTX 3080 GPU and CUDA, ensuring a consistent computational environment across all tests. Performance metrics include test error rate, number of parameters (# Params), floating-point operations per second (FLOPS), average batch processing time (Time), and peak memory usage (Mem). The results (see Table 1) show that although CSODEs theoretically involves iterative computation of g ( u ) , it does not significantly reduce computational efficiency in practice. While MNIST is a relatively simple dataset, the experimental outcomes preliminarily confirm the benefits of incorporating a learnable function in enhancing the performance of implicit networks, effectively maintaining efficiency, and alleviating concerns over the potential trade-off between expressive capability and computational efficiency of the networks.

6 Complex Systems Time Series Prediction Experiments

In this section, we experimentally analyze the performance of CSODEs, based on NODEs, compared to other models based on NODEs and traditional time series forecasting methods in extrapolative time series prediction for complex systems. We select challenging physical dynamical systems from neuroscience (Hindmarsh-Rose model [10]), chemistry and biology (Reaction-Diffusion model [8]), and geophysics and fluid dynamics (Shallow Water Equations [25]), which exhibit rich spatiotemporal variations over long time scales.

6.1 Model Architectures for Experiments

To comprehensively evaluate the performance of different model architectures in dynamic system modeling, this study compares several representative NODEs-based models, including the traditional NODEs [4], ANODEs [6], SONODEs [17], and our proposed CSODEs. Additionally, these models are compared with traditional MLPs [22] and Recurrent Neural Networks (RNNs) [7].

Notable that while CSODE can be integrated with the Latent ODE framework, as demonstrated in our preliminary experiments (in Section 5), we opted to conduct our main experiments without this integration. This decision was driven by our observation that NODE itself effectively models long-sequence dynamical systems, allowing us to evaluate CSODE's performance more directly by eliminating potential influences from additional Latent ODE components like encoders and decoders.

All models within our experiment that are based on the NODE framework utilize a uniform MLP to parameterize the core dynamical system ˙ y = f ( y ) . These models employ the forward Euler method for integration, though other numerical solvers like Dopri5 are also viable options (detailed performance comparisons of different solvers can be found in Appendix I.1). The MLP serves as the update function, iterating to act on the evolution of dynamic system states. Specifically, the MLP receives the current state as input and outputs the next state, thereby modeling the change in state over time. Meanwhile, the RNN employs the same structure as the MLP, using current input and the previous timestep's hidden state to update the current timestep's hidden state. To ensure a fair comparison, we finely align the number of parameters across all models, including the parameterization of the core dynamical system and other components (such as the subnetwork in the CSODE model and the augmented dimensions in the ANODE model). All designs ensure that the total number of parameters remains consistent within ± 1% .

The unique feature of CSODEs lies in the introduction of an additional subnetwork to model the control term g ( u ( t )) , extending the original dynamical system. We employ a unified auxiliary NN to model the changing rate of g ( u ( t )) , with the initial value g ( u 0 ) set to be the same as y 0 . These subnetwork structures are similar to the MLP or use simplified single-layer networks. We also introduce the CSODE-Adapt variant, replacing the function representing the changing rate of the control term with a network consisting of two convolutional layers, to explore the scalability and flexibility of CSODEs. Notably, for fair comparison of fundamental architectures, we used the standard Adam optimizer in our main experiments, though we note that alternative optimizers like L-BFGS could further enhance CSODE's performance (see Appendix I.2 for more details).

At the current research forefront, some researchers have proposed integrating NODEs with Transformer layers (TLODEs) [27]. TLODEs can be considered a special case of CSODEs, where the Transformer layers implement the function f ( · ) . Building on TLODEs, we introduce the control term g ( · ) , creating a more complete CSODEs structure with Transformer layers (CSTLODEs). Considering the widespread application of Transformers in time-series prediction, we conduct comparative experiments between the Transformer, TLODE, and CSTLODE models. However, due to significant architectural differences between Transformers and MLPs and RNNs, directly incorporating them into the main experiment might introduce additional confounding factors, deviating from the theoretical discussion of general NODEs. Therefore, to maintain the focus and clarity of the main text, we have placed the experimental results and discussions related to TLODEs and CSTLODEs in Appendix F.

6.2 Experimental Tasks and Datasets Description

In this subsection, we detail the experimental tasks and datasets used to explore the application of NNs in simulating various dynamic systems. For more comprehensive simulation and experimental details, see Appendix D.

Modeling Hindmarsh-Rose Neuron Dynamics In this task, we explore the application of NNs in simulating the Hindmarsh-Rose neuron model [10]. We validate their potential in simulating complex neuronal dynamics and assess their prospects in broader neuroscience and biophysical research. The Hindmarsh-Rose model is widely used to describe neuronal dynamic behavior, particularly suitable for studying neuronal firing behavior and chaotic phenomena. This model describes the evolution of the neuron's voltage and recovery variables over time with the following three coupled nonlinear differential equations:

where x represents the membrane potential, y represents the recovery variable associated with the membrane potential, and z is the variable for adaptive current. The parameters a , b , c , d , r , s , x 0 , and I determine the neuron's nonlinear firing behavior, voltage response curve, recovery process rate, and adaptive current, simulating the voltage changes and behavior of the neuron.

We conduct 1000 simulations, each lasting 20 seconds. In the first 10 seconds of each simulation, we collect 1500 data points reflecting the changes in the neuron's three-dimensional coordinates ( x, y, z ). We divide these data points into 50 sequences, each containing 30 time points. Our goal is to use these sequences to train an NN to predict the dynamic changes over the next 10 seconds, specifically the sequences of the next 30 time points.

Modeling Reaction-Diffusion System In this task, we employ NN models to simulate the GrayScott equations [8], a Reaction-Diffusion system widely used to describe chemical reactions and diffusion processes. This system is particularly significant in pattern formation, biological tissue, and self-organizing chemical processes, holding important theoretical and practical implications. This task allows us to explore the potential applications of these models in environmental and bioengineering contexts. The model describes the interactions and spatial diffusion of two chemical substances, U and V , through a set of partial differential equations (PDEs):

where D U and D V are the diffusion coefficients, and j and k are reaction rate parameters.

We conducted 1000 simulations with periodic boundary conditions [1] for 150 seconds, each randomizing initial conditions and diffusion coefficients D U and D V , and used data from the first 100 seconds. These 100 seconds of data were divided into 100 sequences, each spanning 25 seconds, to train the NNs. Our goal is to utilize the training results to predict chemical dynamics over the next 50 seconds.

Modeling Shallow Water Equations In this task, we explore the application potential of NODE systems in simulating the Shallow Water Equations [25], which describe the horizontal flow and surface behavior of liquids in confined spaces. These equations have significant physical relevance in fields such as hydrology and environmental engineering, especially in flood simulations and wave dynamics analysis. Mathematically, the Shallow Water Equations are expressed through a set of nonlinear PDEs that represent changes in water depth ( h ) and flow velocity ( u ), incorporating the principles of continuity and conservation of momentum, with gravity g , as follows:

We conduct 1000 periodic boundary condition wave propagation simulations [1] lasting 7 seconds, generating 1500 data points in the first 4.7 seconds to describe water wave depth variations. After normalizing the data using a sliding window, we segment it into 100 sequences, each containing 15 points, to train a global water surface dynamics model to predict water wave depth changes during the remaining 2.3 seconds of the simulation.

6.3 Metrics for Assessing Prediction Accuracy

We employ the following metrics to compare time series predictions and ground truth:

Mean Squared Error (MSE): Calculates the mean of squared differences between predicted and actual values, sensitive to large errors. Used in all three tasks.

Mean Absolute Error (MAE): Measures the average absolute difference between predicted and actual values, robust to noise. Used in all three tasks.

R 2 Score: Quantifies the proportion of variance in predictions relative to actual data, reflecting explanatory power and accuracy. Used in the Hindmarsh-Rose model's 3D time series prediction.

Chamfer Distance (CD): Measures the average shortest distance between points in two sets, emphasizing spatial structure matching accuracy [2]. Used to compare predicted and ground truth physical field sequences in Reaction-Diffusion systems and Shallow Water Equations tasks.

6.4 Experimental Results and Analysis

Experiments are conducted ten times for each task, and the average metrics are presented in Table 2, the reader is referred to Appendix E for more statistical details. NNs in Group B, based on NODE and its variants, outperform traditional models in Group A, such as MLP and RNN, in predicting complex physical dynamic systems. Traditional models struggle with complex time dependencies and nonlinear dynamics, while models based on differential equations demonstrate greater adaptability and accuracy.

Within Group B, the CSODE model surpasses other models due to its control elements that enhance precision and adaptability to changes in initial conditions and system parameters. Figures 4 and 5 show that CSODEs accurately reflect complex dynamic system details through qualitative results, with more results available in Appendix G.

Moreover, the CSODE-Adapt model (Group C) integrates convolutional layers, enhancing its applicability and effectiveness, particularly in dynamic systems with significant spatial features, such as Reaction-Diffusion systems. This model performs better than all others, highlighting the flexibility and highly customizable structure of the CSODEs and its advantages and potential in predicting complex physical dynamic systems.

NODE

ANODE

CSODE

CSODE-Adapt

GroundTruth

-axis

Z

0.8

0.6

0.4

X-axis

8

16

0.5

0

6

0.5

Y-axis

Gray-Scott Model of Reaction-Diffusion,U(dimensionless)

1.2

InitialCondition

GroundTruthforPrediction

1.1

0.9

NODE

CSODE

0.8

0.7

0.6

6.5 Comparison with Observation-aware Baselines

We conducted supplementary experiments (CharacterTrajectories and PhysioNet Sepsis Prediction) with CSODE-Adapt following the experimental setup in [12], maintaining similar network structures,

parameter counts, and optimization methods. Overall, CSODE performs slightly worse than Neural CDE in irregular observation experiments but better in other time-series-related tasks. We have also performed our main experiments for neural CDE. The corresponding results are shown as follows:

As shown in Table 3, for the CharacterTrajectories task with irregular observations, Neural CDE achieves the best performance across all missing data ratios, followed by CSODE and then ODERNN. In the PhysioNet Sepsis Prediction task, without considering observation intensity (OI), CSODE-Adapt achieves the highest AUC value of 0.871, while with OI consideration, Neural CDE performs best with an AUC of 0.885, followed closely by CSODE-Adapt and CSODE. For the Reaction-Diffusion modeling task, CSODE-Adapt demonstrates superior performance across all metrics (MSE, MAE, CD), while CSODE and Neural CDE show comparable performance, both outperforming ODE-RNN. Overall, while Neural CDE exhibits advantages in handling irregular observations, CSODE-Adapt shows competitive or superior performance in tasks requiring complex dynamic system modeling and clinical prediction, demonstrating its effectiveness as a general-purpose time series modeling tool.

7 Model Scaling Experiment

In the experiments above, CSODEs demonstrate significant superiority over traditional NODEs and their variants, under the maintenance of the same number of parameters and architectural configuration. Based on these findings, our scaling experiments focus primarily on exploring the scalability and architectural robustness of CSODEs, without further comparison to other models.

Wealso observe changes in system performance after scaling CSODEs. To maintain consistency in the experiments, each sub-network is configured with two dense layers. We select the Reaction-Diffusion task in Section 6 as an example to explore the impact of increasing the number of sub-networks and the width of each sub-network on system performance. Specifically, the network widths, which refer to the number of hidden dimensions in the dense layers, are set at 128, 256, 512, 1024, and 2048. The number of sub-networks, equivalent to M in NNs (1), is set at 1, 2, 3, 4, and 5.

The experimental design varies network width and number of sub-networks. We employ a learning rate formula: learning rate = k W × √ N , where k is a constant, W is the network width, and N is the number of sub-networks. This adjusts the learning rate based on width and moderates it for sub-network count to handle complexity. For instance, with a width of 1024 and three sub-networks, the learning rate is 0 . 1 1024 × √ 3 . We use the Adam optimizer for training.

In terms of overall performance, the heatmap in Figure 6 shows that under the CSODEs, increasing the network width and number of subnetworks results in stable and enhanced overall performance. Additionally, the scatter plot demonstrates that increasing the number of subnetworks significantly improves the model's generalization ability, with training and validation performance showing a stronger correlation. For further details on comparative experiments, the model's stability, and convergence despite the increase in the number of subnetworks and width, refer to Appendix H.

ConfigurationLossHeatmap

Comparison ofTrainingvsValidationLossforDifferentSubnets

NetworkWidth=512

0.11

128

0.1150

0.0600

0.0390

0.0338

0.0328

0.025

Subnets

0.10

Subnet 1

Subnet 2

256

0.0976

0.0562

0.0341

0.0304

0.0315

0.09

ValidationLoss

0.020

Subnet 3

0.08

Subnet 4

Widths

Loss

Subnet 5

512

0.0879

0.0449

0.0344

0.0254

0.0300

0.07

Validation

0.015

0.06

Final

024

0.0704

0.0365

0.0255

0.0349

0.0327

0.05

0.010

2048

0.04

0.0637

0.0300

0.0335

0.0344

0.0343

0.005

0.03

2

3

4

NumberofSub-networks

0.005

0.010

0.015

0.020

0.025

Training Loss

8 Conclusion

In this work, we analyzed the learning and predicting abilities of Neural ODEs (NODEs). A class of new NODEs: ControlSynth Neural ODEs (CSODEs) was proposed, which has a complex structure but high scalability and dexterity. We started by investigating the convergence property of CSODEs and comparing them with traditional NODEs in the context of generalization, extrapolation, and computational performance. We also used a variety of representative NODE models and CSODEs to model several important real physical dynamics and compared their prediction accuracy.

We presented that although the control subjects (NODEs, Augmented Neural ODEs (ANODEs), and Second Order NODEs (SONODEs)) do not have the physics information-based inductive biases specifically owned by our CSODEs, they can learn and understand complex dynamics in practice. In particular, SONODEs own inductive biases for second-order ODE-formulated dynamics, while the ones of CSODEs mainly are for first-order models with high nonlinear natures, scalability, and flexibility that belong to a broad class of real systems. The experimental results on dynamical systems governed by the Hindmarsh-Rose Model, Reaction-Diffusion Model, and Shallow Water Equations preliminarily demonstrate the superiority of our ControlSynth ODEs (CSODEs) in learning and predicting highly nonlinear dynamics, even when represented as partial differential equations.

Limitations and Future Work The effectiveness of inductive biases of CSODEs varies which, depending on the specific application scenarios, may not be preferable; in the evolution of partial differential equations, there often exists mutual interference between different scales ( e.g. , spatial and temporal scales), which, however, is approximately learned by CSODEs. We believe this work could provide promising avenues for future studies, including: For enlarging the use scope of inductive biases of CSODEs in complex dynamics and reflecting the mutual intervention between scales, one can consider a more general NODE: ˙ x = f ( x, u ) , with guaranteed stability and convergence. This allows a greater scope of dynamics and thus may prompt the improvement of the accuracy of modeling and predicting systems with more complex structures and behaviors. Furthermore, in practice, CSODEs are more sensitive to learning rate selections due to their more complex architectures. Our preliminary investigation in Appendix I.3 reveals the significant impact of hyperparameter adjustments on model performance. Building upon these initial findings, future research will examine this sensitivity more thoroughly and consider methods like adaptive learning rate adjustment and model simplification to address it. Finally, this work maintained standard optimization settings for fair comparison, future research could explore specialized training algorithms that leverage CSODE's structural properties and theoretical foundations.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under grant 62403125, the Natural Science Foundation of Jiangsu Province, and the Fundamental Research Funds for the Central Universities under grants 2242024k30037 and 2242024k30038.

References

M. P. Allen and D. J. Tildesley. Computer simulation of liquids . Oxford University Press, 2017.

H. G. Barrow, J. M. Tenenbaum, R. C. Bolles, and H. C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. In Proceedings: Image Understanding Workshop , pages 21-27. Science Applications, Inc, 1977.

A. Bihlo and R. O. Popovych. Physics-informed neural networks for the shallow-water equations on the sphere. Journal of Computational Physics , 456:111024, 2022.

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems , 31, 2018.

M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks. arXiv preprint arXiv:2003.04630 , 2020.

E. Dupont, A. Doucet, and Y. W. Teh. Augmented neural odes. Advances in Neural Information Processing Systems , 32, 2019.

J. L. Elman. Finding structure in time. Cognitive Science , 14(2):179-211, 1990.

P. Gray and S. Scott. Autocatalytic reactions in the isothermal, continuous stirred tank reactor: isolas and other forms of multistability. Chemical Engineering Science , 38(1):29-43, 1983.

S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems , 32, 2019.

J. L. Hindmarsh and R. Rose. A model of neuronal bursting using three coupled first order differential equations. Proceedings of the Royal Society of London. Series B. Biological Sciences , 221(1222):87-102, 1984.

Q. Kang, Y. Song, Q. Ding, and W. P. Tay. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Advances in Neural Information Processing Systems , 34:14925-14937, 2021.

P. Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems , 33:6696-6707, 2020.

D. Llorente-Vidrio, M. Ballesteros, I. Salgado, and I. Chairez. Deep learning adapted to differential neural networks used as pattern classification of electrophysiological signals. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(9):4807-4818, 2021.

S. Massaroli, M. Poli, M. Bin, J. Park, A. Yamashita, and H. Asama. Stable neural flows. arXiv preprint arXiv:2003.08063 , 2020.

S. Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama. Dissecting neural ODEs. Advances in Neural Information Processing Systems , 33:3952-3963, 2020.

W. Mei, D. Efimov, and R. Ushirobira. On annular short-time stability conditions for generalised Persidskii systems. International Journal of Control , 97(9):2084-2094, 2024.

A. Norcliffe, C. Bodnar, B. Day, N. Simidjievski, and P. Liò. On second order behaviour in augmented neural odes. Advances in Neural Information Processing Systems , 33:5911-5921, 2020.

J. O'Leary, J. A. Paulson, and A. Mesbah. Stochastic physics-informed neural ordinary differential equations. Journal of Computational Physics , 468:111466, 2022.

M. Poli, S. Massaroli, J. Park, A. Yamashita, H. Asama, and J. Park. Graph neural ordinary differential equations. arXiv preprint arXiv:1911.07532 , 2019.

M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

F. Rossi and B. Conan-Guez. Functional multi-layer perceptron: a non-linear tool for functional data analysis. Neural Networks , 18(1):45-60, 2005.

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature , 323(6088):533-536, 1986.

G. Shi, D. Zhang, M. Jin, S. Pan, and S. Y. Philip. Towards complex dynamic physics system simulation with graph neural ordinary equations. Neural Networks , page 106341, 2024.

E. D. Sontag and Y. Wang. On characterizations of the input-to-state stability property. Systems &Control Letters , 24(5):351-359, 1995.

J. J. Stoker. Water waves: The mathematical theory with applications , volume 36. John Wiley &Sons, 1992.

A. White, N. Kilbertus, M. Gelbrecht, and N. Boers. Stabilized neural differential equations for learning dynamics with explicit constraints. Advances in Neural Information Processing Systems , 36, 2024.

Y. D. Zhong, T. Zhang, A. Chakraborty, and B. Dey. A neural ODE interpretation of transformer layers. arXiv preprint arXiv:2212.06011 , 2022.

A Broader Impact

The introduction of ControlSynth Neural ODEs (CSODEs) represents an advancement in the field of deep learning and its applications in physics, engineering, and robotics. CSODEs offer a highly scalable and flexible framework for modeling complex nonlinear dynamical systems, enabling the learning and prediction of intricate behaviors from data, which has implications across various domains.

In the realm of physics, CSODEs provide a powerful tool for studying complex physical processes in areas such as meteorology, fluid dynamics, and materials science. By leveraging their highly expandable network structure and control term, CSODEs may capture the dynamics of systems with elusive internal and external factors, overcoming the limitations of traditional mathematical modeling approaches. This opens up new avenues for understanding and predicting the behavior of these complex systems.

Moreover, the theoretical convergence guarantees of CSODEs lay a solid foundation for applications of neural networks in physical modeling. Despite their high nonlinearity, the convergence of CSODEs can be ensured through tractable linear inequalities. This enhances the interpretability and reliability of neural network models, facilitating their deployment in engineering and safety-critical domains where provable convergence is crucial.

The control term introduced in CSODEs also enables the learning of multi-scale dynamics, which is particularly relevant for systems described by partial differential equations. By simultaneously capturing dynamics at different spatial and temporal scales, CSODEs expand the practicality of neural networks to multi-scale physical problems.

The performance of CSODEs in learning and predicting the dynamics of physical systems, as demonstrated through the comparative experiments, highlights their alignment with physical laws and their ability to effectively uncover underlying patterns in data. This motivates the incorporation of appropriate inductive biases when designing neural network architectures for scientific and engineering applications.

In the field of robotics, CSODEs have the potential to change the design and control of robotic systems. They can be applied to disturbance observation, pose estimation, and the design of control strategies. By learning the dynamics of disturbances, CSODEs enable robots to accurately estimate their states and make appropriate compensations in the face of environmental perturbations. For pose estimation, CSODEs can learn the pose dynamics directly from sensor data, establishing adaptive pose estimators that enhance accuracy and robustness in complex environments.

B Further Technical Details for Section 4

B.1 Further Implications of Assumption 1

With a reordering of nonlinearities and their decomposition, there exists an index ω ∈ { 0 , . . . , M } such that for all s ∈ { 1 , . . . , ω } and i ∈ { 1 , . . . , k s } , lim ν →±∞ f i s ( ν ) = ±∞ . Also, there exists ζ ∈ { ω, . . . , M } such that for all s ∈ { 1 , . . . , ζ } , i ∈ { 1 , . . . , k s } , we have lim ν →±∞ ν ∫ 0 f i s ( r ) dr = + ∞ . Here, ω = 0 implies that all nonlinearities are bounded. The sets with the upper bound smaller than the lower bound are regarded as empty sets, e.g. , s ∈ { 1 , 0 } = ∅ in the case that ω = 0 .

B.2 An Example of Activation Functions Satisfying Assumption 3

In practice, Assumption 3 can be verified based on chosen f j ( e.g. , ReLU, Sigmoid, tanh ). For example, for the tanh function,

It is Lipschitz continuous with constant 1 and satisfies:

In this case, we can choose:

If S j 1 , S j 2 , S j,r 3 , H j 1 , H j 2 , H j,r 3 > 0 , the restrictions are relaxed, verifying that Assumption 3 is less strict than Lipschitz continuity.

B.3 Verification Experiment on Assumption 3

To verify the actual existence of matrices in Assumption 3, we designed a supplementary experiment to validate the trained CSODE models, involving three dynamical system time series prediction tasks in the main

experiments. Retrained CSODE models (3 fully-connected layers, 128 units, Softplus; Adam optimizer, 500 epochs, learning rate = 10 -3 , batch=64, MSE loss). We randomly selected 500 sample pairs ( x i , y i ) from each task's test set, calculated the difference ξ i = y i -f ( x i ) where f ( · ) is the trained CSODE model, and used YALMIP to define optimization variables and constraints. We constructed 500 matrix inequalities ( ξ i ) ⊤ Pξ i ≤ 0 and added the semi-definite constraint P = P ⊤ ≥ 0 . Using the Sedumi solver (precision 10 -6 , max 5000 iterations), we solved the optimization problem and recorded the solving time and iteration count. We then performed individual validation for each sample pair to calculate the satisfiability ratio of Assumption 3.

Experimental Results (average of 3 independent experiments) : For the Hindmarsh-Rose model, solving time was 15.3 minutes with 2684 iterations and 99.8% (499/500) satisfiability. The Reaction-Diffusion System task took 18.7 minutes with 3,217 iterations and 99.6% (498/500) satisfiability. The Shallow WaterEquations required 24.2 minutes with 4,105 iterations and 99.4% (497/500) satisfiability. The solver quickly found solutions satisfying Assumption 3, with >99% satisfiability across tasks, supporting our theoretical assumptions for trained models.

B.4 Formulations of Matrix Inequalities in Theorem 1

The linear matrix inequalities in Theorem 1 are formulated as follows.

where

C Notation, Definitions, and Proof

C.1 Related Notation

The symbol R represents the set of real numbers, R + = { ℓ ∈ R : ℓ ≥ 0 } , and R n denotes the vector space of n -tuple of real numbers. The transpose of a matrix A ∈ R n × n is denoted by A ⊤ . Let I stand for the identity matrix. The symbol ∥·∥ refers to the Euclidean norm on R n .

For a Lebesgue measurable function u : R → R q , define the norm ∥ u ∥ ( t 1 ,t 2 ) = ess sup t ∈ ( t 1 ,t 2 ) ∥ u ( t ) ∥ for ( t 1 , t 2 ) ⊆ R . We denote by L q ∞ the space of functions u with ∥ u ∥ ∞ := ∥ u ∥ ( -∞ , + ∞ ) < + ∞ .

A continuous function α : R + → R + belongs to class K if it is strictly increasing and α (0) = 0 , and K ∞ means that α is also unbounded. A continuous function β : R + × R + → R + belongs to class K L if β ( · , r ) ∈ K and β ( r, · ) is a decreasing to zero function for any fixed r > 0 .

C.2 Related Definitions

Definition 2 The system (1) is called forward complete if for all x 0 ∈ R n and u ∈ L m ∞ , the solution x ( t, x 0 , u ) is uniquely defined for all t ≥ 0 .

Definition 3 A forward complete system (1) with the output y ( t ) := ξ ( t ) is said to be

state-independent input-to-output stable (SIIOS) if there exist β ∈ , γ ∈ such that

for any x 0 ∈ R and u ∈

C.3 Proof of the Convergence Theorem

Proof of Theorem 1 The proof developments are divided into two main parts: ISS and Global Asymptotic Stability (GAS) guarantees, as shown in the sequel.

ISS analysis of (1) for boundedness Consider a Lyapunov function

where the vector W i j is the i -th row of the matrix W j . It is positive definite and radially unbounded due to Finsler's Lemma under the condition (6) and Assumption 1. Then, taking the derivative of V ( x ) , one has

Therefore, under the condition (6) , we obtain

for a function α ∈ K ∞ . Under Theorem 1 of [24], one can verify the first condition of the ISS property due to the form of V , and the second relation can be recovered via V ≥ α -1 ( 2 g ( u ) ⊤ Φ g ( u ) ) ⇒ ˙ V ≤ -1 2 α ( V ) . This means that the ISS property of the NN (1) is guaranteed, and so is the boundedness of its solution.

Global attracting solution of the error dynamics (2) Consider another positive definite and radial unbounded (for the variable ξ ) function

Similarly, taking the time derivative of ˜ V :

Then, under the condition (7) and Assumption 3, it can be deduced that

Therefore, with the conditions (7) , we can substantiate that the system (1) , (2) is SIIOS with respect to ξ , meaning that the solution is GAS. This completes the proof.

D Experiment Setting Details

This section provides a detailed overview of the parameters used in our experiments to ensure reproducibility. Detailed settings for each modeling task are specified in the corresponding tables: Table 4 for the HindmarshRose model, Table 5 for the Reaction-Diffusion system, and Table 6 for the Shallow Water Equations.

Each table includes configurations such as physics parameters, the MLP network structure for function f ( · ) , optimizer, loss function, learning rate, and training epochs, among other parameters.

E Experimental Statistical Analysis

This section shows on the statistical methods employed to ensure the reliability and significance of our experimental results, in accordance with the required scientific standards. We focus on the computation of standard deviation percentages as a measure of variability and consistency across experimental runs in Experiments.

E.1 Statistical Methods for Reliability Analysis

To assess the reliability of our experimental setups, the standard deviation for each performance metric was computed based on multiple runs. The formula for standard deviation is:

In the formula, σ represents the standard deviation, z i denotes individual observations, z is the mean of these observations, and N is the total number of observations.

The percentage standard deviation is calculated to provide a relative measure of variability:

This measure serves as our error bar, which reflects the extent of variability due to experimental conditions such as model initialization and parameter settings.

E.2 Statistical Measures in Experiments

As shown in Table 7, the vast majority of the standard deviation percentages are below 10% for experiments in Section 6, indicating a high degree of statistical robustness across most metrics. This consistency underscores the reliability of our experimental design and the effectiveness of our model implementations.

The same statistical methods employed in Section 6 are also utilized to analyze the Model Scaling experiments, as outlined in Section 7. The heatmap in Figure 6 displays the standard deviation percentages for each metric, all of which remain below 25%, indicating reliability in scaling experiments as well. Additionally, in Section 5, the shaded areas around the loss curves further illustrate the variability of our results. These shadows in the plots provide a visual representation of the variability, akin to error bars, showing fluctuations within a confined range. In Table 1, the test error rate variability is maintained within a tight range of 0.005%.

E.3 Statistical Considerations in Models Performance Comparison

In our study, we have chosen a straightforward approach to presenting model performances directly through metrics in Table 2 without deploying traditional statistical tests such as ANOV A or t-tests. This decision was based on the clarity and immediacy with which the performance metrics convey the effectiveness of each model. By doing so, we aim to keep the main article succinct and focused on substantive evaluations, thereby avoiding the potential complication of statistical tests that might obscure the main findings. This method ensures that the paper remains accessible to readers, emphasizing practical implications.

F Experimental Result of Integrating CSODEs with Transformer Layers

To compare the performance of Transformer, TLODE, and CSTLODE models on time series prediction tasks, we conduct experiments on three datasets: Hindmarsh-Rose Model, Reaction-Diffusion Model, and Shallow Water Equations. The experimental results are shown in Table 8. It can be observed that TLODE and CSTLODE overall outperform the original Transformer model, indicating that combining the Transformer with Neural ODE can enhance model performance on time series prediction tasks.

Further comparing TLODE and CSTLODE, we find that CSTLODE achieves better results on most evaluation metrics. This suggests that by introducing the control term g ( · ) , the ControlSynth framework can better guide the model in learning the dynamics of time series. Specifically, on the Hindmarsh-Rose and Reaction-Diffusion datasets, CSTLODE's MSE, MAE, and CD indicators are significantly better than those of TLODE. On the

Shallow Water Equations dataset, the performance of the two models is more comparable, with TLODE slightly better in MSE and CSTLODE slightly better in MAE.

Overall, the experimental results validate the effectiveness of combining the ControlSynth framework with Transformer. The introduction of the control term g ( · ) can further enhance model performance. This provides a new approach to the field of time series prediction. In the future, we plan to apply and validate the combination of ControlSynth and Transformer in more practical scenarios. We will also explore the possibilities of combining ControlSynth with other advanced time series prediction models.

G Additional Qualitative Prediction Results

Predicted by Neural ODE

+F

Predicted by Augmented Neural ODE

→Predicted by ControlSynth Neural ODE

→GroundTruth

2

0.9

0.8

0.7

0.6

.5

0.4

0.

-axis axis

(a) Hindmarsh-Rose neuron dynamics

Gray-Scott Model of Reaction-Diffusion, U (dimensionless)

Initial Condition

Prediction GroundTruth

1.1

0.9

0.8

NeuralODE

Augmented Neural ODE

ControlSynthNeural ODE

0.7

0.6

0.5

(b) Reaction-Diffusion system dynamics

2D Shallow Water Equations, Water Depth h (Meters)

Initial Condition

Prediction Ground Truth

Neural ODE

Augmented Neural ODE

ControlSynthNeural ODE

(c) Shallow Water Equations

10

0.8

0.7

90

0.5

0.4

'0

0.2

1'0

This plot compares the time series prediction of ground truth, NODE, ANODE, and CSODE models for the Hindmarsh-Rose neuron dynamics.

Initial conditions and later system states for the Reaction-Diffusion dynamics, showing predictions from NODE, ANODE, and CSODE alongside ground truth.

Displays initial conditions and predictions by NODE, ANODE, and CSODE for Shallow Water Equations, compared with the ground truth.

In this section, we present additional qualitative results designed to enhance the reader's understanding of the complex tasks modeled and the performance of various models, specifically focusing on NODEs, ANODEs, and CSODEs. These visual comparisons, as illustrated in Figure 7, provide a clear and intuitive view of each model's capability to predict and simulate the dynamics of complex systems.

The visual results are instrumental in demonstrating the capabilities of the CSODEs, particularly its adeptness in capturing the intricate behaviors of dynamic systems more effectively than traditional models. This is evident from the accurate time series predictions and the fidelity in spatial-temporal dynamics shown across different scenarios-from neuronal activity in the complex nonlinear Hindmarsh-Rose model to fluid dynamics in Shallow Water Equations, and the chemical kinetics in Reaction-Diffusion processes.

H Further Results on Model Scaling

H.1 Comparison with Neural ODE and Augmented Neural ODE

We conducted comprehensive experiments comparing CSODE, original NODE, and ANODE under equivalent parameter counts achieved through width scaling. Taking the Reaction-Diffusion task as an example, with CSODE configured with 2 subnetworks, we systematically evaluated model performance across increasing network widths (128, 256, 512, 1024, 2048). Tables 9 and 10 present the MAE and MSE metrics respectively.

The results demonstrate a consistent pattern: while all models show improved performance with increased width, CSODE maintains a clear advantage across all scales. At smaller widths (128-256), the performance gap is particularly notable, with CSODE achieving an MAE of 0.060 compared to NODE's 0.083 and ANODE's 0.078 at width 128.

As network capacity increases, CSODE's advantage becomes even more pronounced. At larger widths (10242048), while NODE and ANODE show signs of performance saturation (MAE stabilizing around 0.06), CSODE continues to improve, reaching an MAE of 0.030 at width 2048. This suggests that CSODE makes more effective use of additional parameters, likely due to its structured approach to handling dynamical systems.

The MSE metrics further corroborate these findings, showing that CSODE achieves consistently lower error rates across all width configurations. The improvement in MSE is particularly notable at larger widths, where CSODE achieves an MSE of 0.0009 at width 2048, significantly outperforming both NODE (0.0040) and ANODE (0.0033). This suggests that CSODE not only performs better in terms of absolute errors but also shows superior stability in handling larger variations in the prediction task.

These results highlight CSODE's superior scalability and efficient parameter utilization, demonstrating that its architectural advantages persist across different model capacities. The consistent performance improvements with increased width also suggest that CSODE may benefit from even larger model scales in more complex applications. Furthermore, the results indicate that the improvement in the learning ability of these two models with increasing width is not as significant as CSODE.

H.2 Convergence Study in ControlSynth Neural ODE Model Scaling

In the context of scaling the CSODE framework, the impact of network configurations such as subnetwork count and width on model performance is critical. Several key observations are performed:

LossforWidth512withDifferentSub-network Numbers

LossforSub-network3withDifferentWidths

Subnet 1 -Train

0.025

Width128-Train

0.025

Subnet 1-Valid

Subnet 2-Train

Width 256-Train

Subnet 2-Valid

0.020

0.020

Subnet3-Train

Subnet3-Valid

Width512-Valid

Subnet 4-Train

0.015

Width 1024 -Train

S0T

0.015

SS0T

Subnet 4-Valid

Width1024-Valid

Subnet5-Train

Subnet 5-Valid

0.010

0.010

0.005

0.005

200

400

600

800

1000

200

400

600

800

1000

Epochs

Epochs

Increasing Sub-network Count at Fixed Width : When the width is held constant, for instance at 512, an increase in the number of sub-networks initially raises the loss due to the augmented complexity of the network as the left figure in Figure 8 shown. However, as training advances, the model effectively adapts to this increased complexity, ultimately achieving a lower level of loss than configurations with fewer sub-networks. This indicates that a higher number of sub-networks can enhance the model's depth of learning and capability for feature extraction, leading to improved performance after extensive training.

Increasing Width at Fixed Sub-network Count : With the sub-network count fixed, such as at three, expanding the width shows a marked improvement in learning capabilities, as the right figure in Figure 8 shows. The additional parameters afforded by a broader network aid in capturing more complex features of the data, which accelerates the reduction in both training and validation losses. This setup not only speeds up the learning process but also tends to improve generalization across different tasks.

Overall Convergence Trends : Whether increasing the sub-network count or the width, all configurations eventually exhibit a trend toward convergence in losses. This suggests that, with suitable adjustments in training duration and parameter settings, various network configuration designs of CSODEs are capable of uncovering and effectively learning the intrinsic patterns hidden in the data.

Summary : These observations align well with the goals of the CSODE framework, which is designed to enhance NODE system scalability while ensuring strong convergence and high performance.

I Solver, Optimizers, and Hyperparameters

In the training and evaluation of neural differential equation models, the selection of solvers, optimizers, and hyperparameters plays a pivotal role in determining both the performance and efficiency of the models. This section delves into the theoretical underpinnings of these components and provides an in-depth analysis of their impacts through comprehensive experimental studies.

I.1 Impact of Solver Choice

Numerical solvers are essential for approximating solutions to differential equations, which form the backbone of Neural Ordinary Differential Equations (NODEs) and their variants. The choice of solver affects both the accuracy and computational efficiency of the model. Solvers can be broadly categorized into explicit and implicit methods, with further subdivisions based on their order of accuracy and adaptive step-size capabilities.

The Euler method is a first-order explicit solver known for its simplicity and computational efficiency. Despite its low accuracy, it serves as a baseline for comparison due to its straightforward implementation and ease of understanding.

Dopri5 , an explicit Runge-Kutta method of order 5, is a higher-order adaptive solver that adjusts step sizes based on the estimated error, providing a balance between accuracy and computation time. Adaptive solvers like Dopri5 can handle stiff equations more effectively by modifying the step size dynamically, thereby improving numerical stability.

I.1.1 Experimental Setup

To evaluate the impact of solver choice, we conducted experiments on multiple tasks, including the ReactionDiffusion model, which is known for its complex dynamics and sensitivity to solver accuracy. The models compared include:

NODE : The standard Neural Ordinary Differential Equation model.

ANODE : Augmented Neural ODE, which adds additional dimensions to improve model expressiveness.

CSODE : Our proposed ODE model with composite sub-network, is designed to enhance stability and performance.

We implemented both the Euler method and the Dopri5 solver within each model architecture.

I.1.2 Results and Discussion

Table 11: Mean Absolute Error (MAE) Comparison between Euler and Dopri5 Solvers on the Reaction-Diffusion Task

Table 11 presents the MAE for each model using both the Euler and Dopri5 solvers. The results indicate that while Dopri5 offers a slight improvement in MAE across all models, the relative performance ranking remains consistent. Specifically, CSODE maintains its superior performance regardless of the solver used, underscoring its inherent robustness.

Computational Overhead : As expected, Dopri5 incurs a higher computational cost compared to the Euler method. In our experiments, Dopri5 increased training time by approximately 35% relative to Euler. However, the trade-off between computational cost and marginal accuracy gains must be carefully considered, especially in scenarios where real-time inference is critical.

Stability Considerations : Higher-order solvers like Dopri5 exhibit better stability properties, particularly in handling stiff differential equations. Although our primary tasks did not involve highly stiff systems, the use of Dopri5 can be beneficial in extending the applicability of these models to more complex dynamical systems.

I.2 Impact of Optimizer Selection

Optimizers are algorithms or methods used to adjust the weights of neural networks to minimize the loss function. The choice of optimizer can significantly influence the convergence speed, the quality of the solution, and the model's ability to escape local minima.

Adam is a widely used optimizer that combines the benefits of both AdaGrad and RMSProp. It maintains per-parameter learning rates adapted based on the first and second moments of the gradients, making it suitable for problems with sparse gradients and noisy data.

L-BFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno) is a quasi-Newton optimizer that approximates the inverse Hessian matrix to guide the search direction. It is particularly effective for problems where high precision in convergence is desired, albeit at the cost of higher memory usage and computational overhead.

I.2.1 Experimental Setup

To assess the impact of optimizer choice, we extended our experiments by incorporating the L-BFGS optimizer alongside Adam. The models evaluated include NODE, ANODE, and CSODE across all primary tasks, with the

Reaction-Diffusion model serving as the exemplar case. The hyperparameters for each optimizer were carefully tuned to ensure a fair comparison:

Adam : Learning rate = 0.001, β 1 = 0.9, β 2 = 0.999, ϵ = 10 -8 .

L-BFGS : Maximum iterations = 1000, tolerance = 10 -5 .

I.2.2 Results and Discussion

Table 12 illustrates the performance gains achieved by switching from Adam to L-BFGS across different models. On average, L-BFGS improves the MSE by approximately 16%, with CSODE benefiting similarly to NODE and ANODE. Notably, while all models experience performance enhancements, CSODE consistently outperforms the other variants, reinforcing its superior architecture.

Robustness to Local Minima : The quasi-Newton nature of L-BFGS allows it to navigate the loss landscape more effectively, reducing the likelihood of getting trapped in shallow local minima. This property is particularly advantageous for complex models like CSODE, which possess intricate loss surfaces due to their composite structure.

Memory and Computational Trade-offs : While L-BFGS offers superior convergence properties, it demands significantly more memory and computational resources, especially for large-scale models. In our experiments, L-BFGS required approximately twice the memory footprint compared to Adam and increased the per-iteration computation time by 40%. Therefore, the choice between Adam and L-BFGS should consider the available computational resources and the specific requirements of the task at hand.

I.3 Impact of Hyperparameters Adjustment

Hyperparameters such as learning rate and batch size are critical in training neural networks. The learning rate determines the step size during the optimization process, directly influencing the convergence speed and stability. A learning rate that is too high can cause the optimization to overshoot minima, leading to divergence, while a rate that is too low can result in excessively slow convergence or getting stuck in local minima.

Batch size affects the gradient estimation's variance and the computational efficiency. Smaller batch sizes can introduce more noise into the gradient estimates, potentially aiding in escaping local minima but making convergence noisier. Larger batch sizes provide more accurate gradient estimates but may lead to poorer generalization and require more memory.

I.3.1 Experimental Setup

We conducted a systematic hyperparameter search to examine the sensitivity of CSODE to changes in learning rate and batch size. The ranges explored were:

Learning Rate : [0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01]

Batch Size : [32 , 64 , 128 , 256]

Each combination of learning rate and batch size was evaluated on the CSODE model using the ReactionDiffusion task. Performance was measured in terms of MAE on the validation set, and each setting was replicated three times to account for stochasticity.

Additionally, we investigated the impact of network architecture parameters, specifically network depth and width, to understand their influence on CSODE's performance in main text.

by CSODE across various learning rates and batch sizes. The results reveal that CSODE demonstrates strong robustness to hyperparameter variations within certain ranges.

Learning Rate Sensitivity : Within the learning rate range of [0 . 0005 , 0 . 005] , CSODE's MAE remained within a 5% fluctuation band, indicating stable performance. Specifically, at a fixed batch size of 128, varying the learning rate within [0 . 0005 , 0 . 005] resulted in a standard deviation of only 1.2% in MAE. However, at a learning rate of

0 . 01 , the model exhibited signs of instability, likely due to gradient explosion, while at 0 . 0001 , convergence was significantly slower, potentially leading to gradient vanishing issues.

Batch Size Influence : CSODE maintained consistent performance across batch sizes from 32 to 256. Smaller batch sizes introduced minor variances but did not adversely affect the overall performance, suggesting that CSODE can be effectively trained under different memory constraints.

Gradient Stability : The increased network complexity necessitates careful hyperparameter tuning to maintain gradient stability. At higher depths and widths, the model becomes more susceptible to gradient vanishing or explosion. Therefore, selecting an appropriate learning rate is crucial to ensure that gradients remain within a reasonable range, facilitating effective training.

I.3.2 Recommendations for Hyperparameter Tuning

Based on our experimental findings, we propose the following guidelines for tuning hyperparameters in CSODE:

Learning Rate : Utilize a learning rate within the [0 . 0005 , 0 . 005] range to achieve stable and efficient convergence. Employ learning rate schedulers or adaptive methods to fine-tune within this range dynamically.

Batch Size : A batch size between 64 and 256 is recommended. Smaller batch sizes can be used if memory constraints are present, without significant degradation in performance.

Optimizer Selection : For scenarios requiring high precision and better convergence properties, LBFGS is preferable. However, for large-scale or real-time applications, Adam remains a viable choice due to its efficiency.

J Ability to Adapt to Different Spatial Scales

In practice, CSODE demonstrates superior robustness when handling dynamical system datasets across different spatial scales, as evidenced by the stability of its performance metrics. Table 13 presents a comprehensive comparison of the model's stable ability across different spatial scales in the Reaction-Diffusion task.

Table 13: Performance stability comparison across different spatial scales in the Reaction-Diffusion model. The percentages show the standard deviation of model performance. Lower values indicate better stability. Best performing models are marked in blue , second-best in brown .

While all models show comparable performance in the original scale setting (NODE: 5.3%, ANODE: 4.6%, CSODE: 5.3%), significant differences emerge when the spatial domain is expanded. At 5 times the original scale, NODE and ANODE show notable degradation in stability (7.8% and 7.1% respectively), while CSODE maintains a relatively stable performance (5.5%). This trend becomes even more pronounced at 10 times the original scale, where NODE and ANODE further deteriorate to 8.8% and 7.3%, while CSODE maintains its stability at 5.5%.

Particularly noteworthy is the stability drop from the original to 10× scale: CSODE shows only a 0.2% increase in standard deviation, compared to significantly larger increases for NODE (3.5%) and ANODE (2.7%). This remarkable stability across different spatial scales suggests that CSODE's architecture, particularly its control term component, provides inherent advantages in handling spatial scale variations in dynamical systems.

These results indicate that CSODE not only performs well in standard settings but also maintains consistent performance across varying spatial scales, making it particularly suitable for applications where spatial scale invariance is crucial.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clarified the paper's contributions and scope in both abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitations of this work at the end of conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions and proof can be found in Section 4 and Appendix C.3, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In this study, we provide detailed descriptions of model architectures, experimentspecific configurations, simulation parameters, and hyperparameters in Sections 6.1 and D, along with comprehensive dataset information in Section 6.2. Additionally, the scaling of model structures and hyperparameters is thoroughly elaborated in Section 7, ensuring that the reader can accurately replicate our experimental results and verify the statements.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our GitHub repository URL, provided in Section 1, houses all necessary source code, detailed installation instructions, and step-by-step guides for setting up the environment running experiments, ensuring replicability. We provide comprehensive data generation algorithms for synthetic datasets, which collectively facilitate the reproduction of experimental results detailed in this study.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

To preserve anonymity, the source code has been hosted on a dedicated GitHub account created specifically for this submission, which does not contain any other repositories or identifiable information.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The detailed information on training and testing is thoroughly presented in Section 6 and in Appendix D, including data splitting methods, the basis for hyperparameter selection, types of optimizers, and other experimental configuration details, ensuring a complete understanding of the results and the reproducibility of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Our research assesses the reliability of experimental results using standard deviation percentages as error bars, clearly explaining the calculation formulas and statistical significance. Appendix E lists the standard deviation percentages for all performance metrics from experiments, comprehensively demonstrating the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details on the GPU type and CUDA platform used across all experiments are documented in Section 5 and Table 1, highlighting the uniform computational environment. Due to the consistency in resource use across different Neural ODE models tested under similar conditions, specific compute requirements for individual experiments are not listed separately, ensuring that all necessary information for replication is efficiently conveyed without redundancy.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research adheres to NeurIPS ethical guidelines through the use of synthetically generated data, ensuring no risk to privacy, and by sharing all methodologies and code to promote reproducibility and transparency. We ensure fairness in algorithmic design, respect for intellectual property, consider the societal impacts to avoid harm, and maintain anonymity in our submissions.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See Appendix A for the details of broader impacts of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We have not used data or models that have a high risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The related assets have been suitably credited and mentioned.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All assets utilized and created are synthetic or derived from open-source resources, requiring no consent from third parties.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our experiments are not crowdsourcing and this work is irrelevant to human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our experiments are irrelevant to such risks.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an extension to Neural ODEs. At it's core this is an architectural change to Neural ODEs, adding more important structure to the dynamics function through a control signal $u(t)$. The paper shows that this can lead to rich non-linear dynamics with convergence guarantees. Extensive experiments on various synthetic and real systems show that structuring the dynamics in this way leads to faster training, better performance and reliable scaling.\n\nSTRENGTHS:\nThis is a really solid paper in my opinion. The paper identifies a problem with Neural ODEs, proposes a theoretically grounded solution. As far as I can tell the theory is correct. The evaluation is extensive, generally exploring all important aspects of the work.\n\nThe appendix is rich with important details, showing further results.\n\nWEAKNESSES:\nThe paper would be improved most by restructuring the writing. I believe the related work section would be better after the introduction rather than before the conclusion. Section 2 needs to be slightly improved. The description jumps straight into the mathematical detail without very much explanation of $u(t)$ or the subnetworks. I believe the paper would be improved significantly by reducing Section 3 by moving details to the appendix and giving a more intuitive explanation of the main result. Additionally use the space to give a more accessible explanation of CSODE in Section 2.\n\nThere should be some sort of comparison to Neural CDE, since it also uses control signals and is considered SOTA for Neural ODE methods. Other methods that could be compared to to improve the evaluation is Latent ODE and ODE-RNN. \n\nSection 6 on scaling is good but should involve comparisons to other models. Since a claim of the paper is that CSODE has guarantees especially for larger models then this should be shown against NODE and NODE variants. Additionally for the scatter plot in Figure 6, as far as I can tell this is about including more subnetworks, this is not clear from the plot legend or figure caption.\n\nAnother claim is that this method inherently works at different spatial scales. There does not appear to be an experiment testing this, is this possible?\n\nQUESTIONS:\n- Why is the Euler method used for ODE integration and not Dopri5? This is the standard ODE solver used in Neural ODE works.\n- Is it possible to include results for Neural CDE/ODE-RNN/Latent ODE?\n\nLIMITATIONS:\n- Limitations are well addressed in the conclusion.\n- A thorough broader impact statement is provided in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

comments: SUMMARY:
The authors present a new class of continuous-time neural networks, ControlSynth ODEs. This new class of ODEs are able to learn the dynamics of physical systems faster and more precisely. In addition the authors provide theoretical convergence guarantees for these new models, and demonstrate their effectiveness compared to traditional Neural ODEs.

STRENGTHS:
Although the analysis is difficult to follow, the theoretical convergence guarantees make these an attractive model.
In the benchmark experiments, adaptive CSODEs achieve are superior compared to the other models.

WEAKNESSES:
The explanation of the model is far too condensed and unclear.

QUESTIONS:
It would be beneficial to outline what architectural changes are made from standard NODEs to CSODEs.
In addition, to what hypothesis motivated these changes. 
Why are they termed ControlSynth ODEs? The name is uninformative.
What are the inductive biases of these models?

LIMITATIONS:
The limitations of the method are unclear. 
What are the inductive biases of these models?
How does sensitivity to hyperparameters emanate during training?
Why is this specific architecture more difficult to train than the alternatives?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-09

comments: SUMMARY:
The paper introduces ControlSynth Neural ODEs (CSODEs), a novel approach to modeling dynamical systems with neural ordinary differential equations (NODEs). The proposed models constraint the system to a convergent once. The CSODE framework incorporates an additional control term to ensure the existence of the solutions. The authors present theoretical guarantees for convergence and demonstrate the superior performance of CSODEs compared to other NODE variants in their experiments.

STRENGTHS:
* **Experiments**: The authors validate their model on learning dynamical systems. This is a welcome change from several papers testing neural odes on problems like image classifications and such, where neural odes tend not to be the best solution. Hopefully more papers do the same!
* **Technical Rigor**: The paper provides detailed theoretical analysis and proofs for the convergence of CSODEs.

WEAKNESSES:
Overall the paper looks solid. 1 questions regarding the training pointed out in the next section.

QUESTIONS:
In the preliminary experiments section, neural odes seems not to converge properly. Was an alternate optimizer like LBFGS or BFGS tried? In most cases where vanilla neural ODEs tend to not converge, it is more often than not an optimizer issue rather than a model issue.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
In this paper, a method called ControlSynth Neural ODEs is proposed. This method is essentially defined as a neural ordinary differential equation with a control input. In particular, the authors focus on the convergence, of which definition requires the existence of a solution and also the asymptotic stability of the solution. In this paper, theoretical conditions to satisfy the convergence are presented. Some numerical experiments are also provided.

STRENGTHS:
In my opinion, the main contribution is the theoretical analysis of convergence. In fact, stability is very important for the use of neural ordinary differential equations, which are often unstable when used for modeling.

WEAKNESSES:
In this paper, conditions for convergence are presented. Thus, it is expected that the control input and the neural ordinary differential equations should be trained so that these conditions are satisfied. However, I could not find any description of how to achieve this in the paper. In fact, the conditions for the convergence assume the existence of several matrices, and it seems difficult to design an optimization algorithm for training while satisfying these conditions.

QUESTIONS:
As it is stated that the convergence is guaranteed in this paper, I imagine that there must be some tricks in designing the control input and the learning algorithm so that the conditions for the convergence are always satisfied. How are they designed?

LIMITATIONS:
There seems to be no problem.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13