Oscillation Neural Ordinary Differential Equations
Authors: Muhao Guo
Yang Weng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Ordinary Differential Equations
Continuous Deep Learning

Abstract: Neural ordinary differential equations (NODEs) have received a lot of attention in recent years due to their memory efficiency. Different from traditional deep learning, it defines a continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning. However, it has several obvious limitations, such as a NODE is not a universal approximator, it requires a large number of function evaluations (NFEs), and it has a slow convergence rate. We address these drawbacks by modeling and adding an oscillator to the framework of the NODEs. The oscillator enables the trajectories of our model to cross each other. We prove that our model is a universal approximator, even in the original input space. Due to the presence of oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed. We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed. The experiments show that our model can achieve better results compared to the existing baselines.

Under review as a conference paper at ICLR 2023
OSCILLATION NEURAL EQUATIONS
ORDINARY DIFFERENTIAL
Anonymous authors Paper under double-blind review
ABSTRACT
Neural ordinary differential equations (NODEs) have received lot of attention in recent years due to their memory efficiency. Different from traditional deep learning, it defines continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning. However, has several obvious limitations, such as a NODE is not universal approximator, it requires a large number of function evaluations (NFEs), and has a slow convergence rate. We address these drawbacks by mod- eling and adding an oscillator to the framework of the NODEs. The oscillator enables the trajectories of our model to cross each other. We prove that our model is universal approximator, even in the original input space. Due to the presence oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed. We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed. The experiments show that our model can achieve better results compared to the existing baselines.
INTRODUCTION
Neural Ordinary Differential Equations (NODEs) (Chen et al. 2018) are the latest continuous deep learning architectures that were first developed in the context continuous recurrent networks (Co- hen & Grossberg 1983). This continuous deep learning architecture provides new perspective that theoretically bridges the gap between deep learning and dynamic systems. This deep learning architecture can be efficiently trained with backpropagation and has shown great promise on several tasks including modeling continuous time data, classification, and building normalizing flows.
The core idea of NODE is to use neural network to parameterize the vector field (Chen et al.) 2018 Kidger 2022). Typically, a simple neural network is enough to represent the vector field, which will be optimized during the training process. Based on the well-learned vector field, trajec- tories will be obtained as the estimate functions. However, this architecture has several limitations. First, NODEs cannot learn any crossover-mapping functions (Dupont et al. 2019), which results in them not being universal approximators. Second, to optimize the vector field, it will need many func- tion evaluations during both forward evaluation and backpropagation processes of training. Third, the time and convergence rate of the training process is relatively slow.
The first limitation is caused by the continuity of vector-field-based trajectories because the trajec- tories in NODES cannot cross each other at the same time (Massaroli et al., 2020) Norcliffe et al.) 2020). This property causes NODEs to be powerless against some special topologies. such as the cases of concentric circles and intersecting lines mentioned by Dupont et al. (019). We conjecture the reason for the second limitation is caused by the straightforward optimization approach of the vector field. There is no guarantee that learning the vector field is a better choice than learning the estimated functions directly. Sometimes it will need many function evaluations to optimize the vector field, so the difficulties go beyond learning the estimated functions themselves. The third limitation is caused by the trade-off between accuracy and speed for the ordinary differential equa- tion solver (ODE solver). NODEs perform forward evaluation and backpropagation calculations via ODE solvers, which can be treated as black boxes. If we need to ensure the accuracy of an ODE solver, then we must sacrifice the speed.

---

Under review as a conference paper at ICLR 2023
To address the first problem, intend to add some discrete elements during the process of learning the trajectories. A discrete element can be realized by a "jump" in the trajectory. This jump allows the trajectories of the NODEs cross the same time. It solves the above-mentioned cases. We also give proof in Section 4.1 that our approach is a universal approximator. To solve the second and third problems, optimizing the vector field at the same time. It shifts part of the burden from optimizing the vector field optimizing the original estimated function. The addition of g will make the final optimized vector fields less complicated, thus a "simple" vector field can be utilized to estimate the function well. This will result fewer NFEs, less training time, and faster convergence.
propose to join a function 9 that directly optimizes the trajectories while
Based on the above two points, we design an oscillator to enhance neural ordinary differential equa- tions, namely Oscillation Neural Ordinary Differential Equations (ONODE). The architecture shown in Figure This oscillator is designed to achieve both "jumping" of trajectories and opti- mizing the estimation function while simultaneously optimizing the vector field. This design solves the three problems mentioned above to some extent. Our proposed method is not only a universal approximator, but it also reduces the NFEs as well as improves the training convergence speed.
The Proposed Method
The Training Process
Well-trained Vector Field
Input
Output
ODE solver
Oscillator Modeling
Vector Field Modeling
Oscillator
The Learned Mapping Functions
Figure 1:
The diagram on the left shows the structure of our proposed model, and the right shows how the oscillator helps our model learn the cross-trajectory by example.
Specifically, the vector field is parameterized by simple neural network, the same as the NODEs proposed by Chen al. (2018). The vector field can be optimized by ODE solvers which can be used such as Runge (1895), Kutta (1901), and Hairer et al. (1993). The oscillator we designed is parameterized by a shallow neural network structure and it has only one hidden layer. The oscillator can be located before vector field modeling, or after the ODE solver.
The oscillator modeling structure may vary slightly in different tasks. For example, in the usual classification tasks and extrapolating time series tasks, these structures will have the perception of one hidden layer. In the image classification task, the structure will have two convolutional layers with activation function. It is worth noting that since our oscillator is parameterized by a shallow neural network that does not take much space, we can still consider the model to be memory efficient. We will illustrate this in Section 5 Besides, our oscillator simplifies the learning complexity of the vector field, which leads to reduction in NFEs and convergence speed. A simple example is shown in Figure rright, the final trained vector field will be very simple one.
2 RELATED WORK
The basic ideas of neural ordinary differential equations were originally considered in incco--Mrttine ct al. (1992), Rico- Martinez Kevrekidis (1993), and Rico-Martinez et al. (1994). For example, Rico-Martinez et al. (1992) proposed vector field that can be trained using Multilayer Perceptron (MLP). Rico-Martingg & Kevrekidis (1993) used an implicit integrator and recurrent networks for continuous time modeling of nonlinear systems. Chen et al. (2018) specified the architecture of NODEs and applied it to several tasks and achieved good results, including image classification, continuous normalizing flows, and latent ODEs, leading explosion of interest in NODEs.
2

---

Under review as a conference paper at ICLR 2023
NODEs are class of model that transform data continuously through infinite depth architec- tures (Norcliffe et al. 2020). In particular, NODEs can be seen as a continual version of Residual Networks (He et al. 2016) by taking the discretization step infinitely small. This continuous in- finite depth architecture will bring several benefits, one the most important is memory efficiency. According to Chen et al. (2018), the scalar-valued loss with respect all inputs of any ODE solver can be computed directly without backpropagating through the operations of the solver. The inter- mediate quantities of the forward pass will not need to be stored. It causes the NODEs so they can be trained with constant memory cost.
However, this continuous infinite depth architecture will bring several drawbacks. Dupont et al. (219)) showed that NODEs learn representations that preserve the topology of the input space and prove that the existence of functions in NODEs cannot be represented. To address these limitations, Dupont et al. (2019) introduced Augmented NODEs (ANODEs) which add extra dimensions for the vector field learning. This approach increases the degrees of freedom of the trajectory by elevating the dimensionality,, thus circumventing the problem that trajectories cannot be crossed. However, does not inherently overcome the problem that trajectories cannot be traversed and increases the original parameter space. Other limitations of NODEs include the NFEs which
often become prohibitively large (Dupont et al. (2019) and the speed is slow as long as it is needed to obtain good precision (Chen al.] 2018)). These limitations usually caused by the need for the NODE to learn a complex vector field.
Due to the generality of the NODEs' framework, there is already a lot of related work based on NODEs, Some work focuses on modeling for the time series data. For example, Norcliffe et al. 2020 focused on the learning dynamics system and proposed the Second Order Neural Ordinary Differential Equations (SONODEs). It can be seen as a system of coupled first-order NODEs. Dia & Benson 2019) introduced Neural Jump Stochastic Differential Equations that provided a data- driven approach to learn continuous and discrete dynamic behavior. Kidger et al. (2020) proposed Neural Controlled Differential Equations to model the irregular time series. |Morrill et al. (2021) used Neural Rough Differential Equations for modeling long time series. Other works focused on physical modeling. For example,Greydanut Greydanus et al. (219)) introduced Hamiltonian Neural Networks for learning certain physical laws that follow conservation laws. Cranmer et al. (2020) proposed Lagrangian neural networks to learn certain physical laws without requiring canonical coordinates.
PRELIMINARY
NODES a family of deep neural network models that can be interpreted continuous version of Residual Networks (He et al., 2016). Recall the formulation of a residual network:
h++1 h4 f(,, 0y).
(1)
where the f is the residual block and the f represents the parameters of f. The left side of Equation
can be seen the denominator is 1, it can be represented by Acs1 [(h,, 0y). When the
number of layers becomes infinitely large, and the step becomes infinitely small, Equation I will
become an ordinary differential equation format, as shown in Equation
lim hy h dh(() f(h(t),t,g)). dt->0
(2)
dt
Thus, the NODE will have the same format as an ODE:
(((t) ((()=((((()...)) f(h(t), t, 0,), h(0) = Xo,
(3)
where Xo is the input data. Typically, f will be some standard simple neural architecture, such as MLP. The fy represents trainable parameters in
To obtain any final state of h(t) when = T, all that is needed is to solve an ordinary differential equation with initial values, which called an initial value problem (IVP):
h(T) h(0) L-" 5(N()),,00))d.
Thus, NODE can transform from h()) to h(T)) through the solutions to the initial value problem (IVP) of the ODE. This framework indirectly realizes a functional relationship x - F(z) like a general neural network.
3

---

Under review as a conference paper at ICLR 2023
By the properties of ODEs, NODEs are always invertible; or alternatively. integrate f. The Adjoint Sensitivity Method (Pontryagin et al.) 1961) based on reverse-time integration of an expanded ODE, allows for finding gradients of the initial value prob- lem solutions (T) with respect to parameters y and the initial values (0). This allows the training NODE use gradient descent, which allows them combine with other neural network blocks.
reverse the limits of integration,
4
THE PROPOSED METHODS
4.1 OSCILLATION NODE: A UNIVERSAL APPROXIMATOR
As we mentioned, since the trajectories in NODES cannot cross each other at the same time, a NODE on its own does not have universal approximation capability. An example from Dupont et al. (2019) considers continuous, differentiable, invertible function f(x) = -x on X = R. There is no ODE defined on I. that would result in 2T oT(z)) -oo In ODEs, paths (z,, t) between the initial value (Io, 0) and final value (zr: T) have to be continuous and cannot intersect in X' for two different initial values, and the paths corresponding to æ > x and 0 - would need to intersect.
Generally, there is class of functions that a NODE cannot model (Zhang et al. 2020). Let = RP and 2 C X' be set that partitions X' into two more disjointed. connected subsets C, for i = [m]] that is, X' Z (U, C.). Then, no NODEs can model a mapping h X + X' that satisfy both (1) is an identity transformation on 2, i.e., Ve 2, h(2) and (2) h maps some x E C, for i +
Ours
NODE
Reasoning that the trajectories cannot cross at the same moment, we intend to add the factor of dis- cretization before the continuous trajectory. This discretization makes the trajectory no longer subject the effect of not being able to cross over. If the tra- jectory does cross at some point, then the crossover will be achieved by the oscillator. Based on this idea, we introduce the Oscillation NODEs (ONODEs). It solves the problem that NODEs' trajectories cannot be crossed, thus it is universal approximator and we provide proof in Theorem [.1 In addition, it has fewer NFEs and faster convergence speed, which will discuss in Section 4.2
The top two plots show the flow trajectories. The middle layer shows the fea- ture evolution, whereas the first plot in the middle layer represents the toy dataset. The bottom two plots show the mapping to the in- put space. The left side is our method, and the right side is NODE.
Figure 2:
Specifically, oscillator O Rd Rd approxi- mates function g X + X". The oscillator O is parameterized by perceptron with single hidden layer and acts on the input data No. The output ((x)) and input x, of the oscillator will have the same di- mension Rd. A formal description is as follows:
h((1 g(xx, 0g), N((()= = f(h g(x,, (,),, A(0)
g(xo 0g),
(5)
where the Xo is the input data, h(()) is the function want to estimate. The function is the first order derivative equation of & with respect to time i, and Oy represents the parameters in f. The 0, represents the parameters in g. Figure[[ssowss visually how our model works through toy example. We show the proof of our method (Equation 5) is universal approximator in Theorem 4.1 which motivated by Kidger (2022). Before the proof, we give two definitions to represent the space of continuous functions and nonlinear functions, respectively.
Definition 4.1 Let cont(X X Y) be the space of continuous functions X + (with respect to some topologies on X and Y).
Definition 4.2 Let nonl( X Y) be the space of nonlinear functions X -- Y.
4

---

Under review as a conference paper at ICLR 2023
Theorem 4.1 (ONODEs are universal approximators) Let d N. For g nonl - >R and f E cont(R x R + )d) which has a unique solution, let øg. :R + R2 denote the map X - X with h(t), (t) and h(O) = ( 0g), for t E [0, 1]. Take C(RR) to be the family of real functions that one wishes to approximate R and spant gg.5 then (ran)(9.5) is dense in C(R2).
The proof theorem is shown in APPENDIX A.I
4.2 ONODES HAVE A HIGHER COMPUTATIONAL EFFICIENCY
Besides our proposed ONODES, there exists another augmented method that makes NODE universal approximator, i.e., ANODES (Dupont et al. 2019). The ANODES add extra dimensions
to the NODES for the input, thus changing the input from h(0) E IRd to hh()) RRd+
bypasses the trajectory crossing problem by adding extra dimensions of Ra However, this does not essentially change the problem that node trajectories cannot cross. Even in a high-dimensional
space, trajectories still cannot cross at the same time. The high-dimensional space, however, causes a rise in degrees of freedom from d to d + a, leading to an increase in the parameter space and theoretical time complexity of estimating the vector field.
Our approach, on the other hand, allows the trajectory to "jump" between adjacent time points by adding the oscillator g. The degree of its oscillation is controlled by the parameters 0g in g. Thus, when given an input and an output, we do not have to optimize the vector field in the original space X Rdd because the vector field in X can be extremely complex. We can optimize the vector field in another space of the same dimension, i.e., X' E Rd, in which the vector field y may be simple and easy to be optimized. The trajectories based on this vector field V will also become simple.
To further illustrate the time efficiency our method, we consider its training process. For NODE, its forward evaluation and backpropagation can be calculated by using black-box differential equa- tion solver, called ODE solvers (Kutta, [901] Hairer 1987 Runge 1895p. The Adjoint Sensitivity Method (Pontryagin et al. 1961) is used for the calculation of backpropagation. It computes gra- dients by solving a second, augmented ODE backward in time, and it can be applied to all ODE solvers (Chen et al. 2018).
For any ODE solvers, both fixed step size and adaptive step size solvers are often reasonable choices for NODEs. Given final time T, a fixed step size solver will choose the time t; from [0, [] where i. An adaptive step solver, such as
At = fi+1 14- At is fixed in advance and independent Runge-Kutta Method (Runge 1895) is relatively modern solver, which vary the size of the next step so that the error made during the solver is approximately equal to some tolerance.
Typically in practice, a commonly used ODE solver is adaptive step solver. Although it is possible to estimate the absolute time theoretically for an adaptive-step-ssolee--bbe based NODE, we can estimate the computational efficiency by the time complexity as well the NFEs. Thus, we present proposition
Proposition 4.1 With the same vector field modeling structure, ONODE has a higher computational efficiency than ANODE.
Before the proof of the proposition 4.1 first we present lemma 4.1 and its proof.
lemma 4.1 With the same vector field modeling structure, ONODE has less time complexity as long as N₁ < AxNr number of neurons in the hidden layer of f. A is the augmented dimensions, and N, is the input dimensions.
Nr + N.+A where N₁, is the number of neurons the hidden layer of g. N₁, is the
2
The proof of lemma 4.1 is shown in APPENDIX
Since the adaptive step solvers such as Runge- Kutta are used in both ANODEs and ONODEs, we cannot obtain Proposition 4.1 immediately from lemma 4.1 However, since the NFEs are equal to the number of iterations of f in one training iteration and we experimentally show that an ONODE requires much fewer NFEs than ANODE, thus we can obtain Proposition 4.1
5

---

Under review as a conference paper at ICLR 2023
It is worth noting that method can still be augmented by adding additional dimensions. But the nature of our method differs from ANODE that the trajectories of our method are crossable the original input space X' E R.. However, in the extra space R", its trajectories remain uncrossable, which is inherited from an ANODE. Formally, an ONODE with the extra dimensions will be:
h(4) = [(xx,,0)) 0n) a(1) a(0))) W(O) - s( hxx, fg) a(1) ,,,0), h(0) = [x 0g)
(6)
where a() represents the extra dimensions for the original input. The function g represents the oscillator with parameters 0g. The augmented dimension can seen as an extra hyperparameter to tune. When the extra dimension is set 0, it becomes the ONODEs in Equation For more intuitive view, we show how our method works with a toy example and compares with the NODE and the ANODE in Figure
Data
NODE
Ours
ANODE
Ours
Extra dim =1
Extra dim
Figure 3: From left to right: A toy dataset, the NODE, our method, the ANODE with one extra dimension, our method with one extra dimension.
We show that the ONODE with extra dimensions also a universal approximator Theorem 4.2
Theorem 4.2 (ONODEs with extra dimensions are universal approximators) Let a, E N. For E nonl(Rd + IR"), E nonl(Rd + Rd+a), and f E cout x Rdta - Rdta) which has
unique solution, let &g..t : Rd + Rd+t denote the map X + X' with h(t) = [gxx,, 0g))
a(t)
((t)) = 5( -s((biar) [h(xxx000)) 0g)) 1, 1,, M() = ( 0g)
0g) for t E [0, 1]. Take C(Rd+a) to be the family
a(t) a(t)
0
of real functions that one wishes to approximate in Rd+a and set Ea = span(0g.d.) }, then Es is dense in C(Rd+9).
The proof theorem is shown in APPENDIX ??.
It is important note that the extra dimensions change the input space which, depending the application, may not be desirable. We experimentally demonstrate that with or without adding extra dimensions, our model is always better than the original NODEs and ANODEs. In many cases, our model without extra dimensions is better than the ANODEs which have extra dimensions.
5 EXPERIMENTS
We will demonstrate the superiority of our model in terms of accuracy, the number of function eval- uations (NFEs), and convergence speed. In Section 5.1 we introduce the datasets and environment settings. In Section show two toy datasets and three rife-life image datasets that model has better prediction accuracy, fewer NFEs, and faster convergence speed compared to baselines. In Section 5.3 apply model to the extrapolating time series tasks. We also illustrate that the model has less prediction error and a higher computational efficiency compared to baselines.
5.1 ENVIRONMENT SETUP
Datasets. We evaluated our model with two toy datasets and three image datasets in the classification task. We evaluated model using five mathematical functions in the extrapolating time series

---

Under review as a conference paper at ICLR 2023
task. Demonstrations of the toy datasets are shown in Figure 2 and Figure B respectively. The first toy dataset consists of four concentric circles, each consisting of 1, 000 randomly generated data points. The second toy dataset consists of nine equally spaced data stacks, each consisting of 1, 000 randomly generated points based on a Gaussian distribution. For the image classification task, we evaluate our model the MNIST (Deng, 2012), CIFAR-10, and CIFAR-100 datasets (Krizhevsky iet al., 2009). MNIST is a handwritten digit database with training set of 60, 000 examples. The CIFAR-IO training dataset consists of 60, 000 32 32 color images in ten classes, and CIFAR-100 has 100 classes containing 600 images each.
Evaluation Metries and Baselines. For the classification task, we compared our model with NODEs and ANODEs in terms of accuracy. NFEs, and convergence speed. For the extrapolating time series problem, our baselines include SONODE (Norcliffe al. 2020) in addition to NODE and the ANODE. We compare the training and testing mean squared error (MSE), the absolute training time, and the number of parameters. We use the number parentheses to denote the extra dimension, e.g., ANODE (1) denotes ANODE with the extra dimension of one.
5.2
RESULTS ON CLASSIFICATION
Datasets
Loss
NFEs
NFEs Loss
0.6
M202
40
Ous/0)
Ourel)
0.4
-30
0.4
Oure
ANCOE
0.2
0.2
10
0.0
5
10
15
20
25
20
25
30
35
40
45
0.4
Loss
10
0.0
0
4
20
30
40
50
Epochs
Epochs
NFES
Figure 4: From left to right are: two toy datasets, loss, NFEs, and NFEs vs loss over five runs.
Our model performs better than NODEs and ANODEs. We first evaluated our model on two toy datasets. We model the vector field with a multilayer perceptron containing three hidden layers, each containing 16 neurons, and the activation function is ReLU. For the oscillator, with a multilayer perceptron containing one hidden layer with 16 neurons. For our models, we ONODE with and without extra dimension, respectively. Our models show completely different flow trajectories and feature evolution with NODES and ANODES, see Figure Zland Figure 3 Experiments show that our models have better accuracy and converge faster than NODEs and ANODEs, regardless of whether they have extra dimensions. For NFEs, our models are also much lower than NODEs and slightly lower than ANODEs. The results are shown Figure The rightmost plot indicates the relationship between NFEs and loss.
model it
Then, we evaluated our model on the image classification task. We refer to the NODEs' modeling method for vector fields, using three convolutional layers, with the number of filters set at 128. The activation function ReLU. Similarly, our oscillator modeling also implemented by one convolutional layer. We tested our model on three image datasets, each running for 20 epochs. We recorded the validation error and average NFEs, and the results are shown in Table I The results show that our model is the best in both validation error and average NFEs.
Our model has higher computational efficiency. Since our methods only need to learn simpler vector fields and flow trajectories, they require fewer iterations to converge than NODEs and AN- ODEs. To test this, we measure the NFEs in both the forward evaluation and backpropagation process, then obtain the total NFEs for each model. Take the CIFAR- 10 for example. As in Figure 5 the NFEs required by our models increase slower during both the forward evaluation and backpropagation process while it increases faster for ANODEs and NODEs. This phenomenon
7

---

Under review as a conference paper at ICLR 2023
Table 1: Validation errors for 20 epochs over three various image datasets
Validation Error
Average NFEs
NODE
ANODE
Ours
NODE
ANODE
Ours
0.020 0.001
240.01 =
166.97 +
CIFAR-10 10
0.456 + 0.002
0.435 = 0,008
0.410
0.003
65.48
1.92
62.46 = 1.98
59.80 + 1,93
CIFAR-100
0.641 +: 0.003
0.620 + 0.002
0.618 0.005
82.96 2.13
75.37 : 0.59
67.54 +: 3.34
is particularly evident for the backpropagation process, where our method requires roughly 2 of the NFEs of ANODEs and roughly of the NODEs. We obtain similar results for other image datasets such as MNIST and CIFAR-100.
3
Forward Evaluation
Backpropagation
Total
4.0
4.0
LOSs
Loss
LOss
2.0
2.0
2.0
ANOOE
0.0
0.0
20
22
24
26
28
30
20
40
60
70
80
40
50
60
70 80
90
100
110
NFEs
NFEs
NFEs
30
100
NFES
NFEs
60
NFEs
80
40
ANODE
10 Epochs
15
5
10 Epochs
0
15
Epochs
Figure 5: Based on the CIFAR-10, the three plots in the first row show the relationship between NFEs and loss in the forward evaluation, backpropagation, and total training process, respectively. The three plots in the second row show the trend of NFEs as epochs increase.
5.3 RESULTS ON EXTRAPOLATING TIME SERIES
Real Learnt
NODE
ANODE(1)
ANODE(3)
SONODE
Ours
y +1
y 13 +
= In(t +1)
y (tn sin(t)
y t+1
Figure 6: Fitting and extrapolation results of different models for five time series functions.
8

---

Under review as a conference paper at ICLR 2023
We investigate the ability of our model to fit and extrapolate time series. We used five functions, as shown in Figure 6 Each function has 50 uniformly sampled points in to 10 seconds as the training set, followed by 10 points 3 seconds as the pre- dicted values. Our model is set to have extra dimension. The baselines were chosen as NODE, ANODE with one extra dimension, ANODE with three extra dimensions, and SON- ODE. Each model was trained for 500 iterations in the same environment. First, we compared the absolute training time (Figure 7 for each model. We found that our model has faster training speed with SONODE. Then, we compared the training MSE, test MSE, and the number of parameters, as shown in Figure S For visual comparison, we have ranked the performance of the models according to MSE from small- est to largest. Our model has the smallest value in most of the training MSE and all the test MSE. We also found that the extra dimension increases, there is no guarantee that ANODE becomes better. Our model has about the same number of pa- rameters as ANODE, but has much smaller than the number of parameters of SONODE.
Absolute Time (5)
slow
NO0E
B8:30 an:a0
AMODE0
14.44
30.30
00
05.03
38.06
SOHODE
11:38
30.30
90
3021
Clors
Funcl
Func2
Func3
Furcll
Funcs
Figure 7:
The training absolute time for each model in 500 itera- tions under the exact same setting.
Training MSE
Test MSE
Parameters
*
1000
NODE
2
ANODE(1)
600
3
ANODE(3)
SONODE
4
200
Ours
2
1
2
5
Functions
Functions
The middle two plots show the five models ranked in order of MSE from smallest to largest. The rightmost plot shows the total number of parameters for each model.
Figure 8:
CONCLUSION
We propose the ONODE, which is oscillator-based continuous deep learning model. Due to the presence of an oscillator, the trajectories of our model can "jump", making it to be a universal approximator. We also demonstrate that the ONODE computationally efficient. We validate model through various experiments including classification tasks and extrapolation on time series. Compared to the baselines, our model achieves the best results on variety of metrics.
REFERENCES
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Michael A Cohen and Stephen Grossberg. Absolute stability of global pattern formation and par- allel memory storage by competitive neural networks. IEEE transactions on systems, man, and 1983.
cybernetics,
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprini arXiv:2003.0460 2020.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14. 2012.
Emilien Dupont, Amaud Doucet, and Yee whye Teh. Augmented neural odes. Advances Neural Information Processing Systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
E Hairer. Sp nOrsett and wanner. Solving ordinary differential equations I, 8, 1987.
Ernst Hairer, Syvert P Nørsett, and Gerhard Wanner. Solving ordinary differential equations. I, Nonstiff problems. Springer-Vlg, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2)25512257, 1991.
Kurt Horiik Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks uni- versal approximators. Neural networks, 2(5):359-366, 1989.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433. 2022.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Wilhelm Kutta. Beitrag zur naherungsweisen integration totaler differentialgleichungen. Z. Math. Phys., 46:435-453, 1901.
Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken. Multilayer feedforward net- works with nonpolln activation function approximate any function. Neural networks, 6(6):861-867, 1993.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. Advances in Neural Information Processing Systems, 33:3952-3966 -3633, 2020.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough diffcrential equa- tions for long time series. In International Conference on Machine Learning, pp. 7889-78333 PMLR, 2021.
Alexander Norcliffe, Cristian Bodnar, Ben Day, Nikola Simidjievski, and Pietro Liò. On second order behaviour in augmented neural odes. Advances Neural Information Processing Systems, 13:5911-59221 2020.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8:143- 195, 1999.
LS Pontryagin, VG Boltyanskii, RV Gamkrelidze, and EF Mishchenko. Mathematical theory of optimal processes [in russian), 1961.
Rico-Martinez, JS Anderson, and IG Kevrekidis. Continuous nonlinear signal processing: a neural network based approach for gray box identification. In Proceedings of IEEE Workshop Neural Networks for Signal Processing, pp. 596-605. IEFF, 1994.
Ramiro Rico-Martinez and Ioannis G Kevrekidis. Continuous time modeling of nonlinear systems: A neural network-basee approach. In IEEE International Conference on Neural Networks, pp- 1525- IEEF, 1993.
Ramiro Rict-Martinez K Krischer, IG Kevrekidis, MC Kube, and JL Hudson. Discrete-vs. continuous-time time nonlinear signal processing of cu electrodissolltiion data. Chemical Engineering Communications, 118(1):25-48, 1992.
10

---

Under review as a conference paper at ICLR 2023
Carl Runge. Über die numerische auflösung von differentilgllicchunen.. Mathematische Annalen, 46(2):167-178, 1895.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. In International Conference on Machine Learning, pp. 11086- 11095. PMLR, 2020.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 THE PROOF OF THEOREM 4.1
Proof: Suppose the input R the system ODEs will solve a series of equations yo. VI..... .... yM follows: (1) (0) = (xx) Rd and a (t) - 0, (2) (0) - 0 RRd and dks
(3) (0) = 0 E ge and dbs (t) = h (t) (a(t), (4) (3(0) 0 e Rd' and dhg di ho()))ho(l) de
dt (t) = (t) 2 @ho(t).
.M+1) h0) = E gdd and dh (t) = har-((t) @ ho()). The solution will be written oi
immediately: (tt = g(x), (2) (t) = So ((())) tg(x)) 02 (3) ha(t) = = x g(x)ds { (4) ha(l) So g())))) @ g(x)ds = _ (=))(
(M+1) 1l(()) - To ((g())(M++))) @ g(r)ds = g(=))() Set and we will obtain the collection in g(x) E Rd up to degree M +11, i.e. (g(r)).(>)) }g(() {g()) ..... ()) ) The function g(z) is realized by single hidden layer perception utilizing Relu active function. Assume is a non-negative finite measure on IR with compact support, and continuous with respect to the Lebesgue measure. Then Eg is dense in ["))), S p oc, if and only if, the non- linearity in is not polynomial, where L'(μ) is the set of all measurable functions f such that
f [|]) = (fRz (f(=))"du(zz)) < oo (Leshno et al. 1993 Hornik et al. 1989 Hornik 1991
Pinkus 1999). Thus, span (dg.f) is dense in C(R²) and it is universal approximator.
A.2
THE PROOF OF LEMMA 4.1
Proof: For a fair comparison between ONODE and ANODE, we assume that the input and output dimensions are the same, i.c., N; No. where the No represents the output dimensions. For the vector field modeling structure, we assume there exist two hidden layers with each hidden layer having N₂, neurons. For convenience, we presume that there is only one sample. For forward evalu- ation, an ANODE will have the time complexity of O Amode = O( + 11+ A)) + N,, + ON,, (N; + A)) + ((V, + A)) = N2, + 2AN, + No. + NN + N? AN,). Our proposed ONODE will have the time complexity of Onoade = O(N,N)) + O(N,,,) + O((N, + 1)) + Nk) + H,) = O(2N, + 22 + N₂ + 2N,N, Since the backpropagation has the same complexity as the forward evaluation, the total time complexities in one iteration for ANODE
and ONODE are Anode 20 Anode and Onoaee 20 Omode respectively. We can immediately N.
get the solution that Onnode < Anode as long as No < AxNi
N. 2
A.3 THE PROOF OF LEMMA 4.2
Proof: The proof is similar to that of proof A. Since the extra dimensions R will be added into the ONODE through nonlinear function will not change the result in proof A.I Thus, obtain the result that span ( ( } is dense in C(R++a).
12


Reviews:
[{"review_id": "znPem589od", "reviewer": "Reviewer_a9dv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is proposing a modified neural ordinary differential equation (NODE) that augments the function that describes the vector field with an additional term that introduces oscillations into the dynamics. The authors show that this modification can overcome limitations of standard NODEs by (i) proofing that the new model is a universal approximator, and (ii) by the means of experiments on classification and time series extrapolation tasks.", "summary_of_the_review": "In summary, the paper presents some neat results, but I don’t feel that these results are significant enough to accept this paper. That is, because the modification is incremental and the experiments do not show that the proposed model leads to any substantial improvement compared to ANODE (e.g., improving the test error for CIFAR-100 from 0.62 to 0.618 is uninteresting). In the current form the paper is a good workshop paper.", "strength_and_weaknesses": "This paper proposes a very simple modification that leads to a significant effect, i.e., the resulting model is a universal approximator. However, this result is not very surprising since the term that is introduced can be seen as some form of a gating function if a sigmoid is wrapped around the function $g$. Further, the results presented are interesting but limited. Here are more detailed comments that I would like to see addressed:   \n\nThe way you first introduce $g$ is a bit mysterious. First, it would be nice to have an extended discussion on oscillation ODEs (I am not very familiar with these types of systems, and I assume most readers are neither). Where do such ODEs appear in the real-world (I assume there are many interesting examples), and what are their drawbacks and limitations? I assume that ODEs that have highly oscillatory solutions have stability issues and/or cannot be solved efficiently using standard numerical integrators. So why are these models good for motivating a new neural network architecture (is that because jumps are introduced)? Next, a good discussion of how to design / model $g$ is missing. Later in the text, after you prove your theorem, you briefly state that `the function g(x) is realized by a single hidden layer perception utilizing a Relu active function.’ Why do you use Relu here? It seems to be a bit of an unnatural choice, because you don’t want that the activations of $g$ can grow too much, or am I missing something? I would choose a tanh or sigmoid function around $g$ so the activations are bounded or does $g$ need to be some unbounded function in order that your theorem holds (I don’t think so)? \n\nThe experiments are weak for two reasons: \n* Only ANODE is used for comparison in 5.2. I miss a more detailed discussion and the comparison with coupled NODEs / second-order ODEs. Will the proposed model perform better or like these more general formulations on the tasks considered? Moreover, a discussion and comparison to Neural Dealy Differential Equations [1] is missing.  \n\n* The considered set of experiments is limited. It is good to use some toy tasks to demonstrate the advantage and to build some intuition. But I would also like to see how the model performs on more interesting (challenging) real-world tasks.   \n\n* Moreover, the tables need to list the parameter counts of the different models. \n\n* Which ranges for the tuning parameters did you explore, and did you use a validation set for tuning in addition to the final test set? \n\n \n[1] Neural Delay Differential Equations. ICLR 2021. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qLrgcJ6g7k", "reviewer": "Reviewer_p8TZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A universal approximator is proposed in this paper using an oscillator NODE (ONODE) whose trajectories can \"jump\". It was shown that the ONODE is computationally efficient. A number of experiments (including classification tasks and extrapolation on time series) were conducted to validate the model.", "summary_of_the_review": "Overall, I think this paper need more theoretical support. Some claims are not well-supported and need to be proven. ", "strength_and_weaknesses": "**Strength**\n\nThe main idea of the paper is interesting and important.\n\n**Weaknesses**\n\n1) The function  $g: \\mathcal{X} \\longrightarrow  \\mathcal{X}^{\\prime}$ defined in eq.(5) is not clear enough. $\\mathcal{X}$  and $ \\mathcal{X}^{\\prime}$ need to be defined as well.\n\n2) The authors have mentioned that their approach allows the trajectory to “jump” between adjacent time points by adding the oscillator $g$. However, it is unclear how this can be achieved neither through their explanation nor eq. (5). \n\n3) In Theorem 4.1/ Theorem 4.2, it is mentioned: \"$f \\in cont(\\mathbb{R} \\times \\mathbb{R}^d → \\mathbb{R}^d)$/$f \\in cont(\\mathbb{R} \\times \\mathbb{R}^{d+a} → \\mathbb{R}^{d+a})$ which has a unique solution\". What do you mean by the *unique solution* of $f$ in both theorems?\n\n4) By $C(\\mathbb{R}^d)$, do you mean the set of all the real-valued **continuous** maps on $\\mathbb{R}^d$? \n\n5) In the proof of Theorem 4.1, it is proven that $\\sum_d$ is dense in $L^p(\\mu)$. But, how does this imply that $\\sum_d$ is also dense in $C(\\mathbb{R}^d)$ (especially since $\\mathbb{R}^d$ is not compact)?\n\n6) In the proof of Theorem 4.1, it is mentiond: \"Assume $\\mu$ is a non-negative finite measure on $\\mathbb{R}^d$ with compact support, and continuous with respect to the Lebesgue measure\".  What are sufficient and necessary conditions for having such a measure?\n\n7) On page 5, section 4.2, it is written: \" [...] the vector field in the original space $X \\in \\mathbb{R}^d$, [...]\". What do you mean by the \n**space**  $X \\in \\mathbb{R}^d$? Likewise, what does $X^{\\prime} \\in \\mathbb{R}^d$ mean?\n\n8) On page 4, section 4.1, by $\\phi_T$ do you mean the flow of an ODE? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XtEqGcdU_H", "reviewer": "Reviewer_nasm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper suggests adding jumping terms to the training of Neural ODEs to improve training and computational time.\nThe conduct an analysis of the proposed model and several experimental studies.", "summary_of_the_review": "The paper identifies problems in current Neural ODEs but fails to discuss relevant prior work and to fully evaluate their method.", "strength_and_weaknesses": "Strengths:\n\n1. The paper identifies three main challenges in current ODEs and proposes a method to alleviate them.\n\n2. Empirically the method seems to work.\n\nWeaknesses:\n\n1. The figures quality and clarity need to be improved. They do not look professional and are hard to understand.\n\n2. One of the problems that the authors state that exists in current Neural ODEs is \"The reason\nfor the second limitation is caused by the straightforward optimization approach of the vector field.\nThere is no guarantee that learning the vector field is a better choice than learning the estimated functions\ndirectly. Sometimes it will need many function evaluations to optimize the vector field, so the\ndifficulties go beyond learning the estimated functions themselves\" in page 1. However no citations or self explanation is given in the paper.\n\n3. The authors are lacking reference and discussion of prior work, for example \"Stable Architectures for Deep Neural Networks\".\n\n4.  I am not sure why the authors call the method 'oscillatory'. Typically this is a term that is reserved to describe hyperbolic or wave like dynamical systems. However here the authors propose to introduce 'jumps' into the trajectories.\n\n5. The experimental scope is rather narrow and it is hard to draw conclusions from experiments on small datasets like CIFAR-10 and CIFAR-100. I believe that experiments with larger and standard datasets like ImageNet can be more convincing. Also, the authors should compare their obtained accuracy with other CNNs that are not necessarily ODE based, for a fair comparison.\n\n6. A question to the authors: Why do you report the *validation* accuracy on CIFAR-10 and CIFAR-100 ? the norm is to report the test accuracy.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject