Analysis of generalization capacities of Neural Ordinary Differential Equations
Authors: Madhusudan Verma
Manoj Kumar
Venue: Accepted by TMLR ()

Abstract: Neural ordinary differential equations (neural ODEs) represent a widely used class of deep learning models characterized by continuous depth. Understanding the generalization error bound is important to evaluate how well a model is expected to perform on new, unseen data. Earlier works in this direction involved considering the linear case on the dynamics function (a function that models the evolution of state variables) of Neural ODE Marion (2023). Other related work is on bound for Neural Controlled ODE Bleistein & Guilloux (2023) that depends on the sampling gap. We consider a class of neural ordinary differential equations (ODEs) with a general nonlinear function for time-dependent and time-independent cases which is Lipschitz with respect to state variables. We observed that the solution of the neural ODEs would be of bounded variations if we assume that the dynamics function of Neural ODEs is Lipschitz continuous with respect to the hidden state. We derive a generalization bound for the time-dependent and time-independent Neural ODEs. We showed the effect of overparameterization and domain bound in the generalization error bound. This is the first time, the generalization bound for the  Neural ODE with a general non-linear function has been found.

Published in Transactions on Machine Learning Research (10/2025)
Analysis of generalization capacities of Neural Ordinary Dif- ferential Equations
Madhusudan Verma*
2da23m01559ttmz..m
School of Engineering and Science
Indian Institute of Technology Madras, Zanzibar Campus
Manoj Kumar"
manoj
School Engineering and Science
Indian Institute of Technology Modras, Zanzibar Campus
Reviewed on OpenReview: httee///ppaaeeeeeeeee/ooouuuuucs
Abstract
Neural ordinary differential equations (neural ODEs) represent widely-used class of deep learning models characterized by continuous depth. Understanding the generalization crror bound is important to evaluate how well model is expected to perform on new, unseen data. Earlier works in this direction involved considering the linear case on the dynamics function (a function that models the evolution of state variables) of Neural ODE Marion (002)).
Other related work is on bound for Neural Controlled ODE Bleistein Guilloux (0223 that depends on the sampling gap. We consider a class of neural ordinary differential equations (ODEs) with a general nonlinear function for time-dependent and time inme-indeeendent cases which is Lipschitz with respect to state variables. We observed that the solution of the neural ODEs would be of bounded variations if we assume that the dynamics function of Neural ODEs is Lipschitz continuous with respect to the hidden state. We derive a gencralization bound for the time-dependent and time-independent Neural ODEs. We showed the effect of overparameterization and domain bound the generalization error bound. This the first time, the gencralization bound for the Neural ODE with a general non-lincar function has been found.
1 INTRODUCTION
Neural Ordinary Differential Equations (Chen et al. (2018) are class of deep learning models where the transformation between layers is treated as continuous process defined by an ordinary differential equation (ODE). This idea generalizes the concept of residual networks (ResNets), where the evolution of the hidden state ztt) over time modeled by a differential equation
dz(t) ((2(1),,0(1) with =(0) = x,
(1.1)
di
where ((t) represents the parameters of the model.
Unlike discrete representations from the conventional methods, neural ordinary differential equations (Neural ODFs) directly learn continuous latent representations (or latent states) based on a vector field parameterized by a neural network. Kidger et al. (2020) introduced neural controlled differential equations (Neural CDEs), which are continuous- time analogs ResNets that use controlled paths to represent irregular time series. Neural ODEs are also extended to neural stochastic differential equations (Neural SDEs) with a focus on
'Both the authors contributed equally for this work.
Website httpe://site. sttp://sites.gonge wow.lee.oogge..omm.. haamm//iwww/aaaa hhasstt Code is open-source h //ggtt/ttttttttt http://tttuuu..c.o.aaaaeeeeeeeee httph//aahuuddnnnnaammma Verma/Gen- bound- Node

---

Published in Transactions on Machine Learning Research (10/2025)
aspects such gradient computation, variational inference for latent spaces, and uncertainty quantification. In neural stochastic ODEs (neural SDEs, Oh et al. (2024)) )) usually a diffusion term is incorporated but a careful design of drift and diffusion term is essential.
With neural ODEs, generally, it is difficult to handle irregular time-serics data. Neural controlled differential equations Kidger et al. (2020)) generalize neural ODEs by incorporating a control mechanism, allowing them to model the evolution of hidden states as controlled differential equations. Studying the statistical properties of neural ODEs is not trivial task. Since standard measures of statistical complexity in neural networks, such as those discussed by Bartlett et al. (2019), typically increase with depth, it is unclear why models with effectively infinite depth, like neural ODEs, would demonstrate strong generalization capabilities.
Marion 2023 studied the statistical properties of a class of time-dependent neural ODEs described by the following equation:
dH₃ dt (H),H,),
where W, E Rddd is a weight matrix that depends on the time index t, and o : R + is an activation function applied component-wis wise. The model considered by Marion (2023) does not include the case where there are weights inside the non-linearity since they assume the dynamics at time be linear with respect to the parameters.
Contribution. For general class of well-posed neural ODEs, where the neural network involves the non- linear weights within the function, there is no result related to the generalization bound. In this work, we consider a Neural ODE model parameterized by 0(t) of the following form:
dz(t) = =()) with z(0) I,
(1.2)
dt
where f Rd x R x RT Rd 0. ddd and 0 [] + R Time dependent 0 in neural ODEs is introduced by Massaroli et al. (2020b). We provide a generalization bound for the large class of parameterized ODEs instead of linear class, the bound we provided here will hold for linear class as well and is stricter than the earlier bounds for the linear class of functions. To the best of our knowledge, this is the first available bound for neural ODEs for this class of functions.
Organization. Section ! is devoted to the introduction, and in Section 2 we discuss the related works. In section we formulated the learning problem. In this section also discuss some of the preliminarios and definitions that are crucial for understanding the problem setup. Section 4 is devoted to derive results related to generalization bounds. We performed numerical experiments in section 5 In the end, some concluding
remarks are given in section 6
2 RELATED WORKS
Hybridizing deep lcarning and differential equations. The fusion of deep learning with differential equations has recently garnered renewed interest, although the concept has been explored since the 1990s Rico-Martinez et al. (1992) 1994). A notable advancement was presented by Chen et al. (2018). where they introduced model that learns representation u E R2 by setting the initial condition =(0) (eect) (u) for the following ordinary differential equation (ODE):
dz(t) f(x(t),t,
dt
where both and do(i)) are neural networks. The solution at the final time t1, denoted a(t1), is then utilized as input to conventional machine learning model. This approach scamlessly intogrates neural networks and ODEs, offering robust framework for learning complex dynamical systems. Since then, several works have built on this idea, including theoretical advancements and practical applications as seen in Dupont et al.) (2019h; Chen et al. (2019): Finlay et al. (2020). For a more comprehensive overview, readers may refer the reviews by Massaroli al. (2020a) and Kidger (2022), which delve into the intersection of differential equations and deep learning.
2

---

Published in Transactions on Machine Learning Research (10/2025)
Generalization of Neural Controlled Differential Equations. Bleistein & Guilloux (0023) used Lipschitz-based argument to obtain sampling-dependant generalization bound for neural controlled differ- ential equations (NCDEs). The NCDE considered was of the following form:
dæ: (2.)
where Et E IR, and G : RR - IRaxd is neural network parameurized by th, also Fy RP continuous path. In this work, it assumed that (xz) Lipschitz which implics that x x= (2t))e(0.1] of bounded variation. They also analyzed how approximation and generalization are affected by irregular sampling.
Generalization bounds for neural networks. Bartlett et al. (017a) derived margin- based multiclass generalization bound for neural networks that scales with margin-normalizet spectral complexity, involving the Lipschitz constant (the product of the spectral norms of the weight matrices) and a correction factor. Long & Sedghi (2019) established generalization error bounds for convolutional networks based on training loss, parameter count, the Lipschitz constant of the loss, and the distance between current and initial weights, independent of input size and hidden layer dimensions. Experiments on CIFAR-10 10 show these bounds align with observed generalization gaps under varying hyperparameters in deep convolutional networks. [aang & Ma [2022] derive generalization error bounds for deep neural networks trained via SGD by combining control of parameter norms with Rademacher complexity estimates. These bounds, which apply to varions architec- tures like MLPs and CNNs, depend on the training loss and do not require L-smoot hnees, making them more broadly applicable than stability-based bounds. Understanding generalization in neural networks has been approached through various theoretical frameworks, including VC-dimension (Vapnik Chervonenkis, 1971) Vapnik & Chervonenkis (1971), Rademacher complexity (Bartlett Mendelson, 2002) Bartlett & Mendelson 22002), PAC-Bayes theory (McAllester, 1999) McAllester (199),, compression-baser bounds (Arora et al.,
2018) Arora et al. (2018), stability analysis (Bousquet & Elisseefff 2002; Hardt et al., 2016) Bousquet & E Cisseelf (2002); Hardt et al. (2016). and information-theereele approaches (Xu & Raginsky, 2017; Tishby & Zaslavsky, 2015) Xu & Raginsky (2017); Tishby & Zaslavsky (2015). While VC-dimension and covering
number bounds are often loose for deep networks, norm-based and margin-based bounds (Bartlett et al., 2017; Neyshabur al., 2015) [Brtllttt et al. (2017c); Neyshabur et al. (2015) offer sharper estimates. PAC-
Bayes bounds have been effectively applied deep models by optimizing the bound directly (Dziugaite
Roy, 2017) Dziugaite & Roy (2017).
3 BACKGROUND AND LEARNING PROBLEM
Let hx(x) be the solution of Neural ODE 1.1 at the final time t = L (L can also be taken 1 for simplicity) with initial condition and let Y1 be the true label of the differential equation at the final time step. Also, assume that He be the set of all predictors he with different initial conditions. We consider an i..d. sample r;); ~ (), For a given predictor hg, the empirical risk over the training data is:
1 Σ ho(s)).
Rd (mo)))
= n
The expected risk or generalization error over the data distribution is:
R(he) E(r.y)- hg(x))) ([](y,ho))
where is the unknown, true probability distribution the data. R(ha) cannot be optimized, since we do not have access to the continuous data. Let i E arg ming R (ho)) be an optimal parameter and h be the optimal predictor obtained by empirical risk minimization.
order to obtain generalization bounds, the following assumptions on the loss and the outcome are necessary Mohri (0018).
For imput ztt) Rd, with weights At(t) Baxd and biases b.(t) R for 1, N, we assume that the feed forward neural network with N layers and number of hidden units takes the following form.
(((()),,00)) a(AN(!) (AN- (... (Ai(1)= + b.)) bx-1) 1) bN).
(3.3)
3

---

Published in Transactions on Machine Learning Research (10/2025)
is the activation function, 8(t) [A1, A2; AN. bi, b2, bg...by] where IR. We have the following assumptions:
Assumption 1. (()()),,0(t) is assumed to be Lipschitz continuous with respect to z(t), that means
(z(=,,,,(()= - 1,,,, (())) S foco |21 =22/-
Assumption 2. Weights (t) and biases (t) of feed forward neural network defined in 3.3 are Lipschitz continuous.
Assumption 3. The oulcome y E Rd is bounded almost surely.
Assumption 4. The loss é R2 _² 14 is Lipschitz continuous with respect to its second variable, that is, there exists Le such that for all u' E and y Y
MM u) (y,,"') s Luju u'l.
Assumption is satisfied for most of the classical loss functions, such as the mean squared error, long the outcome and the predictions bounded. Table gives the details about the notations used in this work and Table 2 gives the details about the notations used in Marion (0023 and Bleistein & Guilloux (2023)
Symbol
Mcaning
x1) e Rd
Hidden state solution of the Neural ODE at time Input / 'iiitial condition, ie., z(0)
Rd
True outcome (bounded, Assumption 3)
((t) E R
Time-dependent parameters of the Neural ODE Dynamics function (vector field), assumed Lipschitz in Lipschitz constant for f(z(t),,,0(t w.r.t. z Final time horizon (ODE solved on LJ) Dimension of solution space
((=(1),,0(1)
L
ho(x)
Predictor: solution of the Neural ODE at final time t L Hypothesis class of predictors Expected risk (generalization error) Empirical risk on samples Loss function, assumed Lipschitz in prediction argument
Hø
R(ha)
(ho)
llv, D)
Me
Upper bound on the loss function Lipschitz constant of the loss function
Le
Activation function (c.g., ReLU, Tanh), Lo-Lipschitz Weight matrices and bias vectors of the feedforward network parameteriving f
A,(1), bi(1)
LA La
Lipschitz constants of weights and biases (time-dependent case)
V
Upper bound on the solution norm of Neural ODE
(HL)
Empiriaal Rademacher complexity of hypothesis class Covering number of hypothesis class H under metric p Lipschitz constant of the loss in its second argument. Confidence parameter in generalization bounds Data distribution
( H.
P
Rademacher random variables (+1 with equal probability)
Table 1: Table of Notations
3.1 Preliminaries
In this subsoction, we present essential preliminary results and definitions necessary for constructing the theoretical framework to derive the desired generalization bound. We begin with the definition of functions of bounded variation, the solutions of the ordinary differential equations (ODEs) under consideration
4

---

Published in Transactions on Machine Learning Research (10/2025)
Symbol
Mcaning
m
Number of basis functions (Marion, 2023)
Re
11,00 bound on parameter function 0 (Marion, 2023)
Ke
Lipschitz constant of paramcter function 0 (Marion, 2023)
Pe
Class of parameterized ODEs (Marion, 2023)
Depth of the neural network (number of layers) in Go. (Bleistein & Guilloux, 2023)
Width dimension of hidden layers in Go (Bleistein & Guilloux, 2023)
(+())
Dynamics function parameterized by & (Bleistein & Guilloux, 2023)
An, bn
Weight matrices and bias vectors at layer h (Bleistein & Guilloux, 2023)
U, Φ
Additional network parameters in NCDE bound (Bleistein Guilloux, 2023)
KP. KP
Discrotization and depth-dependent constants (Bleistein & Guilloux, 2023)
Me
Upper bound on [0)|(0))lop (Bleistein & Guilloux, 2023)
Table 2: Table of Notations (Prior Works:
Marion (023 Bleistein & Guilloux 2023
are observed to exhibit this property. Next, we review several formulations of Gronwall's lemma, critical tool for establishing bounds on ODE solutions. We then introduce the concept of covering numbers, which plays a pivotal role bounding the Rademacher complexity. Finally, we summarize key results related to Rademacher complexity, enabling the subsequent derivation of the generalization bound.
Definition 3.1 (Dutta & Nguyen (2018)). The function E L' (2) is a function of bounded variation on Q (denoted by (2, if the distributional derivative of is representable by finite Radon measure in Q, i.e., is
I. u 0p Dx
dx d==-1 - yDD for all & E (2, i (1, ... n}.
for some Radon measure Du = (Diu, Dzu..... DDu). We denote by Dul the total variation of the vector mensure Du, i.e.,
{ u(x) dio(d) da o (2,, 10l1 < $1
(
=su
Lemma 3.2. (Particular case of Gronwall's Inequality) Let. I denole an interval of the real line of the form [a, oo) or Ja, b] aa, b) with < b. Let a, B, and u be real-valued functions defined on 1. Assume that 3 and u are continuous and that the negative part of a is integrable every closed and bounded subinterval of I.
Assume that 3 is non-negative, function a is non-decreasing. If satisfies the integral inequaläty
m(t) a(1) [ B(s))(() ds, VET
then
u(t) $ a(t) exp [()) t I.
Lemma 3.3. (Gronwalll's Lemma for sequences). of real nembeers such that
Let (un)a2 (ba)a2o, and (fk)azo be positive sequences
yn S Sa
Σ br
for all n 2 0. Then
n2
Un S fn +
fubr
II
(1 (b+b,)
1=0
/=|+1
for all 2 0.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Proof can be found in Holte (2009) and Clark 1987
We need variant of Gronwall's Lemma for sequences.
Lemma 3.4. Let (wx) k20 a sequence such that for all 1,
ug S 1 + bb
for (4x)a21 and (bx)x21 two posilive sequences. Then for all k 2 1,
Wk
aj
ug
bj
04
Definition 3.5 (artlett al. (2017b)). Let (M, p) be a metric space, subset T M called an T-cover of T C M for every E T, there crists m' E T such that p(m,m') S T is called proper cover if T T. The covering nambber of T is the cardinalily of the smallest 7-cover of T, that is
N(r, T)), = min min([T] an t cover of T)
Lemma 3.6 (Gaulschi 1959
For O and 0 < < < 1, the inequality holds
+(x+1) T(x +)
S (a 11=1
Rademacher Complexity Rademacher complexity is a concept from statistical learning theory that measures the richness a class of functions in terms how well they can fit random noise. It commonly usod derive bounds on the gencralization crror of lcarning algorithms.
Definition 3.7 Moohii 2018)). Given class of functions H mapping from an inpul space X to R and sample $ = {21, .2...... InJ drowm from a distribution D, the empirical Rademacher complexity of with respect to the sample is defined as:
1 Σ nc((zz)
R. (H) Ey sup NEW
where Th are independent Rademacher variables, taking values or -1 with equal probability, and the expectation E, is taken over the distribution of the Rademacher variables.
Lemma 3.8 (Srebro & Sridharan). Let (F. (P())) denote the data data-dependent L2 metric space, given by the metric
pfff)= 1 (f(x.))((x))) n
where II are samples from space S, and Fa...... denotes the restriction of the function class F to that sample.
For any function class F containing functions f S - R and sample {a11, 22, .n,) drown from distribution D we have
sup VRI
log N(T, (z(PP))
Rn(
inf +20
4e + 12
dr
where N(rFF, (P))) denotes the covering rumber of F.

---

Published in Transactions on Machine Learning Research (10/2025)
Lemma 3.9 MMorri 2018
Rademacher complerity regression bounds
Let H (h: [
be a of functions from an input space to an output space that a learning algorithm con choose from during braining. Let
y + IR be a non-negative loss function, upper bounded by M > (f(y, w¹) S M for all y.y E )), and such that for any fized y' E V, the function y Ely. y/) is sipihcchtt for some > 0 (M(v,,y') 1(22. v')l I plyn v2l Vyn, )). Let 5 E (0,1), then with the probabiliby at least 1 5.
1 v.) + + (.MH))++3))))
log 3 2n
(xx [e(h(x),
7
4 MAIN RESULTS
We state and prove important lemmas before proceeding to the proof of the main Theorem 4.9 We assume that (=((t), A(t)) is Lipschitz continuous with respect to So, by the Mean Value Theorem, the solution to equation 1.1 will be of bounded variation. More details can be found in appendix A.1
Lemma 4.1. Consider the feed forward neural network fw(2(()) defined in 3.3 assume that Lo
Lipschitz, and A, 's are bounded by A and biased terms bounded by B. Let [A,() S BA, [bb; s Bo t E [0, L] and LA and La Lipschitz constant for weights and biases respectively.
Using equation AA.5 we have, |AA(00] < ||A(()]] + LAL S BAO + LAL A and [b6,(0) s (1,(0)) + L, S Bbo + TT = B. Then,
==(0||| < (I=CO0II + LL, B LOAA-11) exp (LL
TioA
Corollary 4.2. In the case of time-independent Neural ODE Lipschitz constonts for weights and biases will be 0, hence BA, = A and Bad = B and
| [Ia(0)]+ + LLg Bio
BLo -1 exp (LLsa). La L.BAo
(4.4)
This bound on the solution will be useful to obtain the explicit form for covering number bound. This bound inwolves the Lipschitz constant, bound on biased terms and weights.
Let Dz be the distributional derivative of solution function 2 and
I = {a E - L)) is non decreasing]
B {z L (f0, ((0, [Dz(((0, M).
Then we have the following lemma:
Lemma 4.3. Let
V = [12(00]] + LL (LOA--1) 1))
exp (LL
Lead
and 0 <T
LV 7, then
21
I(P,I, <
18
Remark 4.4. Observe that the covering number bound increases exponentially with domain size and the bound of solution. We obtain stricter bound on covering number for the class of bounded voriation functions in T space, earlier bound can be found in Lemma 2.3, Dutta & Nguyen (2018).

---

Published in Transactions on Machine Learning Research (10/2025)
Corollary 4.5. Let 0 < I LV then
2
L2r,
324
Proof. From Dutta & Nguyen (2018). we know that
((.. SN² (,,
which proves the required result.
Remark 4.6. In the above lemma, the bound is dependent on covering number of non-decreasing functions with the radius of balls getting half. But for the class of bounded variation functions, we do not assume that the functions are non non-decreasee non-decreasing.
Lemma 4.7. Let B be the class of Rd valued functions with domain [0, L] that are of bounded variation, then
RB' S 96 V bLV log2
LVd log
576
Remark 4.8. Lemma 4.7 ensures that the bound on Rademacher complezity increases with the dimension of range space for bounded varialion functions. Also, due the conslant V, we also get the dependence on weight parameters and Lipschitz constant of activation functions.
Theorem 4.9. (Generalization bound for Neural ODEs) Let V be the upper bound of the solution of neural ODE, d be the dimension of the solution, h be the optimal predictor and h" be the true solution and L be the upper bound for time and M > be an upper bound of non-negative loss function I: [0, VJ x [0, V + TR, i.e., 1(h, h²) I M for all h ht E V1 (assumption 3). Also, assume that for any fixed h E [0, V), the mapping y + ((h,ht) is L-Lipcchitz for some 0 (assumption 4). Then generalization error is bounded with probability at least - 8 by:
96 V
bLV log
LVd log
log 2
R(h) I A²(h) (h) + 2p
576
+ 3M
Vn
2n
Outline of the proof. We observed that the solution of the Neural ODEs described by equation I.1 will of bounded variations. We found stricter bound for covering number of this class of functions. We observed that the covering number related to the nunber of positive integer solutions of equation which is equal to central binomial coefficients. The central binomial coefficient obeys recurrence relation which has closed form solution. We then used inequality which the ratio of gammaa functions satisfies. In this way, obtained stricter bound for the covering number of bounded variation functions. We assumedd the parameters to be Lipschitz continuous and obtained a bound on weights and biases. We then found Rademnachrr complexity bound using Dudley's entropy integral stated Lemma 3.8 Finally, we used the result for the Rademacher complexity Lemma 4.7 regression bound stated in Lemma 3.9 Since Rademacher complexity non negative, b > 36LV log?
7
Remark 4.10. Our analysis first establishes stricter bound for increasing functions in the L class, and by leveraging the relation between covering numbers of bounded variation functions in L' and those of increasing focccinnuuniiooi functions in Lt, we obluin refined covering number bound for for bounded variation functions. Unlike
prior work that operates in parameter space, we conduct the analysis directly in solution space, which offers several advantages such as the resulting bounds depend on intrinsic trajectory properties (e.g., Lipschitz continuity) rather than parameter-nore norm constraints; moreover, solution solution-space space bounds remain valid across different parameterizations and architectures that realize the same dynamics.
8

---

Bounded variation solution with bound Published in Transactions on Machine Learning Research (10/2025)
4.1 Comparison with other bounds (Neural ODEs)
To highlight the novelty of our result, we compare it against the most closely related generalization bounds for Neural and parameterized ODEs. Table B summarizes the assumptions, asymptotic form of the bounds, and whether the dependence is solution-space or parameter-space. Unlike prior bounds that are inherently tied parameter-space quantities (e.g., norms, Lipschitz constants, network depth), our result depends solely solution-space properties. This shift broadens the applicability of the theory, since solution regularity can often be verified or controlled directly, even in models where parameter norms are hard to interpret constrain.
Work / Theorem
Assumptions
Generalization Bound
Theorem
4.9
(This work)
bLV d3/2 log2
576LV 2/2 log 2
n(h) nt" (t)
Bounded loss with bound M p- u-Lipschitz loss.
vn
+ 3M
1ox(2/4) 2m
Theorem 1 (Mar-
| < Re, each Ke- Ki-Lipschitz
ion (2023)
R(O.) R" ((.) + B
Emest 1) loug Remmn)
Ke
lok(1/6)
Theorem 4.1
q-layer MLP, Lipschitz activation, bounded weights and biases.
(Bleistein & Guil- loux (2023))
Ro(fo) R²())) = 24M
V20UP + (a 1)(( + 1)UP
vn
24M
4p(2 M)
Table 3: Comparison of generalization bounds across different works.
Constant
Definition / Meaning
Marion (2023)
Re
Radius of the parameter space θ (parameter norm bound).
Ke
Uniform bound on parameter sensitivity of ODE solutions (controls stability w.r.t. 0).
B
Explicit constant:
6K,K, exp(K Rx + M exp(KyRo) + Ry.
Bleistein & Guilloux (2023)
UP
log(vnC, depends on first-layer weights.
Up
log(vnpC,, KP), depends on hidden-layer weights.
Up
log( (n((VddC,,,K)) depends on bias norms.
KP
Constant depending on network weights: max Bb.CC, Constant deponding on architocture: max{B,c]) BACK, Bu (8q + 12), depth-dependent scaling factor.
KP
C
Table 4: Definitions of constants appearing in generalization bounds in the works of Marion 2023 and
Bleistein & Guilloux (2023

---

Published in Transactions on Machine Learning Research (10/2025)
The bound obtained in work is tighter in terms of the sample size for the linear case, as it achieves rate of O(n for the term involving the Lipschitz constant of the weights, whereas the corresponding rate in Marion (2023) is only (nn Moreover, while the bound in Marion (2023) deptliindepennden but exhibits worse dependence
the network width, our bound shows the opposite behavior: it depends depth but not on width. In the bound given by Bleistein & Guilloux 2023). if we consider the case z(t) = t, which corresponds to neural ODE, the dependence on matches ours; however, their bound depends on the discrenization of time, whereas ours is independent of the discretization size. In addition, our bound is structurally simpler, as involves fewer parameters.
5 NUMERICAL ILLUSTRATIONS
The experiment presented in Figure ! investigates the impact of varying the number of hidden units in a Neural ODE model on generalization crror. Neural ODEs, which model continuous transformations of data using ordinary differential equations, allow for varying model capacily by adjusting the number of hidden units in their ODE block. The input data, denoted as X, consists of 7smmlls random vectors, each with a dimensionality of input dim, and is drawn from a standard normal distribution:
X~ NO0 1) Resmplesxx2 X-
The target values, y, are generated by applying a sinusoidal transformation to the sum of elements in each input vector, where
2
y/ sin Σ
X
for all i (1, Prampless
This non-linear transformation introduces complexity into the data, while keeping the output values within bounded range.
By adjusting the number of hidden units in the ODE block, we analyze how the model's capacity affects its generalization ability, defined as the difference in error on unseen test data after training. The results show that as the number of hidden units increases, the generalization error also increases. This observation empirically validates Theorem 4.9 which suggests that the generalization gap is influenced by the norm bound of the network parameters. As the number of hidden units grows, the norm of the weight. matrices also increases, amplifying the Lipschitz constant of the transformation and making the model small input perturbations. This heightened sensitivity reduces robustness and leads to a larger generalization gap. Thus, the experiment provides empirical confirmation of the theoretical prediction, demonstrating that increasing model capacity leads to higher generalization error due to the growing norm bound for weights.
sensitive to
Generalization Error vs Number Hidden Units
0.005
Generabration Froor
0.004
0.002
0.001
Figure 1: Generalization Error vs Number of Hidden Units in Neural ODE.
We conducted experiments on the MNIST and CIFAR-10 datasets investigate the relationship between the Lipschitz constant of the dynamics function in Neural ODE and its generalization performance. Tn
10

---

Published in Transactions on Machine Learning Research (10/2025)
both cases, we trained Neural ODE model in which the feature dynamics are governed by two-layer fully connected neural network with Tanh activation. The dynamics function f was defined as f(z, W2 tanhha and its Lipschitz constant was approximated by ||W2ll2 WWWlll-llWilll ||Will2, where || denotes the spectral norm. At each training epoch, we recorded the model's training and test accuracios, computed the generalization gap (Lest error minus train error), and measured the Lipschitz constant of the dynamics.
The plots in Figure 2 show that for both MNIST and CIFAR-10, the generalization gap increases with the Lipschitz constant of the dynamics function. This positive correlation is stronger for CIFAR-10, which is more complex dataset than MNIST. The findings suggest that Neural ODEs with higher Lipschitz constants (i.c., more sensitive dynamics) tend to overfit the training data and thus generalize poorly. This behavior aligns with theoretical expectations from generalization bounds for Lipschite-contimmous functions, which suggest that the generalization gap scales proportionally with the Lipschitz constant:
Gen Gap S ff O
VTl
These results highlight the imporlance of controlling the Lipschitz continuity of the learned dynamics Neural ODEs ensure good generalization. Moreover, they provide empirical support for the theoretical notion that models with smoother transformations (lower Lipschitz constants) are less prone to overfitting.
Lipschitz vs Generalization Gap (Neural ODE on MNIST)
Lipschitz vs Gemeralization Gap (Neural ODE on CIFAR-10]
Triin Error)
Generalzattinn Tlost Fino))
0.125
0.075
-0.02
-0.03
Generalization
0.02
-0.44
0.000
0.05
25
60
Lipachitz Constant
Lipschitz Constant
(a) MNIST
(b) CIFAR-10
Figure 2: Generalization gap vs Lipschitz constant for Neural ODEs on MNIST and CIFAR-10.
The experiment shown by figure 3 is to investigate how the generalization gap is related to the Lipschitz constant of weights suPosksL-1 W 1++1 - W The Neural ODE is defined with time-varying weights, where the forward pass involves applying sinusoidal time dependency to the weights of the hidden layer. The model computes the Lipschitz constant by calculating the largest singular value of the weight matrices, which serves a measure of how sensitive the model is to input changes. Lipschitz constant of weights is added as a penalty term in the loss function with different regularization paramcters (A )values. The results are summarized in a box plot, showing the generalization gap versus X, to visualize the impact of varying the penalization factor on the model's generalization performance. As we increase the value of the penalization factor the average generalization gap decreasos which possible only if Lipschitz constant of weights decreases which indicates models with less Lipschitz constant of weights have less generalization gap. So, the Lipschitz constant of weights should be in the numerator, which is confirmed by theoretical bound.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Gap Penalty
3.22
3.34
0.83
0.92
8892
0.80
-0092
001
1 2otion ractor
Figure 3: Plot of generalization gap against regularization parameter for time dependent Neural ODE. Lipschitz constant for weights which is in the bound of solution is added as a penalty term to the loss function. Four different À values (0, 0.01, 0.1, and 1) are lested over 20 trials. For each trial, the generalization gap is calculated the difference between the validation loss and training loss after training for 50 epochs.
An important implication of our generalization bounds is the explicit, dependence on the Lipschitz constant Livee of the Neural ODE's vector field. Since larger Liface leads to looser generalization guarantees, one can improve robustness by explicitly penalizing high Lipschitz constants during training. This connects theory to practice, reducing the Lipschitz constant not only promotes stability of the learned dynamics but also directly tightens the theoretical bound. A simple strategy is add regularization term Jacobian norm V (=(((t,) t)]l as used in Josias & Brink (2022). Alternative proxies include spectral norm constraints on weight matrices or layer-wise gradient penalties, both of which upper-boun the Lipschitz constant.
6 CONCLUSION AND OUTLOOK
In this work, we present the first generalization bounds for both time-independent and time-dependent neural ODEs. Specifically, we derived generalization bounds for time-dependent neural ODEs of the form
d=(t) (())),,,() dt,
and, leveraging this result, we also obtained the corresponding generalization bounds for tide--nne neural ODEs. One of the key insights our analysis is the dependence the gencralization gap on the Lipschitz constant of the weights in the case of time-dependent neural ODEs. This relationship highlights the crucial role of the model's smoothness and stability in its ability generalize to unseen data. An important open direction is to determine the tightness of the error bounds derived in this work. In particular, it remains unclear whether the dependence state-Tipp Lapte-iisschiz constant of dynamic function of Neural ODE can be improved without imposing additional structural assumptions. To the best of our knowledge, matching minimax lower bounds for the general Neural ODE class are not available in the literature. Future work could therefore focus on (i) establishing lower bounds for natural subclasses of Neural ODFs (e.g., vector fields with prescribed smoothness, bounded Jacobians, or contractive dynamics), and (ii) performing systematic scaling experiments, varying time horizon and stale-Lipschitz constant of dynamic function of Neural ODE, and architectural choices, assess whether the theoretical exponents and constants align with empirical generalization behavior.
the key quantities such as sample size, time horizon and
We recognize that stochastic neural ODEs, which have been shown to emerge as the deep limit of wide class of residual neural networks, provide a natural extension to our results. Exploring generalization ion bounds for Neural Stochastic Differential Equations (Neural SDEs) presents an exciting avenue for future work, as it would further deepen our understanding of the connection between stochastic dynamics and generalization in deep learning models. Another promising direction is the application of the mean-field approach, which could offer tighter and more accurate estimates for generalization bounds in both lime-independent and
12

---

Published in Transactions on Machine Learning Research (10/2025)
time-dependent settings. By extending analysis to these more complex models, we hope to contribute to more comprehensive theory of generalization for neural ODEs and their stochastic counterparts.
References
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learning, pp. 254-263, 2018.
Peter L. Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463 482, 2002.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. eegetrally-normalizee margin bounds for neural networks. Advances in neural information processing systems, 30, 2017a.
Peter L. Bartlett, Dylan J Foster, and Matus J Telgarsky. Opectrally-normalle margin bounds for neural networks. Advances in neural information processing systems, 30, 2017b.
Peter L. Bartlett, Dylan J. Foster, and Matus J. Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp. 847-856, 2017c
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight ve-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20 (63):1-17, 17, 2019.
Linus Bleistein and Agathe Guilloux. On the generalization and approximation capacities of neural controlled differential equations. arXiv preprint Xrx::2900 16791, 2023.
Olivier Bousquet and Andre Elisseeff. Stability and generalization. Journal of Machine Learning Research, ((Mar):499 526, 2002.
RTQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equa- tions. Advonces in neural information processing systems, 31, 2018.
RTQ Chen et al. Symplectic recurrent neural networks. ICLR, 2019.
Dean S Clark. Short proof of a discrete gronwall incquality. Discrete applied mathematics, 16(3):279-2 1987.
E. Dupont et al. Augmented neural odes. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Prerona Dutta and Khai T Nguyen. Covering numbers for bounded variation functions. Journal of Mothe- matical Analysis and Applications, 468(2):1131-1144 2018.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing non-vacuous generalization bounds for deep stochastic networks with many more parameters than training data. In Proceedings of the 34th Interna- tional Conference on Machine Learning, pp. 884-893, 2017.
C. Finlay et al. How to train your neural ode: the world of jacobian and kinetic regularization. arXiv preprint Xrx:::2002 02798, 2020.
Walter Gautschi. Some elementary inequalities relating to the gamma and incomplete gamma function. J. Math. Phys, 38(1):77-81, 1959.
Moritz Hardt, Benjamin Rocht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on Machine Learning, pp. 1225 1234, 2016.
John M Holte. Discrete gronwall lemma and applications. In MAA-NCS meeting at the University of North Dakola, volume 24, pp. 1-7, 2009.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Shane Josias and Willie Brink. Jacobian norm regularisation and conditioning in neural odes. In Proceedings of the South African Conference on Artificial Intelligence Research (SACATIR) 2022, 2022. URL https: //appliedmaths. sun.acu.ucc wwt.aac..aaabbaae h/tpa///////amppppp//////1111//0 Africa.
pdf Stellenbosch University, South
P. Kidger. Neural odes: Foundations and applications. ar preprint arXiv:2202.02,35, 2022.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Newral Information Processing Systems, 33:6696 6707, 2020.
Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. arXiv preprint ar xix: 12600, 2019.
Pierre Marion. Generalization bounds for neural ordinary differential equations and deep residual networks. Advances in Newral Information Processing Systems, 36, 2023.
S. Massaroli al. Dissecting neural odes. ar preprint arXiv:2002.0807 08071, 2020a.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissccting neural odes. Advances in Neural Information Processing Systems, 33:3952 3963, 2020b.
David A. McAllester. Pac-baycsian model averaging. In Proceedings of the 12th Annual Conference on Computational Learning Theory, pp. 164-170, 1999.
Mehryar Mohri. Foundations of machine learning, 2018.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
YongKyung Oh, Dongyoung Lim, and Sungil Kim. Stable neural stochastic differential equations in analyzing irregular time series data. arXic preprint arXiv:2402. 14989, 2024.
Saico-Martines et Contimuation of models for nonlinear dynamical systems via the karhunen-loéve decomposition. Proceedings of the 31st IEEE Conference on Decision and Control, 1992.
R. Rico-Martinez et al. Discrete-time models for nonlinear dynamics. Chemical Engineering Communications, 1994.
Nathan Srebro and Karthik Sridharan. Note on refined dudley integral covering number bound. Retrieved from https://www. cs, cornell. edu/-sridharal /td///dddlly//////e pdf
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 5, 2015.
Vladimir N. Vapnik and A. Ya. Chervonenkis. On the uniform convergence relative frequencics of events their probabilities. Theory of Probubility Its Applications, 16(2):264 280, 1971.
Mingze Wang and Chao Ma. Generalization error bounds for deep neural networks trained by sgd. arXiv preprinl ar Xix: 03299, 2022.
Anqi Xu and Maxim Raginsky. information-theorett analysis of generalization capability of learning algo- rithms. In Advances in Newral Information Processing Systems, pp. 2521-2530. 2017.
Appendix
Organization of the Appendix: Section A in appendix provides the proofs for three lemmas which we used prove the main theorem. We proved lemma 4.1 details related to numerical experiments.
4.3 and 4.7 in this section. Section B is devoted
14

---

Published in Transactions on Machine Learning Research (10/2025)
A Proofs
A.1 Proof of bounded variation solutions:
Assume f(z(t), (()) is Lipschitz in with constant Loco as stated in assumption 1, i.e,
(f(=,,t,0(t) f(2,, (())) S Loco 121 22|
Assume f(=(t), 1, (()) is continuous in & on the compact rectangle [0, 1] x (a, bj. By continuity on compact set, f, (()) is bounded:
(((2,4,(0)) & M for some M > 0.
The solution z(t) satisfics
d= = f, with z(0) = x.
dt
By the Picard-Lindelöf theorem, z(t) exists uniquely and is absolutely continuous. Since
dz (=(=((t,,, s M,
dt
the solution z(1) is Lipschitz continuous with constant M:
I=(4,) =(4--)) s M| 4-11 V4,44-1 [0, 1].
For any partition 0 0= to < 1 < ... < tn = 1, apply the Mean Value Theorem (MVT) to 2(t):
z(t;) 2t( = d= (c) (ti - 1;11 for some e; E (t:-1,t4).
dt
Summing the absolute differences gives the total variation:
Σ (t+)) 2t,--)) = n
Σ ((t())) 4-1)-
Since ((, < M, the total variation is bounded by:
(=+2) M (t. t-11 M (1 - 0) = M 80.
The solution 2(t) has finite total variation (2) < M, so it is of bounded variation. The Lipschitz continuity of (==()),,0(1)) in combined with its boundedness due to continuity in 6, ensures the result.
A.2 Proof of lemma 4.1
Proof. Let A(t) be a time-dependent matrix. We assume that A(t) is Lipschitz continuous as stated in assumption 2, meaning there exists a constant LA such that for all t,, 12 E [to.
[A(t)) - A(t2)l < LaIt1 ttl
where LA is the Lipschitz constant and || || is a suitable matrix norm (e.g., Frobenius norm or operator norm).
To express A(t) as a function of initial value A(to), we use the intogral representation:
A(t) = A(to) + f" dA(s)
ds
t0 ds
15

---

Published in Transactions on Machine Learning Research (10/2025)
where dA(s) is the time derivative of A(s), and the integral captures the accumulation of changes over time. ds
Using the assumption that A(I) is Lipschitz continuous, the time derivative constant LA. Therefore, for E [to, tyl, we have:
dA(a) is bounded by the Lipschitz ds
dA(ss
LA
ds
Substituting this bound into the integral representation of A()):
[IA(() A(to)l < ('1 dA(s)
ds S La ds
ds
t0
This simplifies to:
[1(()) - A(60) & Lalt 4ol
Thus, have the bound:
||A(t))l - [[A(tt)] + Lalt - tol
Finally, to remove the time dependency, maximize the bound over the interval [to, tyl:
|((O)) < [[((t)]l + La(ts tt)
Thus, the matrix A(I) is uniformly bounded by time-independent constant MA
||A())) < MA |A([))] + La(l, (o)
Since E [0, L].
MA = ||A(O))I + LAL
(A.5)
For f(x(t), t, (t) defined in 3.3 let us first consider the case when d=1. Let Dz be the distributional derivative of solution function 2 and
- (fo, L)) 2 is non decrcasing] I = {a E L
{2 E (l), |D=[((0, L)) s M).
We know that finding solution to neural ODE 4.1 is equivalent to finding solution to the integral equation
z(t) = x())= f f(x(t),,,0(t) dt.
(A.6)
Taking norms, this yields:
||x(0]|| ||=(0)]| L. (If()(()),,00()) de.
(A.7)
Notice that since we assumed f is Lipschitz with respect to 2, we have that for all E Rd:
(If(=(t),,, < ()f(z(t))t, f(0) 1/(0,,,,)))))) - - (f.. (1000,,0))+++100..000000 + (f(0,t,0(t)) 11f0,,t0)
(A.8)
(A.9)
(A.10)
16

---

Published in Transactions on Machine Learning Research (10/2025)
where the last inequality follows from the fact that f is Lipschitz. It follows that:
||x()]| S [1)1+ L'' (100,,0)+++ dt.
(A.11)
Using the fact that Jo dt = t, one gets:
|=(0|| & ||=(0]|| + 41,(0,,4000) + L, L. [1=(0)1
(A.12)
Applying Gronwall's inequality stated in Lemma B.1.1 yields,
||=((]] s (II=(0)I + (0000, exp
(A.13)
Let ||A,(0)] & BA, and [b:(0)]| & Bb,
Then using equation A.5 we get, [|),(t)]| s [A.(0))) S BAd +L A and [b)( s [|,(00]] L.L s Bog L,L B
Since
[lx( = (1/x(0)) a0)1 S [.]]AA())sx + I,B,
(A.14)
S LLaASsx1100) L.
(A.15)
Using lemma 3.4
N-2
[fx(00)] & L. B
Σ
(LOA)'
(A.16)
==0
N-1 S L. B Σ
(TanA)
(A.17)
1=0
(LA))
L - L.B
(A.18)
This implies
|=(0]] < (I:(0]]
(TonA -1) exp (tL.)) L
tL,B
(A.19)
A.3 Proof of lemma 4.3
Proof. For fixed positive integer n, let us the discretization size as Ar n Ay n To each E I, associate the pair of functions (y++ (xb+(2), [=]) defined by
17

---

Published in Transactions on Machine Learning Research (10/2025)
N-1
*+|=|
Σ k=
Vh
I Ar, (k+ 1) Ax),
(A.20)
where
a(k: . Ax + 0)
Ay
T2((k + 1) Ar - 0)
+1.
Ay
For X+ E EI, define
U(X xx)
(= f= S=Skt}.
Since 2 (X' (2)) the
u = [x]
is covering of I.
Since
#u < 10 < au <
S. 11
SNN N (ax E 2))
and
#10 au S a1 S ... aN-1 S N (ax E N)}
(( .N++1) N P1 PN+1 N}
(2N)
IN
the covering number for the class of functions in bounded by (2)) Consider sums of powers of binomial coefficients: 0% = Em=o (Z) For r = 2, the closed-form solution is given by
12n n
(2) =
i.c., the central binomial coefficients. a obeys the recurrence relation
(n 1)a (2) (4n 2)a 0.
18

---

Published in Transactions on Machine Learning Research (10/2025)
After solving the recurrence relation get,
2
4
3
C1 T(n + 1) 2
n
2n-1
((x), denotes Pochhammer symbol.)
22(n-1)
3
= [ + 1)
2p-1
(since C₁ = 2, which we can
obtain by sctting n previous equation.)
22(n-1) r + n 1) I(n +
r
22(n-1) [(n + 2)
[(n +1)
V /
22(m-1) 1) T(n + 2)
V T(n + 1)
2 T(n + 2)
Vx T(n + 1)
2
5
V (using Lemma 3.6
2
V
(2n)
2
2
(if
6)
6x
2 2
18
Let = [4]] + 1, then
244V
L2(,)) -
18
A.4 Proof of lemma 4.7
Proof. Since,
2164x
<
324
For E Rdd
216LVV3
324
Observe that,
4 LV log = =(
Vlog B' <
19

---

Published in Transactions on Machine Learning Research (10/2025)
Therefore,
J g(r)dr 8y LVd log [vb val
(A.21)
We know that from Lemma (3.8) that empirical Rademacher Complexity (B*) has the following bound
4c + 12 /."
log (T,, B¹, (2)))) dr
(B') S inf <20
where b = SuP P Using A.21
we have
[vb [v- vol }
96 LV log 2
RB'(B') S inf c20
4e +
vn
This implies
V /bV log 2
LVd log
RB. _ 96
576
Experiment details
B.1 For experiment illustrated by figure
The objective of this experiment to analyze the effect of the number of hidden units on the generalization error of a Neural ODE model. The generalization error is defined as the model's performance unseen test data, measured using the mean squared error (MSE). The study investigates the relationship between model complexity, as determined by the number of hidden units, and its ability to generalize.
The dataset is synthetically generated and consists training and testing samples. The training set comprises 100 samples, while the test set includes 30 samples. Each input sample has two features, sampled from a standard normal distribution. The target values are computed using a non-linear function of the inputs with some added randomness. This introduces non-linear relationship between inputs and targets, mimicking the challenges of real-world data.
The Neural ODE model used in this experiment consists of three main components. First, linear input layer maps the imput data into a higher-dimensional space determined by the number of hidden units. Second, the ODE function models the dynamics the hidden state using a fully connected layer with ReLU activation, solving the ODE using the "torchdiffeq..oeiill solver over the time interval [0.0, 1.0). The final state of the ODE solver is passed through an output layer to produce the scalar prediction.
The independent variable in this study is the number of hidden units, which is varied across the follow- ing values: [100, 200, 400, 500, 600, 700, 900). For each configuration, the model is trained for 100 epochs using the Adam optimizer with a learning rate of 0.01. The loss function used is the mean squared error (MSE), and the training process is conducted on a GPU if available. The dependent variable is the generalization error, which evaluated as the mean squared error on the test dataset.
Reproducibility is ensured by sctting random scods for both torch and numpy. The model performance is evaluated by calculating the MSE on the training and test datasets after training. The generalization error is analyzed as function of the number of hidden units, and line plot is generated to visualize this relationship. The x-axis represents the number of hidden units, and the y-axis represents the corresponding generalization error.
The hypothesis the experiment is that increasing the number hidden units will initially reduce the gencralization error as the model's capacity improves. However, beyond certain point, overfitting may occur, leading to an increase in the generalization error. The experiment is designed identify this trend and explore the optimal model complexity for the given task.
20

---

Published in Transactions on Machine Learning Research (10/2025)
B.1.1 Data Generation
The dataset used in this experiment is synthetically generated to test the Neural ODE model's capability to generalize to unseen data. The process creates inpul-oulput pairs based on random inpul vectors and a non-linear transformation for the target values. This ensures that the task is sufficiently challenging while allowing for reproducibility.
The steps for generating the data are as follows:
The input data, denoted as X, is a set rsamples random vectors, where each vector has dimen- sionality of input dim. The elements of X are drawn from standard normal distribution:
X (0, 1) PeamplnxX2
2. The target values, denoted as y, are generated by applying a sinusoidal transformation to the sum of the elements in each input vector:
2 E (1,2,... Rsamples Peappeeel
= sin Σ
Xis
This non-linear transformation introduces complexity into the data while ensuring bounded range for the output values.
3. The inputs X and the corresponding targets y are paired together to form the dataset:
Dataset {(Xi, y:) (
Two datasets are generated:
A training dataset with Psamples = 100.
A testing dataset. with Tisamples 30.
Both datasets are created independently using the same generation process to ensure the test data remains unseen during training.
The generated data stored as PyTorch tensors, making compatible with the Neural ODE model. This enables efficient data loading and processing during training and evaluation.
This synthetic data generation process provides a controlled setup for evaluating the generalization capa- bilities of the Neural ODE model. The use of sinusoidal target function introduces a non-trivial learning problem while maintaining interprotability and case of reproducíbility.
B.2 For experiment illustrated by figure 2
We conducted experiments on both the MNIST and CIFAR-10 datasets to investigate the relationship between the Lipschitz constant of the dynamics function a Neural ODE and generalization gap. The MNIST dataset consists of 60,000 training and 10,000 testing grayscale images handwritten digits, cach of size 28 x 28. The CIFAR-10 dataset includes 50,000 training and 10,000 testing color images of size 32 32 with three RGB channels.
For both datascts, we used a Neural ODE model in which the hidden state evolves according to learned ordinary differential equation (ODE). The ODE's right-hand side function [(((1), 4, 0(1)) is implemented as a two-layer fully connected neural network with Tanh activation. The solution at time t = 1 is passed through linear classifier to produce class logits. The Lipschitz constant of was estimated by computing an upper bound given by the product of the spectral norms (operator L2 norms) of the two weight matrices, i.c., || A
21

---

Published in Transactions on Machine Learning Research (10/2025)
Training was performed for 10 epochs using the Adam optimizer with learning rate of 1 x 10-3 and cross- entropy loss. A batch size of 128 was used for both training and evaluation. At each epoch, we measured the training and test accuracies, computed the generalization gap (defined as test error minus training error), and recorded the Lipschitz constant. To ensure reproducibility, all sources of randomness were controlled by fixing random seeds. Finally, we plotted the generalization gap against the Lipschitz constant. In both datasets, we observed a consistent empirical trend showing that larger Lipschitz constants correspond to higher generalization gaps, suggesting that increased sensitivity in the learned dynamics function may degrade generalization performance.
B.3 For experiment illustrated by figure
This experiment investigates the impact of Lipschitz regularization on the generalization gap Neural Ordinary Differential Equation (ODE) models. A Neural ODE model implemented where the parameters of the ODE depend on time. The primary goal is to examine how adding penalty term proportional to the Lipschitz constant of the model's weights influences the generalization gap, which is defined as the difference between validation loss and training loss.
The Neural ODE model consists of an ODE function with two fully connected layers. The first layer maps 2-dimensional input data to a hidden representation of size 50 with ReL.U activation. The second layer projects this representation back to a 2-dimensional output. To incorporate time-dependency, the hidden layer's output is modulated by sine function of time, introducing dynamic weight adjustment. The ODE is solved using the odeint function from the torchdiffeq library over fixed time interval of [o, 1].
To mcasure and regulate the Lipschitz constant of the model, the singular values of the weight matrices are computed. The Lipschilz constant is defined as the maximun singular value across all weight matrices. During training, the loss function combines the mean squared error (MSE) between model predictions and ground truth labels with a penalty term proportional the Lipschitz constant. The overall loss expressod as:
Loss MSE + X
sup OsksN-1
||At+1 AAll
where À is the regularization strength, and suposkgN-1 | - A&l || is the Lipschitz constant of weights.
The datasots for training and validation are synthetically generated. Both datascts consist of 2-dimensional samples drawn from a standard normal distribution, (0, 1). The training datasel contains 100 samples, while the validation dataset contains 20 samples. The corresponding labels generated by scaling the input data by factor of 2, resulting in a simple lincar relationship. This ensures clear evaluation of the model's generalization capabilities.
B.3.1 Data Generation
In this experiment, the input data and corresponding labels are synthetically generated to evaluate the generalization capability of neural ODE model with a Lipschitz constant penalty. The input data consists of random 2-dimensional points, generated independently from a standard normal distribution. Specifically, for each input data point x = (I1, both features V1 and 22 are independently drawn from the standard normal distribution (O, 1). This ensures that the dataset contains diverse points distributed across the 2-dimensional space. The dataset used for training consists of 100 such points, and the dataset used for validation consists of 20 points.
The corresponding labels for the input data are generated by simple linear transformation. The label for each data point (21 is computed as twice the value of the input features, i.e., y 2 x. This linear transformation ensures that the label is directly related to the input data, which makes it casier for the model to learn the mapping. The training labels Vtrain and validation labels Veal are computed as Ptrain 2 Itran and yval = 2 Teal, respectively.
The datasct randomly split into training and validation datasets. The training datasct consists of 100 data points, and the validation dataset contains 20 data points. This splitting is done to ensure that the model is evaluated unseen data, allowing for the measurement of its generalization performance.
22

---

Published in Transactions on Machine Learning Research (10/2025)
To summarize, the input data is generated by independently sampling 2-dimensional points from standard normal distribution, ensuring variety of input values. The corresponding labels are generated through a
simple linear scaling by a factor 2. The dataset is split into training and validation sets, with 100 samples for training and 20 samples for validation. This dataset setup serves to evaluate the performance ncural ODE model with Lipschitz penalty term.
23


Reviews:
[{"review_id": "V1KJ1VzTqA", "reviewer": "Reviewer_NcmR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the generalization capacities of Neural Ordinary Differential Equations (Neural ODEs), focusing on deriving the first generalization bounds for both time-dependent and time-independent cases where the dynamics are governed by general nonlinear Lipschitz functions. By showing that solutions to Neural ODEs have bounded variation, the authors establish covering number and Rademacher complexity estimates, leading to explicit generalization error bounds. They further compare their results with existing works on Neural ODEs and Neural Controlled ODEs, demonstrating stricter and more informative rates. Numerical experiments on synthetic data, MNIST, and CIFAR-10 validate the theoretical insights, highlighting the role of Lipschitz continuity and parameter norms in shaping generalization performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper provides the first generalization bound for nonlinear Neural ODEs under Lipschitz assumptions, going beyond earlier linear analyses and establishing a new theoretical foundation.\n\n2. The analysis is supported by empirical experiments (synthetic, MNIST, CIFAR-10), which clearly link the theoretical dependence on Lipschitz constants and parameter norms to observed generalization gaps.", "audience_yn": "Yes", "audience_explanation": "This paper focuses on theoretical aspects of Neural ODE which is both timely and important in AI for Science.", "requested_changes": "1. Clarify the source of improvement in Section 5.1: In Section 5.1, the paper compares its bounds with Marion (2023) and Bleistein & Guilloux (2023). It would strengthen the contribution to explicitly identify the technical reason for why the new bounds achieve better rates (e.g., use of bounded variation properties, refined covering number estimates, or sharper use of Gronwall’s lemma). This would highlight the technical novelty beyond applying existing complexity arguments.\n\n2. Discuss the tightness of the bounds: The paper could benefit from analyzing whether the derived error bounds are tight. For example, is the $O(1/\\sqrt{n})$ dependence improvable under additional assumptions, or is it likely optimal given the complexity of Neural ODEs? If current literature lacks lower bounds, the authors could sketch a potential future research direction, such as investigating minimax optimality or empirical tightness under specific architectures.\n\n3. Connect theory to applications: The results could be more impactful by discussing possible application insights. For instance, how might the derived generalization bounds guide regularization strategies in Neural ODE training (e.g., penalizing Lipschitz constants), or inform the design of ODE-based models in time-series tasks, physics-informed learning, or control problems?", "broader_impact_concerns": "I did not find any broader impact concerns because this paper is purely theoretical.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O0EgaFIUHf", "reviewer": "Reviewer_XZUr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on providing a generalization analysis of Neural ODEs, an important and timely problem. However, the presentation is difficult to follow due to unclear definitions, scattered exposition, and inconsistent notations. The manuscript requires significant polishing and restructuring to improve clarity and readability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are plausible and partially supported, but the evidence is neither sufficiently clear nor fully convincing due to unclear definitions and limited experiments.", "audience_yn": "Yes", "audience_explanation": "Audience work on the theory of deep learning, generalization analysis, and neural differential equations would be interested in the findings.", "requested_changes": "1. The paper is difficult to follow, as the problem background and definitions are scattered across Sections 1–4 in a rather confusing manner. I suggest restructuring Section 3 to start with a clear problem definition, providing detailed explanations of the studied problem, the loss function, training data, and ERM. Afterwards, the necessary assumptions and the mathematical tools to be used should be introduced in a systematic way. If possible, it would also be very helpful to include a table summarizing the notations used throughout the paper.\n\n2. Many notations are left undefined, which seriously affects the readability and understanding of the paper. For example, what are $A_i$ and $b_i$ in Assumption 2?  Data distribution $\\mathcal{P}$ is not defined, further, is $\\mathcal{P}$ and known distribution? I notice that the authors state \"$R(h_\\theta)$ cannot be optimized, since we do not have access to the continuous data.\"\n\n3. There are many typos throughout the paper, I strongly suggest authors go through the paper carefully and polish the manuscript. \n\ne.g., \n\n(1) In Lemma 3.9, $L$ should be $\\ell$, and Lipschitzness of a function is not defined.\n\n(2) Throughout the paper, $\\sigma$ sometimes denote activation function, sometimes denote independent Rademacher variable.\n\n(3) First paragraph of Section 4, $1.1$ should be $(1.1)$; $\\sim y, x$ should be $(y,x)$; sometimes use $(y,x)$ sometimes use $(x,y)$, it should be consistent; in the definition of $R$, $\\ell(y, h_\\theta$ should be $\\ell(y, h_\\theta)$. \n\n(4)  In Assumption 1, \"with respect with\" should be \"with respect to\"\n\n(5) At the beginning of Section 5, theorem should be Theorem. \n\n4. The paper’s lemmas and theorems do not specify which assumptions are required for their validity. While the main theorem may indeed rely on all of Assumptions 1–4, several intermediate results (e.g., Lemma 5.1) seem to depend only on a subset of them. This lack of precision is potentially misleading for readers. To improve clarity and rigor, the authors should explicitly state, for each lemma and theorem, which assumptions are being used.\n\n5. The first explicit expression of the neural network appears only in Lemma 5.1. However, the paper presents it purely as a mathematical formula without any accompanying explanation or clarification of the underlying architecture. This lack of exposition significantly hinders readability. In addition, the dimensional consistency is problematic, as biases and weight matrices are not clearly aligned with input, hidden, and output layers. In particular, attention should be paid to the dimensional consistency of $b_i$ and the input $x\\in R^d$.  A clear and systematic definition of the network architecture should be given earlier, including precise dimensions of all parameters.\n\n6. I believe that Theorems 5.10 and 5.12 do not need to be fully stated in the main text. Instead, in the comparison section, the authors could summarize more concisely what each related work studied, what assumptions were made, what type of bound was derived, and in what sense the present results improve upon those bounds. Such a focused comparison would make the contribution much clearer. It may even be helpful to include a table that highlights the differences in assumptions, problem settings, and results between this paper and prior works.\n\n\n7. The experimental section only verifies the qualitative trend that larger Lipschitz constants or higher capacity lead to larger generalization gaps. However, it does not include any direct comparison with prior works such as Marion (2023) or Bleistein $\\&$ Guilloux (2023).\n\nOverall, I am not an expert in Neural Ordinary Differential Equations, and the paper is very difficult for me to understand. This is mainly due to the lack of clarity in definitions, the scattered presentation of problem setup across multiple sections, and the inconsistent or missing explanations of notations and dimensions. These issues significantly reduce the readability of the paper and make it hard for non-experts in the area to follow the technical contributions.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0z94OdPv5S", "reviewer": "Reviewer_HgLk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper establishes generalization bounds on the neural ordinary differential equations. The paper clearly extends in the prior art as it is more general (handles non-linearity in neural networks). I am not expert in this field, but I always welcome contributions in these areas, as I consider this refreshing from experimental works. Moreover, reading these papers gives me better intuition on where model shines and where are their bottlenecks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is quite heavy on the math. I have not redone the proofs, as it would require substantial time investment and I might not be most suitable. But the claims seems to be well supported. What I have enjoyed is how the paper is structured. It starts with review of math for the main theorem and the corresponding proof. From this the reader is primed how the proof will be carried.", "audience_yn": "Yes", "audience_explanation": "I highly value theoretical works and I think that TMLR audience as well.", "requested_changes": "No changes.", "broader_impact_concerns": "Since this is theoretical works, I do not think there are concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)