On the Generalization and Approximation Capacities of Neural Controlled Differential Equations
Authors: Linus Bleistein
Agathe Guilloux
Venue: ICLR 2024 poster (2024)
Keywords: statistical learning
generalization theory
differential equations
time series
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We prove generalization and approximation bounds for neural controlled differential equations that depend on the sampling frequency of the considered time series.

Abstract: Neural Controlled Differential Equations (NCDE) are a state-of-the-art tool for supervised learning with irregularly sampled time series (Kidger 2020). However, no theoretical analysis of their performance has been provided yet, and it remains unclear in particular how the roughness of the sampling affects their predictions. By merging the rich theory of controlled differential equations (CDE) and Lipschitz-based measures of the complexity of deep neural nets, we take a first step towards the theoretical understanding of NCDE. Our first result is a sampling-dependant generalization bound for this class of predictors. In a second time, we leverage the continuity of the flow of CDEs to provide a detailed analysis of both the sampling-induced bias and the approximation bias. Regarding this last result, we show how classical approximation results on neural nets may transfer to NCDE. Our theoretical results are validated through a series of experiments.

Published as a conference paper at ICLR 2024
ON THE GENERALIZATION AND APPROXIMATION CA- PACITIES OF NEURAL CONTROLLED DIFFERENTIAL EQUATIONS
Linus Bleistein Inria Paris, UEVE
Agathe Guilloux Inria Paris
ABSTRACT
Neural Controlled Differential Equations (NCDEs) are a state-of-the-art tool for supervised learning with irregularly sampled time series (Kidger et al. 2020). In this article, we provide the first statistical analysis of their performance by merg- ing the rich theory of controlled differential equations (CDE) and Lipschitz-base based measures of the complexity of deep neural nets. Our first result a generalization bound for this class of predictors that depends on the regularity of the time series data. In second time, we leverage the continuity of the flow of CDEs to provide detailed analysis of both the sampling-induce inndueed bias and the approximation bias. Regarding this last result, we show how classical approximation results on neural nets may transfer to NCDEs. Our theoretical results are validated through a series of experiments.
INTRODUCTION
Time series are ubiquitous in many domains such as finance, agriculture, economics and health- care. A common of tasks consists in predicting an outcome y V, such as scalar label, from a time-evolving set of features. This problem has been addressed with a great variety of meth- ods, ranging from Vector Auto-Regressive (VAR) models (Hamilton) 2020) and Gaussian processes (Roberts et al. 2013), to deep models such as Recurrent Neural Networks (RNN. Elman 1991 and Networks (LSTM, Hochreiter & Schmidhuber 1996).
Long-Short- Song-Short-Tem-s Teiet-TTe---emeeee
Irregular Time Series. In real-world scenarios, time series are often irregularly sampled (when data collection difficult or expensive for instance). Mathematically, for a time series x (Xtg.-.., xtx) E [Rdx this means that Aj == 1g f--1 may vary with j 1,..., K. We are interested in understanding the effect of coarseness of sampling on the learning performance. Intuitively, coarser sampling makes learning harder: bigger gaps between sampling times increase the probability of missing important events say, a spike in the data that help predict the outcome. Formally, it is natural consider that the time series x is a degraded version through subsam- pling of unobserved underlying path
(xz))ejo.1) which, in turn, determines the outcome E ) c R. On high level perspective, we have that y/ F(z) + where F is an operator that maps the space of paths {x : [0. 1] + I to V, and a noise term. For instance, in healthcare, the value of a vital of interest of patient is determined by the continuous and unobserved trajectory of her biomarkers, rather than by the discrete measurements made by a physician.
Models for Irregular Time Series. While many models and approaches have been proposed for learning from irregular time series (see Section 2 quite fruitful point of view is to view the irregular data and the outcome as being linked through a dynamical system. In this article, we focus on Neural Controlled Differential Equations (NCDE), which take this point of view. NCDEs learn dime-dependant vector field, continuously evolving with the streamed data, that steers latent state. The terminal value this latent state is then eventually used for prediction as one would for instance do with RNNs.
Our setup. We restrict our attention to a supervised learning setup in which sampling times are arbitrarily spaced. Consider a sample ((v', x., where y² E y is the label of time series

---

Published as a conference paper at ICLR 2024
tx). We will always require that this sampling grid is constant across individuals, and assume that to = and tx = 1. We consider a predictor fD obtained by empirical risk minimization (ERM) on this sample.
x [d+()+1) discretized arbitrary grid D fto, .t..........
While NCDE have been proven to yield excellent performance supervised learning tasks (Kidger et al., 2020; Morrill et al. 2021a Qian et al. [022]] Seedat et al. [022]] Norcliffe et al., 2023). the literature their theoretical properties still sparse. Remarkably, Kidger (2022) proved that
NCDEs are universal approximators i.c. that given wide enough neural vector field, NCDE can approximate any continuous function mapping the space of paths target space. This result hence transfers a classical result on functional approximation by neural networks to NCDEs. In this article, we aim at filling a theoretical gap on the learning capacities of NCDEs by adressing three critical questions.
1. How do NCDE generalize i.. how well does fo perform on unseen data drawn from the same distribution that the training data ? How strong the curse of dimensionality for NCDEs ?
2. How well can NCDE approximate general controlled dynamical systems ?
3. How does the irregular sampling affect generalization and approximation Put otherwise, how does the predictor jn compare theoretical predictor f obtained by ERM on the unseen streamed data? ?
Our work is, our knowledge, the first address these critical questions. The last point is particu- larly crucial, since it determines how temporal sampling affects prediction performance which is central question both from a theoretical perspective and for practitioners.
Contribution. Our contribution is threefold. First, compling-dependan generalization bound for NCDE. In second time, we build on a CDE-based well- sell-specified model to obtain precise decomposition of the approximation and sampling bias. With these techniques at hand, we are able to precisely assess how approximation and generalization are affected by irregular sampling. Our theoretical results are illustrated by experiments on synthetic data.
a Lipschitz-based argument to obtain
Overview. Section covers related works. Section 3estionndettin details our setup and assumptions. In Section we state our first result which a generalization bound for NCDE. JectionSlgives an upper bound on the total risk, which is second major result. Finally, provide numerical illustrations in Section
2 RELATED WORKS
Hybridization of deep learning and differential equations. The idea of combining differential systems and deep learning algorithms has been around for many years [Rico-Martines et al. 1992] 1994]. The recent work of Chen al. (2018) has revived this idea by proposing model which learns a representation of x E IRd by using it to set the initial condition 20 = (x(()) of the ordinary differential equation (ODE)
dz = (z)
where G. and 2 are neural networks. The terminal value of this ODE is then used as input
of any classical machine learning algorithm. Numeral contributions have build upon this idea and studied the connection between deep learning and differential equations in recent years (Dupont
et al., 2019 Chen et al. 2019 2020b; Finlay et al. 2020). Massaroli et al. 2020 and Kidger (0022) offer comprehensive introductions this topic.
Learning with irregular time series. Numerous models have been proposed to han- dle time-dependant irregular data. [Che et al. (2018) introduce a modified Gated Re- current Unit (GRU) with learnt exponential decay of the hidden state between sam- pling times. Other models hybridizing classical deep learning architectures and ODE in- clude GRU-ODE (De Brouwer et al. 2019) and RNN-ODE (Rubanova et al., 2019).
2

---

Published as a conference paper at ICLR 2024
Neural Controlled Differential Equations have been intro- duced through two distinct frameworks, namely Neural Rough Differential Equations (Morrill et al. 2021b) and NCDEs (Kidger et al. 2020). In this article, we focus on the latter which extends neural ODEs to sequential data by first interpolating time series x" with cubic splines. The first value of the time series and the interpolated path x are then used as initial condition and driving signal of the CDE
2
2
dæ = Gz(Zz)dRee
or equivalently
-1
0
2 0
2
24 22 (20) + S/" (2,
Figure 1: 2D latent state of CDE driven by the smooth path I; = (t. t), i.c. a reg- ular ResNet, on the left and a CDE driven by Brownian motion with drift z, = (1 + W( t + W(()) on the right. The square markers indicate the initial values of the la- tent states. The color gradient indicates evo- lution through time.
We refer to Appendix A.I for formal definition. As for neural ODEs, the terminal value of this NCDE is then used for classification or regression. Other methods for learning from irregular time series include Gaussian Pro- cesses (Li & Marlin, (2066) and the signature transform (Chevyrev & Kormilitzin 2016 Fermanian 2021 Bleis-
tein et al. 2023).
Generalization Bounds The generalization capacities of recurrent networks have been studied since the end of the 1990s, mainly through bounding their VC dimension (Dasgupta & Sontag, 1995 Koiran & Sontag 1998). Bartlett et al. (2017) sparked line of research connecting the Lip- schitz constant of neural networks to their generalization capacities. [Chnn et al. (2020a) leverage this technique to derive generalization bounds for RNN, LSTM and GRU. Fermanian et al. (2021) also obtain generalization bounds for RNN and particular class of NCDE. However, they cru- cially impose restrictive regularity conditions which then allow for linearization of the NCDE in the signature space. Generalization bounds are then derived using kernel learning theory. Hanson & Raginsky 2022) consider an encoder-decoder setup involving excess risk bounds neural ODEs. Closest to our work is the recent work of Marion (2023), which leverages similar tools to study the generalization capacities of a particular class of neural ODE and deep ResNets. Our work is the first one analysing NCDEs. Recent related work is summarized in Table I and detailed comparison between ResNets and NCDEs is given in Appendix [:4-- - see also Figure [ for an illustration. We refer to Appendix
2
and F33 for detailed comparison with Golowich et al. (2018) and Chen et al.
2020a).
Article
Data
Model
Technique
Approcimation
Burtictt et aL 2017
Static
((x) W1. a Wi.1 .... (x))
Lipschitz comiinuit w.t., (W
x
(MLP)
Soquential
H1+1 o (AH, BX, C') (RNN)
Bartlet et al. 2017
Formanizn
Soquestial
Me-1 H - BX, + C) (Residuil RNN)
Kernel-hused
X
generalizazion bounds
Hanson & Raginsky
2023
Statie
Am (m)fm (mee-Amennodee))
(1)
Control theary hoand
NA
Marion
2003
Static
dH, W.n(H (Neuma
Bartlet all 2017
Bett = Hi + Wes1o(HH) (ResNee
This article
Sequenial
dH; Go(H, )dX, (Neural
Banten et all 2017 Contmnit
H++1 Hi + (B...))Xe
the Hlw
(Controlled ResNe))
Table 1: Summary of recent related works on generalization bounds. We do not include prior results leveraging VC dimensions. We refer to the cited work for precise definitions.
Approximation Bounds.
The approximation capacities of neural ODEs have notably been studied
by Zhang et al. (2000) and Ruiz- Balet & Zuazua (2023). In their seminal work on NCDE, Kidger et al. (2020) show that their are dense in the space of continuous functions acting time series. In contrast to this work, we provide a precise decomposition of the approximation error when the outcome is generated by an unknown controlled dynamical system.
3

---

Published as a conference paper at ICLR 2024
3 LEARNING WITH CONTROLLED DIFFERENTIAL EQUATIONS
3.1 NEURAL CONTROLLED DIFFERENTIAL EQUATIONS
We first define Neural Controlled Differential Equations driven by very general class of paths.
Definition 3.1. Let x == (o).) E R be a continuous path and let G : R + R be a neural vector field parameterized by tb. Consider the solution 2 (2r)xe(0.1) E TRP of the controlled differential equation
dæp = GGp
with initial condition 2n (g((x)) where Pp IRd We call z the latent space trajectory and
Rp is neural network parameterized by &-
s T ž =: fo(x) E E,
for & E IRP, the prediction of the NCDE with parameters (2, 2),
The Picard-I Jindelöf Theorem, recalled in Appendix A.2). ensures that the solution of this NCDE is well defined if the path (are (e).1))) is of bounded variation. We make the following hypothesis on in line with this theorem.
Assumption 1. The path (are)e((o.) is -lp-.iise continuous, that | - xzll S Try]t s| for all s, & [0, 1].
Assumption indeed implies that x has finite bounded variation, or put otherwise for all E [0, 1].
| [o.0] sup Σ |ett - 2t. S L S Le.
where the supremum taken on all finite discretizations [0, for N E N Turning to our data ((xPx)), y: we make two assumptions.
(0 to <1 <.. iy t) of
Assumption 2. For every individual (11))((01) satisfying Assumption Assumption 3. There exists a constant B, > such that for every individual i = 1,...,P. n, | s Br.
... n, the time series D, a discretization of path i.e. x xte for all k o...., K.
Neural vector fields. The vector field G can be any common neural network, since most ar- chitectures Lipschitz continuous with respect to their input (Virmaux & Scaman 2018). This ensures that the solution of the NCDE is well defined. We restrict our attention to deep ford-forward neural vector fields with q hidden layers of the form
(2) = a (A,0 [
(
bb)) + bby-)) +by)
with T. Tip-Lipsehitz activation function TR. TR. acting entry-wise that verifies ar (0) 0. Our framework can be extended to architectures with different activation functions: this case, I. will be defined the maximal Lipschitz constant. Also note that the last activation function can be taken to be the identity. For every h = 1....,g, A : IRP + R is a linear operator and E P bias term. Since the neural vector field G maps Rp gaxd the dimensions of the first and last hidden layer must verify Po and Pg-1 dp. To alleviate notations and without loss of generality, we fix the size of hidden layers (but the last one) be uniformly equal p. Put otherwise, for every h = 1, 1...... 1, A E RPxp and b, RP, while A, E RPxdy and by E Rexd
Following Kidger (2022), we restrict ourselves to initializations of the form
20 (x oUUz t),
where U E Rxxd and v E RR, We use the notation NÑU.e : u u- (Uu + v) to refer to this initialization network. The activation function is assumed be identical for the initialization layer and the neural vector field, but such an assumption can be relaxed. The learnable parameters of the model are therefore the linear readout + the hidden parameters v (Aq...., by, by.... bi). and the initialization weights ç = (U,v). We write = (4) sb, E) for the collection of these parameters.
4

---

Published as a conference paper at ICLR 2024
Learning with time series. To apply NCDEs to time series x², needs to embed x into the space of paths of bounded variation. This can be done through any reasonable embedding mapping
p:x (RR) K+1 + (Xt)1
such as splines, polynomials or linear interpolation. In this work, we focus on the fill-forward embedding, which simply defines the value of & between two consecutive points of D as the last observed value of x²
Definition 3.2. The fill-forward embedding of a time series x" = (Xig)...,, Xtx) sampled on D {0 to
lk 1] is the piecewise constant path (Ki)eejo.j defined for every t & + x, Xt, for k E {O..... 1}. and X Xxx I1.
Using this embedding in NCDE with parameters 0 = (Φ, 1h, E) recovers piecewise constant latent space trajectory 2 = (=) ee(0.1) recursively defined for every 1 [ts, 4++1 by
P = - = + Gz(
x(a_1)
(3)
for k = 1, I..... K - 1 and initialized as =8 = = o(U.r + +)).
Formally, the prediction & Tx is equal to + Tf f0 PFF (xP) where FF is the fill-forward
operator. To lighten notations, we simply write t =p fo(x D). This recursive architecture has been studied with random neural vector fields by Cirone al, (2023) under the name of homogenous controlled ResNet because of its resemblance with the popular weight-tied ResNet (He et al.) 2015). We highlight that the latent continuous state z is more informative about the outcome y than =0: embeds the full trajectory of while z0 embeds version of degraded through sampling.
Restrictions on the parameter space. In order to obtain generalization bounds, we must further- more restrict the size of parameter space by requiring that the parameters 0 lie in a bounded set θ This means that there exist Bq, BA, Bb, Bu, B, such that
A < BA. ||bıl| S B, || S Bu, ||||]| < B., || < Ba
(4)
for all h 1, ................................ y and for all NCDEs considered, where ||-|| the Frobenius norm. Such restric- tions are classical for deriving generalization bounds (Bartlett et al.) 2017
Bach 2021
Fermanian
al. 2021). We call
Fo= f (xt) + fe(x) R s.t. 0
this class of predictors. We also write
F8 fo x" E B = S.
s.t.
for the class of homogenous controlled ResNets. We now state important lemma, which ensures that the outputs of NCDEs remain bounded. It is direct consequence of Gronwall's Lemma, re- called in Lemma A.4 We let |D] max
tyl be the greatest gap between two
sampling times.
Lemma 3.3.
The value fe(x) is uniformly upper bounded by
Me = BgLo exp ((BALo) (BuBa + B, + mo))L)))
and [x(xP) ) is upper bounded by
My -B (1 (L,, (Bu + + (k)(
(I. (N. T.. is an upper bound of (O)lop =: max (O)ul]
where Ke (0) Log Bb
||||==
Remark 3.4. Remark that the constant MD grows exponentially with q if BAL, > 1. In practice, NCDEs are however used with shallow neural vector fields, i.e. is typically chosen to be less than 3 (Kidger al. 2020). Additionally, if one wants to very deep neural vector fields, this result suggest controlling their Frobenius norm by setting it equal to to avoid exponential blow-up of the network's Lipschitz constant see for instance Li et al. (2019).
Remark 3.5. It is natural to wonder whether MD > Me as [D] + 0. This is indeed true for well- behaved samplings, i.c. when adding more sampling points decreases the mesh size [D] at sufficient speed. Indeed, if K - = O( Dl). one can use the fact that (1 + ax -'"" + exp(a) as x + to see Me.
that MD
5

---

Published as a conference paper at ICLR 2024
3.2 THE LEARNING PROBLEM
We now detail our learning setup. We consider an i.i.d. sample {(y', x ) Ms x with given discretization D. For given predictor So E Fe, define
(fo) ==1
Ely', fe(xx)))))) and (f(fo)= = En, [(V. fo(xx)))
the empirical risk and the expected risk the discretized data. Similarly, define
R² (fo = Σ (oy.)o(=') and R(fo) En.y (((y)f(()))
as the empirical risk and expected risk on the continuous data. We stress that Rr (Ja) cannot be optimized, since we do not have access to the continuous data. Let fD E arg min R2 (fe) be an AEe
optimal predictor obtained by empirical risk minimization on the discretized data. order to obtain generalization bounds, the following assumptions the loss and the outcome are necessary (Mohri ct al. 2018).
Assumption 4. The outcome E IR is bounded a.s. Assumption 5. The loss l R
x R R is Lipschitz with respect its second variable, that is there exists Le such that for all u, u E y and E 3, (e(y, u) - Ely u')l < Lelu-u')
This hypothesis is satisfied for most classical losses, such as the the mean squared error, as long as the outcome and the predictions are bounded. This is true by Assumption 4 and Lemma 3.3] The loss function is thus bounded since it is continuous a compact set, and we let My be bound on the loss function.
A GENERALIZATION BOUND FOR NCDE
We now state main theorem.
Theorem 4.1. With probability at least the generalization error - is upper bounded by
2AMEL Le
log 1/8
Vn V 22UUP + +(g 1)p(p + 1)U: + dp(2 + p)U + Me
2n.
with UP = log( log( Ve((VCC U. := log(vnpC,K3)) U. = log( (og(vmi //ndCC,K KKP), and Cg =
(8q + 12). KP and KD are two discretization and depth dependant constants equal to KP == max B& MP. B,C.) and K.P" == max BBC. BACK BuCu).
where C Cp. Cv, Cu are Lipschitz constants given in Appendix D.7
This result is similar results obtained for instance by Bartlett et al. (017)) Chen et al. (2020a) and Marion (203) for different models. Indeed, the generalization error is upper bounded by a noise induced term and a complexity term which grows with the square root of the model's complexity measured by the number of hidden layers q and the dimension
the hidden state. This theorem is obtained by upper bounding the covering number of Fe and the Rademacher complexity of NCDEs.
Proposition 4.2. When using discretized inputs, the covering number N (FF.. of F is bounded from above by
C₂KP 1+
(g+1)p(1+p))
(1
C2K R (VD+ 2m
2dp
CKKP
dp²
VP
1+
va)
1+
dp
2m
2m
Furthemoor, the empirical Rademacher complexity (FD) associated to an i.i.d. sample of size discretized D is upper bounded by
4
24M
(a+ 1)p(p 1)UP + 2dpUß + dp?Up.
+
Vn
When using continuous inputs, the constants appearing in the upper bound of Rad(Fo) are dis- cretization independent.

---

Published as a conference paper at ICLR 2024
-1.0
-1.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.0
0.00
0.25
0.50
0.75
1.0
Time
Time
Time
Figure 2: Left: in bold, plot the solutions of two shallow (g 1) NCDEs who only differ in their vector field A and A We then plot the solutions the NCDEs with interpolated vector field 6A (1 A) for 8 [0, 1] in red-blue gradient. Center: consider given NCDE and interpolate linearly between two initial conditions ø0 and zn- Right: we consider a given NCDE and drive it with the linear interpolation of two paths (z:); and (xī) In all three cases, the solutions evolve continuously as we interpolate between the models.
This result is obtained by showing that NCDE are Lipschitz with respect their parameters, such that covering each parameter class yields a covering of the whole class of predictors. combination with arguments borrowed from Chen et al. (2020a) and Bartlett et al. (2017), may then upper bound the empirical Rademacher complexity by making use of the Dudley Integral.
A BOUND ON THE TOTAL RISK
5.1
MODEL
In order to bound the approximation bias, we introduce model that links the underlying path æ to the outcome y/
Assumption 6. There exists vector field G* R Rexd function yp" R - R2 and a vector Φ, E R" that govern a latent state (27 1e through the CDE
dæp G G* )dz
(5)
with initial value aj (xro such that the outcome y is given by
y/ + +
where: is noise term bounded by M2 0.
Assumption 7. The parameters of the well-specified model satisfy ||,,1| S Ba, 8* E C(Rd ((R,)R)) RP) and G Lip( Rexd) with Lipschitz constant LG..
Remark 5.1. This is a very general model, which encompasses ODE-based models used for instance in biology or physics. Indeed, by setting 2; & for all f, one recovers a generic ODE. It also allows for the modelization of complex dynamical systems whose dynamics depend on external time-dependant data. For instance, Simeoni al. (2004) model tumor growth kinetics as a CDE driven by a time series recording the intake of anticancer agents, which perturbs the dynamics of the growth of the tumor.
We call the map f* T. 2f the true predictor. Uniqueness and existence to equation 5 is given by the Picard-Lindesd theorem stated in Appendix[A.2 [A.2] We have the following important lemma, which ensures that outcomes of the well-specified model bounded. Its is analogous to Lemma
3.3
Lemma 5.2. Let y be generated from the CDE (5) from Lig-Lipschitz path æ Let B ||yp* (w))l Then Assumption 4 holds, i.e. y is bounded, and |.
|vl S Bi B + G La) exp (Lo. Ms.
5.2 BOUNDING THE TOTAL RISK
We first decompose the difference between the expected risk of fD learnt from the sample and the expected risk of f*

---

Published as a conference paper at ICLR 2024
Parameter norm
5
Parameler norm
9
00
Value
-0.
0.0 0.25
0.75
1.40
500
1050
1500
2000
Time
Training iterations
Figure 3: Left: Evolution of the prediction (Ø To zc): of a NCDE with shallow neural vector field at
different moments of the training process the true process (Φ, =f); ) is shown in red. Center: Evolution of the parameter's s norms, normalized by their value at initialization, during training. Right: Parameters' norm at the end of training. without normalization, over 25 training instances of the same model. Training is performed with Adam (Kingma & Ba 2014).
Lemma 5.3. For all Ja E Fo, the total risk RD( R(P*) is almost surely bounded from above by
(R) - -R
(R(g) -R(()) + (e))
sup ,EFR
sup
gETs
Discretization
Approximation
Tumning to the total risk, we get the following result by bounding every term of Lemma 5.3 Theorem 5.4. The total risk R (fp) - Rf(*) is bounded from above by
max (Fa) (ad(FF))
(Worst-case bound)
+ L,Ba Lo 1 + (Lo (LoB)) (A)
min(Mg. .Me
+ ke(()) Lu DD] (Discretization Bias)
Be
Lg min max + LeB exp(LG
/ (G)(u) - G"(lillupp + min max IIp² - (N) U. I|u S.
well
(Approximation Bias)
Let us comment the three terms of the bound of Theorem 5.4 The first term is common worst case bound when deriving generalization bounds and is bounded using Proposition 4.2 The second term corresponds to the discretization bias, and is directly proportional to [D]. It therefore vanishes at the same speed as [D] For instance, we consider the sequence Dx equidistant discretizations of [0, 1] with K points, the discretization bias vanishes at linear speed. Finally, the approximation bias writes as the sum of approximation errors on the vector fields and the initial condition, rather than general approximation error the true predictor f*. The diameter of Ω is upper bounded in Proposition C.2
Remark 5.5. This decomposition of the approximation bias allows to leverage approximation re- sults for feed forward neural networks to obtain precise bounds (see for instance Berner et al. 2021 Shen et al. 2021). This left for future work.
Outline of the proof. The proof schematically works in two times. We first bound the two sources of bias by leveraging the continuity the flow of a CDE. Informally, this theorem states that the difference A (teemiinal value) between the terminal value of two CDEs decomposes
A vector fields) + (initial conditions) + A (driving paths).
This neat decomposition allows bound the discretization bias since it depends the terminal value two CDEs whose driving paths differ. This property illustrated Figure Combining these results with Theorem 4.1 gives Theorem 5.4
NUMERICAL ILLUSTRATIONS
Training Dynamics. Our proofs rely the restriction that all considered predictors, including jD obtained by ERM, have parameters lying in bounded set O. To check whether this hypothesis
8

---

Published as a conference paper at ICLR 2024
00
Gen. ciror
0.36
Gen. error
0.16
Value
0.0
2.5
0.36
0.36
0.0
1.0
0.4
0.5 0.
1.0
14 path variation
Time
Sumpling gup IDI
Avg.
Figure 4: Left: latent state (4) T 2t)x at initialization (top) and after 10 (middle) resp. 100 (bot- tom) training steps as the NCDE learns to separate rough (in red) from smooth (in blue) paths on a fine grid. The figure shows examples from the test set. Center: Generalization error |n=! "n=-Ee (xx yy) Ex" ((xx.v) vs. sampling gap. Right: Generalization error vs. aver- age maximal path variation. We train 300 NCDEs, with and the initialization layer left untrained to isolate the effect of the discretization, with time series downsampled on K 5 points randomly chosen in [0, 1] for each run. Training is done with Adam with default parameters (Kingma & Ba
2014).
is reasonable, we examine the evolution of the parameters' norms during training. Figure 3 displays the evolution of the latent state a shallow model (g = 1) through training, along with the evolution of the normalized parameter norm, in a teacher-student i.e. well-specified setup. The trained NCDE achieves perfect interpolation of the outcome but also, more surprisingly, almost perfect interpola- tion of the unobserved latent state. We also observe little dispersion of the parameters' norm over multiple training instances.
Effect of the Discretization. Our bounds highlight the effect of the discretization gap [D] on the generalization capacities of NCDEs. In our second experiment, presented in Figure [ we consider classification task in which a NCDE is trained to distinguish between rough (H 0.4) and smoother (H 0.6) discretized sample paths of fractional Brownian Motion (fBM) with Hurst exponent H. We randomly downsample the paths, train the model and compute the generalization error in terms of binary cross entropy loss. We clear correlation between the generalization error and the sampling gap D). Since our bound involving [D relies on the inequality
max
Dii X
D.i X
max k=I...,K
L - tal S Lg D),
we also check for correlation with the average maximal path variation
X D&
max D......
As predicted, this quantity also correlates with the generalization error. This is consistent with the findings of Marion (203)). who observe that the generalization eиror of ResNets scales with the Lipschitz constant of the weights (see Appendix [44 for a detailed discussion). All experimental details are given in Appendix E
7 CONCLUSION
We have answered three open theoretical questions on NCDE. First, we have provided general- ization bound for this class of models. In a second time, we have shown that under the hypothesis of CDE-based well-specified model, the approximation bias of NCDE decomposes into sum of tractable biases. Most importantly, all our results are discreeizatioonddeeedenn and allow to precisely quantify the theoretical impact of the time series sampling the model's performance. Practical implications for the design of NCDEs and control theory are discussed in Appendix F.I
We thank Pierre Marion and Adeline Fermanian for proofreading and pro- viding insightful comments and remarks, and Cristopher Salvi for discussions Controlled ResNets. LB conducted parts of this research while visiting MBZUAI's Machine Learning De- partment (UAF).
Acknowledgements.

---

Published as a conference paper at ICLR 2024
REFERENCES
Francis Bach. Learning theory from first principles. Online version, 2021.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Julius Berner, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. The modern mathematics of deep learning. arXiv preprint arXiv:2105.04026 pp. 86-114, 2021.
Linus Bleistein, Adeline Fermanian, Anne-Sophie Jannot, and Agathe Guilloux. Learning the dy- namics of sparsely observed interacting systems. arXiv preprint arXiv:2301. 11647, 2023.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Minshuo Chen, Xingguo Li, and Tuo Zhao. On generalization bounds of a family of recurrent neural networks. International Conference on Artificial Intelligence and Statistics, pp. 1233-1243. PMLR, 2020a.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for
invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Learning neural event functions for ordi- nary differential equations. arXiv preprint arXiv:2011.03902, 2020b.
Ilya Chevyrev and Andrey Kormilitzin. A primer the signature method in machine learning. arXiv preprint arXiv:1603.033
2016.
Johnathan Chiu, Samuel Duffield, Max Hunter-Gordon, Kaelan Donatella, Max Aifer, and Andi
Gu. Exploiting inductive biases in video modeling through neural cdes. arXiv preprint arXiv:2311.0498 2023.
Nicola Muca Cirone, Maud Lemercier, and Cristopher Salvi. Neural signature kernels as infinite- width-depth-limits of controlled resnets. arXiv preprint a7X1: 2023.
Dean S Clark. Short proof of discrete gronwall inequality. Discrete applied mathematics, 16(3): 279-281, 1987.
Bhaskar Dasgupta and Eduardo Sontag. Sample complexity for learning recurrent perceptron map- pings. Advances in Neural Information Processing Systems, 8, 1995.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically-obsevee time series. Advances in neural information processing sys- tems, 32, 2019.
Emilien Dupont, Arnaud Doucet, and Yee Whye Tch. Augmented neural odes. Advances in neural information processing systems, 32, 2019.
Jeffrey L Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7:195-225 1991.
Adeline Fermanian. Embedding and learning with signatures. Computational Statistics Data Analysis, 157:107148, 2021.
Adeline Fermanian, Pierre Marion, Jean-Philippe Vert, and Gérard Biau. Framing mn kernel method: neural ode approach. Advances in Neural Information Processing Systems, 34:3121- 3134, 2021.
Chris Finlay, Jörn- Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regularization. In International conference on machine learning, pp. 3154-3164. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Peter K. Friz and Nicolas B Victoir. Multtiiimenioonal stochastic processes as rough paths: theory and applications, volume 120. Cambridge University Press, 2010.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. Conference On Learning Theory, pp. 297-299. PMLR, 2018.
James D Hamilton. Time series analysis. Princeton university press, 2020.
Joshua Hanson and Maxim Raginsky. Fitting an immersed submanifold data via sussmann's orbit theorem. In 2022 IEEE 6lst Conference on Decision and Control (CDC), pp. 5323-5328, IEEE, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arxiv 2015. arXiv preprint arXiv:1512.03335, 14, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Lstm can solve hard long time lag problems. Advances in neural information processing systems, 9, 1996.
John M Holte. Discrete gronwall lemma and applications. In MAA-NCS meeting at the University of North Dakota, volume 24, pp. 1-7, 2009.
Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433. 2022.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equa- tions for irregular time series. Advances in Neural Information Processing Systems, 33:6696- 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Pascal Koiran and Eduardo D. Sontag. Vapnik-chervonenkis dimension of recurrent neural networks. Discrete Applied Mathematics, 86(1):63-79, 1998.
Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks.
IEEE transactions on pattern analysis and machine intelligence, 43(4): 2019.
Steven Cheng-Xins Xian Li and Benjamin M Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. Advances neural information processing systems, 29, 2016.
Pierre Marion. Generalization bounds for neural ordinary differential equations and deep residual networks. arXiv preprint arXiv:2305.06648, 2023.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural odes. Advances in Neural Information Processing Systems, 33:3952-33900 3963, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press. 2018.
James Morrill, Patrick Kidger, Lingyi Yang, and Terry Lyons. Neural controlled differential equa- tions for online prediction tasks. arXiv preprint 1.Xi:2200.111228 2021a.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equa- tions for long time series. In International Conference on Machine Learning, pp. 7829-7838. PMLR, 2021b.
Alexander Luke Ian Norcliffe, Lev Proleev, Diana Mincu, Lee Hartsell, Katherine A Heller, and Subhrajit Roy. Benchmarking continuous time models for predicting multiple sclerosis progres- sion. Transactions on Machine Learning Research, 2023.
Zhaozhi Qian, William Zame, Lucas Fleuren, Paul Elbers, and Mihaela der Schaar. Integrat- ing expert odes into neural odes: pharmacology and disease progression. Advances in Neural Information Processing Systems, 34:1 34:11336. 1361-3441111 11383, 2021.

---

Published as a conference paper at ICLR 2024
Rico-Martinez, JS Anderson, and IG Kevrekidis. Continuous-time nonlinear signal processing: neural network based approach for gray box identification. In Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 596-605. IEEE, 1994.
Ramiro Rico-Martinez, K Krischer, IG Kevrekidis, MC Kube, and JL Hudson. Discrete-vs. continuous-time nonlinear signal processing of cu electrodissolution data. Chemical Engineering Communicaatonss, 118(1):55-45 1992.
Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne
Aigrain. Gaussian processes for time-series modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1984)22010555) 2013.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-amppled time series. Advances in neural information processing systems, 32, 2019.
Domenec Ruiz-Balet and Enrique Zuazua. Neural ode control for classification, approximation, and transport. SIAM Review, 65(3):735-773, 2023.
Patrick Schäfer and Ulf Leser. Teaser: early and accurate time series classification. Data mining and knowledge discovery, 3455):133--136,, 2020.
Nabeel Seedat, Fergus Imrie, Alexis Bellot, Zhaozhi Qian, and Mihaela van der Schaar. Continuous- time modeling of counterfactual outcomes using neural controlled differential equations. arXiv preprint (rXiv:2206.083// 2022.
Zuowei Shen, Haizhao Yang, and Shijun Zhang. Neural network approximation: Three hidden layers are enough. Neural Networks, 141:160-1777 2021.
Monica Simeoni, Paolo Magni, Cristiano Cammia, Giuseppe De Nicolao, Valter Croci, Enrico Pe- senti, Massimiliano Germani, Italo Poggesi, and Maurizio Rocchetti. Predictive pharmacokinetic- pharmacodynamic modeling tumor growth kinetics xenograft models after administration of anticancer agents. Cancer research, 64(3):1094-11 1101, 2004.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural odes and invertible residual networks. International Conference on Machine Learning, pp. 11086- 11095. PMLR, 2020.
Jingdong Zhang, Qunxi Zhu, and Wei Lin. Neural stochastic control. Advances in Neural Informa- tion Processing Systems, 35:9098-91 19089990,,,,1111111111111 2022.
12

---

Published as a conference paper at ICLR 2024
The appendix is structured follows. Appendix A gives preliminary results CDEs and cover- ing numbers. Appendix [33 collects the proofs the generalization bound. Appendix details the proof of the bounds of the approximation and discretization biases and the bound on the total risk. Proofs of Appendix are given Appendix for the sake clarity. Appendix gives all experimental details.
A MATHEMATICAL BACKGROUND
A.1 KIEMANN-STIELTUE INTEGRAL
Lemma A.1. Let f: [o, + Rexd be a continuous function, and g [0, - RR be of finite total variation. Then Sremann-Stieljer integral
S [ggg, R",
where the product is understood in the sense of matrix-vector multiplication, is well-defined and finite for every E [0, 1].
refer to Friz & Victoir 2010) for a thorough introduction to the Riemann-Stieltee integral.
A.2 PICARD-LINDELOE THEOREM
Theorem A.2. Let x be a continuous path of bounded variation, and assume that G RP + Rxxd is Lipschitz continuous. Then the CDE
dz, = G(z)dd
with initial condition zo -p" (2o) has a unique solution.
full proof can be found in Fermanian et al. (202I). Remark that since in our setting, NCDEs are Lipschitz since the neural vector fields we consider are Lipschitz (Virmaux & Scaman 2018 This ensures that the solutions to NCDEs are well defined.
We also need the following variation, which ensures that the CDE driven by x is well-defined.
Lemma A.3. Let be a piecewise constant and right-continuon path taking finite values in 1R with finite number of discontinuities at 0 < ti....., tk = 1, i.e.
lim It Nt; 6-
and
Tt. It;
for all e [t,, Sestebiffor all == I,..., Then the CDE
....., K 1. Assume that IR [Rpx Lipschitz continuous.
dz, G(z))dz
with initial condition zo =p* (mr) has unique solution.
This result be obtained by first remarking that since is piecewise constant, the solution to this CDE will also be piecewise constant, with discontinues at 0 < 41 < ... < lx: indeed, the variations of æ between two points of discontinuity being null, the variations the solution will also be null between these two points. The solution can then be recursively obtained by seeing that for all E [t,, t+11
= 24, = G(2t-1)(x4, -Xt-1)
with 2, (=0) for L. (1. where In 0, and 2tx 91 Utx G(UtK (Xtr -(x-1)-
13

---

Published as a conference paper at ICLR 2024
A.3 GRONWALL'S LEMMAS
Lemma A.4 (Gronwall's Lemma for CDES), Let x [0, 1] - Rdd be continuous path of bounded variations, and Φ [0, IR be a bounded measurable function. If
Ki Liz [' Ø(s) ||xx
for all t {0, 1I, where K₁ is a time dependant constant, then
<(t) & K₁ exp TTx || I-aa [,4)
for all 1 2 0.
See Friz & Victoir (2010) for a proof. Remark that this Lemma does not require Φ(.) to be contin- uous. We also state variant of the discrete Gronwall Lemma, which will allow to obtain the bound for discrete inputs x
Lemma A.5 (Gronwall's Lemma for sequences). Let (yk)a20, (bx)x20 and (fx)220 be positive sequences of real numbers such that
Yn S fn
Σ
biyt
1=0
for all n 2 Then
Un S fn +
fib
II
(1
bb))
1=0
Ø=1+1
for all n 2
A proof can be found in Holte (0099) and Clark 1987.
We finally need variant of Gronwall's
Lemma for sequences.
Lemma A.6. Let (up)220 be a sequence such that for all 2 I.
Us S [g + bb
for (ax)x21 and (bu)k21 positive sequences. Then for all k 2 1
uk S
aj
u0 +
(6)
Ø=1
i=j+1
A.4 DUDLEY'S ENTROPY INTEGRAL
We recall the following Lemma from Bartlett et al. (2017).
Lemma A.7. Let H. be class of real-valued functions taking values in I-M, M). for M > 0. and assume that 0 H. Then the empirical Rademacher complexity associated to a sample of datapoints verifies
4n Vn
12 +
2M
Rad(H) S inf
8),,
m>0
n
A.5 COVERING NUMBERS
First, we recall the definition of the covering number of class of functions.
Definition A.8. Let H {h 21 C ge + R// be class of functions. The covering number A/ (H, of 7 is the minimal cardinality of subset C c H such that for all h E H. there exists h such that
h(x)) h(x) s
sup a
14

---

Published as a conference paper at ICLR 2024
We state Lemma from Chen et al. 2020a) bounding the covering number of sets of linear maps.
Lemma A.9. Ler G = {A E R/xe s.t. ||A] S }} for given X > 0. The covering number N(G.1) n) upper bounded by
1+ Smin(ve, VTJA of
7
15

---

Published as a conference paper at ICLR 2024
We now proceed to prove our main results. Let us precisely detail the different steps of our proofs.
1. For the generalization bound, the first step consists in showing that NCDEs are Lipschitz with respect to their parameters, which we do in Theorem[B.X This leverages the continuity of the flow, since we aim bounding the difference between two NCDEs with different parameters but identical driving paths.
2. Building on Chen et al. (2020a), then obtain covering number for NCDE.
3. We connect the covering number of Fe and F& to its Rademacher complexity building on arguments made by Bartlett et al. (2017) in Proposition 4.2
4. Once this last result is obtained, we can obtain the generalization bound by resorting to classical techniques (Mohri al. 2018).
5. The bound of the discretization bias, stated in Proposition C.1 follows from the continuity of the flow given in Theorem [..] Indeed, the discretization error depends the difference between 21 and zf, which are solutions to identical CDE, but with different driving paths a and embedded path X.
6. The bound of the approximation bias is obtained in similar fashion. Indeed, it directly depends on the difference between =j and 21. However, this time, only the initial condition and vector fields are different, while the driving paths are identical.
7. The bound on the total risk is obtained by combining all the precedent elements and using symmetrization argument upper bound the worst bounds.
B PROOF OF THE GENERALIZATION BOUND
B. LIPSCHITZ CONTINUITY OF NEURAL VECTOR FIELDS
We first state a series of Lemmas on the Lipschitz continuity of neural vector fields. We first prove Lipschitz continuity with respect to their inputs.
Lemma B.1. Let Goy be neural vector field of depth 9 2 1 as defined in Equation Then for all 2, W E R
| (=) - (w)()) < BA || will
(8)
Our second Lemma shows that the operator norm of neural vector fields grows linearly with Lemma B.2. Let G IP + IRp be neural vector field of depth q. For all E IR? have
(A) -
[G BA) || + L. Bb L BA
and in particular
BA) = (0).
GG)] S Lb
(10)
Lg BA
Let G and Gya be two deep neural vector fields, as defined Equation T with learnable param-
eters 3 = (A)..... A]. Lemma.
AA b........ .......,)))) bi) and 2² ? A
b......bb b We have the following
Lemma B.3 (Lipschitz Continuity w.r.t. the parameters). Let TR and G G : Rp Roxd be two newral vector fields defined as above. Then for all E R
G
G (z)
-
|A] - -3ll
(11)
1=1
for (C and (CA (z))²= two sequences of constants explicitly given in the proof.
16

---

Published as a conference paper at ICLR 2024
B.2 AN UPPER BOUND ON THE TOTAL VARIATION OF THE SOLUTION A CDE
We have the following bound on the total variation of the solution to a CDE.
Proposition B.4. Let F R + R be Lipschitz vector field with Lipschitz constant Ly and
(z.)) i) be continuous path of total variation bounded by Laz- Let 2 (21)re(o.1) be the solution of the CDE
dz (2e)
with initial condition 20 RP. Then for all E [0, 1}, we have that
|| |ell I C1 (F) || [000
with
C1 (F) = LF (F(112011 + [F((O]l Lo) exp(LpL,) + [F(O)) exp(LFLo)
One also has that the total variations of and z are both upper bounded and we have
|| $ [LooBa)
MM + ko) BA)((LoBa)Laa))) La)
Bs
and
|| 1-var
(LoBB))
Mg ne(o) exp(LLBB))) Lx)
(12)
Ba
The proof is given in Appendix D.5
B.3 CONTINUITY OF THE FLOW OF A CDE
The following theorem is central for our analysis of the approximation bias. Its core idea is that the difference between the solution of two CDEs can be upper bounded by the sum of the differences between their vector fields, their driving paths and their initial values.
Theorem B.5. Let F, G Rp Rxxd be two Lipschitz vector fields with Lipschitz constants LE LG. Let x, r be either continuous or piecewise constant paths of total variations bounded by Le and Ly,, Consider the controlled differential equations
dw, FF and du, G(ve)dre
with initial conditions wo I2 and vo Rº respectively. We have that
1/,ee
uell 5 |lw0 voll + I|:0 roll
|| - rlloo,uu. (1 Lp
+ max F(v) - G llop Jep
x exp(LpLzz)
where the constant C₁ (F) is given in Proposition B.4 and the set St is defined as
= (u E TRP || < (|lool |I + G I.)) exp([x....)
The proof is given in Appendix D.6 We stress that any combination of continuous and piecewise constant paths can be used.
Using this result, we obtain the following theorem on the difference between predictors.
Theorem B.6. Let fe, fez E Fe two predictors with respective parameters 0, = (vi, U1, 01, 21) and (Wi,, U2, U2,, 42). We have that
1so (x) S0z(x) (x)
17

---

Published as a conference paper at ICLR 2024
is upper bounded by
9 Σ A
9
Me 1
2211
CA
A3ll
+ Cb Σ
[b] - -b3l + C U2/ + C. 1|0 - -2ll
1=1
f=1
In turn,
Vo,(x))
S02
is upper bounded by
9 C
9 E1 b31 + CullU U2ll + +C 1|0 U2
MP $1 221
11
A31 + Cp
;=1
j=1
The Lipschitz constants are given explicitly in the proof.
The proof is in Appendix D.7
B.4 PROOF OF PROPOSITION 4.2 (COVERING NUMBER)
We follow the proof strategy of Chen al. 2020a). Starting from Theorem B.6 one that since
sup [o, (a:) - (e)())l
S Me 101 - -211 + CA Σ ||] - -33 + C. Σ bb -b3l + Cu |U - -21 + C. 1| u2ll. j=1
where the supremum is taken on all such that ||=1ll
S L and |!0م|| S Bz, it is sufficient to
have 0 = (Ø) 3. Ú, i) such that for all j = I,....
|| A A S
77
(13)
(2q + 3)CA' :
(24 + +)
and
11 (2q + Me
- S (2q 3)Cu
7
||0 -11 5
(14)
(2q 3)C,
to obtain an 17 covering of Fe, since in this case we get that
sup sup (x) fa(x)) En
(15)
We now use Lemma A.9 and denote by Go = {h: 2 + T 2, ||0 ing definitions for GA,. Sby. Gu and G... We first get that
S Ba), and use the correspond-
(9e. (2q )5
(4q B) M
1+
(16)
(4g + 6) mint vD. C...V.le vasloCe) de
NGGu (2q +3)CU
1
and
(17)
" (4g + B) ) 1
N( (2q 3)C,,
(18)
Turning to 16, for all j = 1,.... --1
(4g + G)BBACC x²
((A,,) (2q + 3)CA
1
(19)
17
(( (2g + +3)
11
(44 + GbbBbb
1+
(20)
(21)
18

---

Published as a conference paper at ICLR 2024
and finally, since Ag and by do not have the same dimension as the previous weights and biases,
1+ (44 (4q 6) 4x²
N(SA,: (2q +3)C2
(22)
7
2 1 (4g ( 6) min (vd, /D) BbCb
Ng (2q 3)Cb
(23)
The covering number of Fe is obtained by multiplying the covering number of each functional class (Chen et al. 2020a). Defining
Ki max MB, B,C.),
(24)
K₂ == max BACA
(25)
(26)
and using the fact that min{vd, vD) S Vdp.we finally get that ([Fe. n7) is upper bounded by
28
(0-1)p(1+p)
1 (4q + 6
(4q + 6)
x
1+
vp
(27)
17
2dp ( (1+ 1+
(49 + 6)
(4q + 6)
x
1+
/dp
v dp
(28)
11
which is itself bounded by
(4g +6]K2
2p
(4q + K2
(a- (o-1)x((+p)
(1 (1+ | (4g + 6) /dp
do(2+t)
1
x
14
vp
"
"
(29)
The proof is identical for inputs x, but with constants
KP = max BB B.Co),
(30)
KD = max{BB B BuCu)
(31)
where the constant MD is given in Lemma 3.3 the constants B B,, By and B, are given in Section 3.1 and the remaining constants come from the proof provided in D.7 This concludes the proof.
B.5 PROOF OF PROPOSITION 4.2 RADEMACHER COMPLEXITY)
We apply Lemma A.7 to the class Fe. We trivially have that 0 E F, since this function is recovered by taking 0 = By Lemma 3.3 the value of So bounded by
Me Lg exp ((BAL)) Bu + B. + ko()L).)
(32)
19

---

Published as a conference paper at ICLR 2024
In our setup, get, from Proposition 4.2 that
2Mavi
VlogN
(33)
2plog 11+ (49 ()
(q 18-111 11-()p(p-1 1) 1) (1 (1+ (4q
(4q - (o-ppp
(34)
1/2
+ dp(2 + p) log (1+ (4q+ 614+6lydpl dpkd
ds
(35)
< 2M 2plog (1+ (4q + 6)
+ 1)ppp + l) 11 (4g + 6) 6)kk
(36)
7
+ dp(2 + log (1 (4q + (
(37)
1
Since for x > 1, lolgl11 + x) S log(2x)), we have, for 7 small enough that
MMv
v (oo. s)ds
(38)
(8q + 12)K
(8g + 12)
2MMv
2plog
+ (1- 1)p(p + log
(39)
"
"
(8q + 1
+ dp(2 + p) log
(40)
1
Taking 1)
- one gets that (ad(Fe) is upper bounded by
Vx
4
24Me +
2plog (CoVTiKi) + (g 1)p(p + log (Coy/pp22) + dp(2 + p) log (C, V ndpk22)
Vn
with Cy (8q 12).
B.6 PROOF THEOREM
We can direcltly apply the results from Mohri et al. (2018). Theorem 11.3. Since our loss is L Lipschitz and bounded by Me. this gets us that with probability at least 8,
RD()P) Rn(f²) 24MBL Le Vn
220U3 (q 1)p(p 1)U2 + dp(2 p)U + Mep
log 1/8
2n
(41)
with UP log CyynKP. UP log CyynpK? and UP log C. /ndpkp
20

---

Published as a conference paper at ICLR 2024
C
PROOF OF BIAS BOUNDS
C.I PROOF OF LEMMA 5.3
For all f Fe. have the decomposition
RD R - R((*) = Rf()D) - (P
(42)
+ (fD) R2()
(43)
+ (f) Rn(())
(44)
+ +R.(f) - R())
(45)
+R(f) R(f").
(46)
By optimality of fP, have that almost surely
- - 5.
(47)
One is then left with the inequality
RD()D) - - Rf()P) (PD
(48)
+ +R Rn(
(49)
RR(ff) R(f)
(50)
+ +(()) - -R(f*).
(51)
Taking the supremum on the two differences between the empirical and expected risk yields that
PP))P) - R(P*) < sup gEfu
[R"(g) R2 (g) + sup gEfe
(R(s) Rr (g)
(52)
(f) (f)))f((()))) R(f) R(f").
(53)
This concludes the proof.
C.2 BOUNDING THE DISCRETIZATION BIAS
We have the following bound of the discretization bias. Proposition C.1. For any fo E Fe, we have that
Rh (f) S D).
where Cl. is equal to LeBabe + (LoBA) (LolBa))'((A)) (Lo
min MM. Me}
ne(
Bg
Proof. Take fo E Fg. We have
RD) Rr (n)) - n1 Σ n
Ml' So(' - sa("))
E('',
(54)
Considering a single individual i fl,..., nJ. we have
e So , fe(x")) < (x - so(x")
(55)
using the Lipschitz continuity of the loss function with respect to its second argument. From the Cauchy-Schwarz inequality, it follows that
[ fo(x*) [ (2 -1)1 B 2
(56)
21

---

Published as a conference paper at ICLR 2024
where 2d and zi refer to the endpoint of the latent space trajectory of the NCDE, resp. with discrete and continuous input, associated to the predictor fa.
= P.)) and 2t = fa(=²) correspond to the endpoint of two CDEs with identical vector field and identical initial condition, but whose driving paths differ.
We now ready zi = fe(x') correspond to the endpoint of two CDEs with identical vector field and identical initial condition, but whose driving paths differ. Theorem B.5 thus collapses
the continuity of the flow stated in Theorem B.5
21
fo(xP.)) and
D. zi - exp (LG.Ls) L.) 1+ LG, L., || ||--
10.10.11
(57)
where La, is the Lipschitz constant of the neural vector field Gw. Since, according to Lemmas B.1 and B.2
LG = =LLo
and
CG = ( (Io Bæ
min(Me, MB) + (0)
we obtain the inequality
D - o
21
(58)
min(Mg. .M)
S exp (L))
||=
x x1100,00011
1+ (LoBa)* (LBBB)((LLBBA))
Ko (e(())
L
Bq.
(59)
We also have that
x D. x²
lo0.,000) j=l....K x
Bi - S Ln
(60)
since the discretization is identical between individuals. Putting everything together, this yields for all i 1....,,2 n that
1fo(xD.i) - fa(x')]
(61)
m Me] + (0) L
S By exp (Lo Lu 1 (A) (L,
[D]
Bx
(62)
and finally (fo) (fo) - R, (fo)
(63)
S TTeBo exp (A) Tiz 1 (T.BAa) (ITBA)(((TTBA))
min[MM. Me)
+ ke())) Tur
[7]
B
(64)
=: C: DI.
(65)
This concludes the proof.
C.3 BOUNDING THE APPROXIMATION BIAS
We have the following bound the approximation bias.
Proposition C.2. For any fo E Fe with parameters 0 = (Φ, U, the approximation bias
R(fo) R(f") is bounded from above by
B. exp(Le- Lo G
G looc + LB exp(LG. I1pp NNu,elbo.BB
Me
| -11,
B+
where
Ω {u RRP || & (Bu Br BB) Lap Bxx) BA
22

---

Published as a conference paper at ICLR 2024
f1P
Figure 5: schematic illustration the main bounded sets used the proof. The initial values of the time series used for initializing the NCDE lie in the orange ball of diameter Bz. The value 20 then lies in the blue ball B1, whose diameter is upper bounded by function of Bu, Bu, Br and Lg. Finally, the trajectories evolves during time but stay within the ball ΩDD
Proof. For any So Fo with continuous input, the approximation bias writes
R(fe) R(f*) | [[ fa(z)) f(y. (x)))
(66)
Using the Lipschitz continuity of and the Cauchy-Schwarr incquality, one gets fly, fo(x)) fly, f* (1)) T4 +. + || B. ||= 21|l
(67)
Since o1 and 21 the solution two CDEs with different vector fields and different initial condi- tions, the continuity of the flow stated in Theorem B.5 yields
| - =1ll S exp(I.g. lw))
L max welt
(G)())
G*()lll + max 11s*(n)) - NNN.o(1)ll]
(68)
|
where
Ac fu R2I ||||| <
MM}
B
Putting everything together and taking expectations yields
R(fo) R(f*) < LeBs exp(LG. L.) Lx max
G.(() G* G'(']
(69)
well
+ max
||00 (u) (NN,,)NNN,.,.
NNU,
(70)
||a|| B.
Me
+ Bs
|b. 10-11 111
(71)
This concludes the proof.
C.4 PROOF OF THEOREM 5.4
We gave bounds for the discretization and approximation biases Propositions C.I
and C We
now have to bound sup
'R" (g) R2 (g) and sup gEFe
R((g) R, (9) obtain bound for the
geFe
23

---

Published as a conference paper at ICLR 2024
total risk via the total risk decomposition given in Lemma 5.3 parameterized by = (#, क,, U, U) we have
It is clear that for any predictor fo
Ra (f)) -R(f")
(72)
$ sup gefn
PR_ - (R((()) sup gefo
(R(g) R. (0)
(73)
+ D
(74)
+ B exp(Le Lxx)
Ly |G - Go + +1l1--
NNU,ello,
(75)
Le
+ B
11.- 1 $11
(76)
Now, by a classical symmetrization argument see for instance Bach 2021) Proposition 4.2 the obtained bounds on the Rademacher complexity imply that
sup (R(g) Rn(()) S (Ra)(Fo)
(77)
gEFe
and
sup gET
Rg) RR s (Ro).
(78)
This gives
sup gEfn
(g) (2) + sup
R(g) - (0) < 4 max (Rad(Fg)) Rad (Fo))
(79)
gEfo
To finish the proof, notice that since the risk decomposition
Ra R R(f") I sup
-R? (o) + sup
[R(g) - Rn(o)]
gEFe
gEFs
R (f) - RR(() + R(ʃ) - R(f*)
& max (Rad Rad( (Fo)) + D
+ La Bu exp I,o)
Lu Gp G* + 11 - -NNU, loo,
Me
10. - -1
B.
holds for every fo Fe parameterized by (b, U, may take the infimum over all possible predictors Fe. Since this parameter space is compact by definition, the infimum a minimum and
( -
< 4 maxx (Ra(FFF) [ad(Fo)) + C.. D
LeBo exp La Lxx) Ly min | Go G
+ min ||+" - -N 0.0
LeMe
min 1b.
Φ
B4
which concludes the proof.
24

---

Published as a conference paper at ICLR 2024
D SUPPLEMENTARY PROOFS
D.1 PROOF OF LEMMA B.I
Take G RP Ixxd neural vector field and 2, E IRP. Define
(x)=) = o
bn-1) + bn.)
for all h 2. One directly has
[G)((z) (w=)l = (1Ha(z) - Ho (()wwll S Lo (()) - Mg-1(w))
(80)
s BA 11)))(22) - M
(81)
(82)
$ (L,BA) ) 1|2 -lll-
(83)
This proves our claim.
D.2 PROOF OF LEMMA B.2
Let E RR and define
px(2) = o A A
(A AA
b/)) ...00. (...(AA2 b/-1) bj.
for all h = .,..... We have
[G)(())lopp ||1a(2)]log
(84)
(1))(2) By (0) H4 (0)] (1) By (0) + (0)l
(85)
(86)
Using Lemma B.1 we have that
[1)](2) -,,(0)) S (LoBA) ||
(87)
To finish the proof, we have bound [0)]] Since
|| (0)l = (1,,(()) (0)] < Toa 11 (0)] + Tig Bb
(88)
L, BA [ + L. Bb.
(89)
using Lemma A.6 yields
9-2 [|,( S (LoBA)* (I((b))) L.Bb Σ (LoBA)' 5=0
(90)
L. (L)
(91)
(L,BA)
L. L B A
(92)
Combining everything yields
(A) -
[G)(l))l < (I.BA)'Ilx
||||| |122||
LoBo
(93)
Lg BA
25

---

Published as a conference paper at ICLR 2024
D.3
PROOF OF LEMMA B.3
Define
(h(2)) == a (Apo (Ah ((.. (Al2+bb)) + -b-1)) + bj. + b2 + by
and
(z)())== =- o (A (A.
for all h = 2....., 9. The indice h therefore refers to the depth of the neural vector field.
For the first hidden layer, we have
|0 (A - o bi) Lo
b) b b²
Lo || - A3pl
(94)
S bj bil
+ L. ||A] - -All |=||-
(95)
Similarly, for any h = .....,
[1) - (=)] = ( (A) 1 11) + (x)++ - -
(A) (2) bj, ||
(96)
S T.
| 1
A + L. |b] - b
(97)
(A) A + AA (3) - - + Lo ||b/ bj
< Lo
(98)
$ L.
A All [1a) (3) + L. || 11/ (z) - -1h-1(z))
(99)
+ L
[b b|
(100)
Since we have assumed that
| <
BA, this yields
[|p((2) p=)
< L
A
All [1)]] + Lo BA ( -
(101)
Lo
by
bill |l
(102)
We may now use Lemma A.6 get that [1))(=) - (=)) S La
Σ
||A]
A/] + |bb] - -33l] b3]l] [Lo
(103)
1=2
+ (Li BA) 9-1
[A - o (Af}= + +b)l
(104)
Lg
||A] A31l (3)] + [bb] bill (L. 9-
(105)
j=2
+ Lg A
|bb] - bill + + A A
(106)
We may now use Lemma B.2 For all h- = 1, 1...,.., 4. we have
[|)] - an ||=|| + 3r
(107)
with
Ba) -
ag = (Lo BA) h and 3 = La Bb
(108)
BA -
We furthermore let ag := 1 and B0 = 0. One then has
[1x](2) - n²())]]
(109)
< Lo Σ
||A} - (gg +| + - [|b]- (LoBA)
(110)
1=1
and thus
Gui (z) G=)(:)ll 5 Σ -31 |Aj -331]
(111)
j=1
with C²(2)== = (LoBA) 8-3 Lo (az- ||=||| + 8/رر) and Cg = (L.BA) 9-3 Lo. This proves our claim.
26

---

Published as a conference paper at ICLR 2024
D.4 PROOF OF LEMMA 3.3 AND LEMMA 5.2
We recall that |[W lop == max [W and that || hp S
[W], which means that ||a] S
|=]|=|
[| ||]||| for all æ. We first prove this result for a general CDE
dzp
dz
with initial condition E RP, where the driving path x is supposed to be continuous of bounded variation (or piecewise constant with a finite number of discontinuities.)
By definition,
( F(zi)dr
24 20
(112)
Taking norms, this yields
- S | || { [F(z)) ||daell
(113)
Notice that since we have assumed to be Lipschitz, we have that for all 2 E R
FF) < F(2) F (0) + [F(O))
(114)
F 22) F(O) [F
(115)
s LF | + [O)ll)
(116)
where the last inequality follows from the fact that Lipschitz. It follows that
|| s || + S
((L (LD || [[F(O) ||
(117)
Using the fact that lo |dx = || [lalll one gets
a |0 + [[ ||=|| var. Lo /''² ||zal| ||dxall-
(118)
Applying Gronwall's Lemma for CDEs yields
| |l < || + |F(O)]l || (va,))) exp (LF || || (.var.
(119)
Now, turning to the generative CDE equation 5 we obtain as a consequence that
v S B$
B G (0)ll L..) exp (Lg. Lm) M
(120)
since y is equal to the sum of linear transformation of the endpoint of CDE an noise term bounded by Mg. This proves Lemma 5.2
Turning to E Fe, have that
[f(())] Φ 21 < || Bx ||
(121)
and since is the solution of NCDE, can directly leverage the previous result to bound |||||| which yields
|| $ L, exp ((BAL,) Bu B, (0)
(122)
As direct consequence,
[fo(z)) BgL, exp ((BAL.)LL.) ( + B, ko(()).) <0.
(123)
27

---

Published as a conference paper at ICLR 2024
We turn to fo(x?). Bounding the value of xP be done by applying this lemma to the values of 2d at points D. since it is constant between those points. The proof leverages the discrete version of Gronwall's Lemma, stated in Lemma A.5
We have that
| S ||= +
Σ 1=0
|1
G [
(124)
1=0
for all k 1...., K. Now, since
(A) -1 La Ba
[G)(l (Lo | + LoBb
(125)
we have
BA) -1 k-1
|| 5 |00|| Bb
|| A ||
(To BA)
(126)
L BA
1=0
1=0
(L. -1
$ ||00|| || + LbBo
Lu + (Lo
| ||
(127)
Lgg BA
1=0
and one gets by the discrete Gronwall Lemma that
(Lg BA) -
R-1
1 < ||00|| + Lig Bb
(A)
A
(128)
Lig BA
I=0
s ( |2ull + Lb
Lo Ba)% Lu
k-1
11 [L. (A. LLo
(129)
LABA
I=0
since A
< Lxltt41 t/l S L D), and finally
| < (LoBuBe + B. + LoBD (Lo Ba)
1-1
Lu
1 + LLo
(130)
La BA
1=0
In the end, this gets us that
BA) -1
K-1
[so(xP) & B. (Lg BuB + B + Lgg Bb
II
11 1 (L)
L BA
1=0
(131)
(Lo 9
ByLo LoBuBB Bu LoBb
Lod
(1+ (Lo
LABA
(132)
Note that by Gronwall's Lemma, one also has the less tighter bound
[o(x S Bo T. exp ((BA..) (Po (By + B, + (.)
(133)
This concludes the proof.
D.5 PROOF OF PROPOSITION
We first consider general CDE
dz, F(2
(134)
28

---

Published as a conference paper at ICLR 2024
with initial condition 20 E RR. Take 8, E [0, By definition,
2, 2
F(zu)dau
(135)
I F(Zu 2x + =2) da)
(136)
(Ix 11=u-2
- 22 + 2zll + [[F ||drull
(137)
Σ / (Lr || - -2 + LF | + [[(()) (p)pp)) ||dxull
(138)
|| var.sr)) + LF [ || zall |dr.ul
(F ||= + |I((O))
(139)
Now since 2, is the solution of CDE evaluated at it can be bounded by Lemma 3.3 by
||>/|| = ( z + |[F(O)) || exp(Le || (|xrlla-var.0o.)))
(140)
This means that
2r - zall
(141)
$ LP [|z0|] + [[F(O)] varl) ) exp(LF ||x - ||F(0)]lmp || I
(142)
+ LF [ ||zu - z. [|dall-
(143)
We may Gronwall's Lemma and the fact that || ||r Ly for all [0, 1] obtain that
||,, 2z S Tr || + [[F(0)]l.p T..) exp(Ixr + [[F(O))..p exp(IxTeor) ||rrIl
(144)
and thus
|| 2.11 S CF(F)
[sar) (
(145)
This means that the variations of z on arbitrary intervals [r, s] are bounded by the variations of x on the same interval, times an interval independent constant.
From this, we may immediately conclude that
|| ||=||i-ar,10.0] I CF(F) | [1-var
(146)
which concludes the proof for a general CDE. We now turn to fo E Fe. The previous result allows to bound the total variation of CDE and thus of the trajectory of the latent state. Using this proposition with
(G
(147)
(LoBA)" LLBBuB (LoBa)((LoBUB,, + Lo + (0))) exp ((L, + ke( exp "
(148)
1 = (LnBA
MM ko( (sp((LBAA)II)
(149)
Ba
yields
| (G
(150)
This concludes the proof for fo with continuous input. For discretized inputs, the proof follows readily with constant MD.
29

---

Published as a conference paper at ICLR 2024
D.6 PROOF OF THEOREM B.5
For all E [0, 1J. we have the decomposition
Wr - U/ = wp 2o + / F(wa)drr L/" G(vs)drs
(151)
vo + /'" (F(ws) - F(us)) + ! F(us)dzs - f G(u,)dr.
(152)
wp 0o
(153)
+ ! (F(ws) - F(u
(154)
/ F(va)d(os-1) -1))
+
(155)
/'" (F(v)) Grr)))drs.
(156)
We control every one of these terms separately and conclude by applying Gronwall's Lemma. Writ- ing
A =
(F(ws) - F(v.))
B I F(U,)d(xx -ra)
C | (F(v.) G(vs))drss
it is clear that
||u, 211l I Ilu0 zoll + A+ B B+C CC
(157)
Control of the term A.
We have
A AS //' F(w.)) F(rs)llop ||dx
(158)
Since F' is LE- Li-isccchitz this gives
AS LF / I|w, - uzll ||dx=ll-
(159)
Control of the term B. One gets using integration by parts
L F(v.)d(x, - re) = (x: rz) - (ro ro) S' (T) TdF
(160)
This gives
B -0 roll ||x r rlloo,10.1) (1 + |[(()) ( (..0)
(161)
Since is Lp-Lipschizz one gets
F(v)ll 1 LF |||||| (
(162)
Since is the solution CDE, resort to Proposition B.4 to bound its total variation. This yields
[|u]l Ci(LF, F, L.) ||]|| var.
(163)
and finally
B
120 roll + || rll30,10.. (1 CrCi(Tr. Tzz) ||rl (arlo)
(164)
30

---

Published as a conference paper at ICLR 2024
Control of the term C. Finally, get that
c 0 [(.)) G G( ||dr,ll
(165)
< max
[[()) - (())lllg Lr
(166)
ven
where we recall that Ω is the closed ball
u E Re
||||| < (1/= + [|G I.,) [x))xx((
Indeed, since is the solution of a CDE, its norm any time E [0, is bounded as stated in Lemma 3.3 One can therefore bound the difference [(.)) - G(v,) ap by considering all possible values of U.
Putting everything together.
Combining the obtained bounds on terms, one left with
||3 rell s ||wg voll
(167)
+ || - roll + || - to.0 (1+ (4p,
(168)
+ L, max [F(u) Gu wesl
(169)
+ LF / ||w, v.s ||dr=ll-
(170)
The proof is concluded by using Gronwall's Lemma A.4 If the path (1r) is continuous, we resort to its continuous version. If piecewise constant, we use its discrete version. The path (I]wg - -vll); only needs be measurable and bounded. There are no assumptions on its continuity. This finally yields
||w - well & 11200 vall + || roll
(171)
+ ||x - r1loo,10.0)
+
+ max well
[((v) - G(v))lop Lr
(172)
exp(LrLa))
(173)
Now, notice that in our proof, the two CDEs are exchangeable. This means that we immediately get the alternative bound
Ilwee wzll s [Iw0 voll + ||
roll
(174)
|| rlloo,10.0 (1+ LILL G, + max
((()) G( L
(175)
Des?
exp(IGGtr)
(176)
where we recall that
{u QuE R |||||| < ([wwoll [[F(0)) L.) exp(((
D.7 PROOF OF THBOREM B.6
Proof for continuous inputs.
We have
(x) - (x) 2 2T < | 22l | + Bg || - =
(177)
Using Theorem B.5 one gets that
| 2 exp (LL.
-201 L. [. Gu
(178)
31

---

Published as a conference paper at ICLR 2024
where
1 = {u u E ||]|| (Lo + Bb) + [[) exp ((Lo BA)
The quantity 1 [0) is bounded by Lemma B.2 The radius of the ball Ω is thus bounded by constant that does not depend on the value of the network's parameters, but only on the constants defining θ.
We first bound
| =01l
La |0
U2l B Lo 1| U2
(179)
One also has that
G G llo.n0 I
max sEQ
Cb - -bj1 + (2) cx) - -
(180)
Since Q is bounded, (Cg) is bounded and the functions (CA) (z)) are continuous with respect to 2,
they are bounded on Q2 Taking maximas, we get
9
G GGaalllc
max isi5
bb
max sER.ISiS
A A31
(181)
Since z1 is bounded the endpoint of NCDE, gets using Theorem B.5 that
+ 22ll || s I. exp ((BAL))LL.) BuB, B, + (o) ||
F2
(182)
Putting everything together, one gets that
fl (x) - fo)(1))l S Me || -2ll
(183)
q
C Σ ||| A3ll CG Σ [bb -b31
(184)
==1
j=1
+ Cu |U U2l + C. ||0 - 12ll
(185)
with
Me == Lo exp ((BALo)
BUB, B, (0)
(186)
CΛ B exp (A,,
(2)
(187)
s00,151g4
C. LT exp (L. L1z)
max C
(188)
15159
Cu = B, expl L, B Lz)
(189)
C. = B4 exp L. B Lo.
(190)
which concludes our proof.
Proof for discretized inputs. The proof for [,fo (x") fxz((x)) (xP) follows in similar fashion, but with discretization dependant constants. Indeed,
Le, (xP) - Sog (x (xx) P <
+
<
[b
Φ2
1
+ B
2 - =P21 (191)
By Theorem 3.3 we have
D21 1
(Lo BA) -
Lo ) (Lo K
< Lo Lo LuBB B, + Lb
(192)
La BA
32

---

Published as a conference paper at ICLR 2024
To bound the difference
| - -21 we resort to Theorem B.5 which gives us
D. -1- -1 = S exp (A)TT))
=b = + Ly G Guz Goilloonoo
(193)
Remark that the ball Ω depends on the discretization. Indeed, the diameter of this upper bounds the value of zP for all possible controlled ResNets in è and initial values of xP, which depend on the discretization. As for continuous inputs, we have
|| =i1l La 1 U2l B. Lo 1| U2/l
(194)
Now, one also has that
G e Gupllooooo
max sE
c6 - + (+) A3
(195)
and taking maximas again but this time ΩD, one is left with
fl, (a) - fe,(x)) & Mg || &211
(196)
2 [bb]
A Σ A A3l C Σ
bill
(197)
/=1
+ Cu U - U2l + Co |111 - v2l
(198)
with constants
(L) - L,0) (1+ (Lo
Mg Lo (L. B, Lo Lb
(199)
L. BA
C Bab (Lo
max
(A(z)
(200)
=EiS
CB B. exp ( BA)
max C 15:59
(201)
CU == B, exp Lg Lo
(202)
C. B exp(Le LL
(203)
33

---

Published as a conference paper at ICLR 2024
E EXPERIMENTAL DETAILS
E EXPERIMENT 1: ANALYSING THE TRAINING DYNAMICS
Time series. The paths are 4-dimensional sample paths from a IBM with Hurst parameter H 0.7, sampled at 100 equidistant time points in the interval (0, 1]. The paths are generated using the Python package stochastic. We include time as supplementary channel, as advised in Kidger et al. 2202)).
Well-specified model. The output generated from shallow teacher NCDE
dæj Tanh Az + b) dr
initialized with
= Uvy +
with random parameters for which = 3.
Model. We train student NCDE with identical parametrization.. The model is initialized with Pytorch's 's default initialization. In the first and second figures (starting from the left), the model trained for 2000 iterations with Adam. We use the default values for a, 8 and a learning rate of 5 X 10-3 The size of the training sample is set to = 100. For the third figure, we run 25 training runs on the same dataset, for the same model, with 1000 training iterations and record normalized value of the parameters at the end of training.
E.2 EXPERIMENT 2: EFFECT OF THE DISCRETIZATION
Time series. We consider a classification task, in which shallow NCDE is trained to distinguish between dimensional fBMs with respective Hurst parameters H 0.4 (y and H 0.6 (y = 1). Time is included as supplementary channel.
Model. For the first figure, train shallow NCDE classifier with p on n 100 time series sampled at 100 equidistant time points in [0, 1] for 100 iterations with Binary Cross Entropy (BCE) loss. We use Adam with default settings and learning rate of 10-2 For the second and third figures, we fix the initialization and the last linear layer o of the NCDE and perform 300 training runs with Adam (default parameters and learning rate randomly downsample = 50 time series by selecting K = sampling points, which always include and 1. among the 200 initial points. The model is then trained for 600 iterations, and the expected generalization error is computed on 50 test samples. We freeze the initialisation layer and following Marion (2023) to isolate the effect of the discretization on the generalization error.
x 10-4). For each run,
34

---

Published as a conference paper at ICLR 2024
F SUPPLEMENTARY ELEMENTS
F.1 PRACTICAL AND THEORETICAL IMPLICATIONS OF OUR RESULTS
We see three main practical and theoretical implications of our results.
Designing new architectures for NCDEs. Our paper offers performance guarantees and sheds light on the theoretical behaviour of NCDEs. We think that a promising application of our work lies in the design of new architectures for the neural vector field G... In our work, as in most appli- cations of NCDEs, this neural vector field is chosen as a simple MLP. However, recent and rich literature has highlighted the benefits of more structured architectures (for instance through orthogo- nal weights see Li et al. (2019). Our work could help to understand the theoretical implications of such extensions to NCDEs. On a sidenote, we would like to mention the recent article by Chiu et al. 2023), which was unpublished by submission date, and which takes first step in this direction by using& a U-Net as a neural vector field for learning from video data.
Quantifying the impact of sparse sampling. On an applied level, our results can serve as a guide- line to practitioners that use irregular time series. For instance, from our personal experience, MDs often wonder whether they have collected enough longitudinal data in a clinical setting to conduct inference with NCDEs. Our results the sampling bias may help to make informed decision in such a case. Additionally, our results can help a practitioner facing a "decide now or sample more" dilemma, in which agent needs to decide whether she defers decision in order to acquire more information through time-consuming sampling. See for instance (Schäfer & Leser (0020) for a discussion of similar problems.
Connexion to Control Theory. Control Theory can be seen an inversion of the setup of NCDEs as noted by Kidger (022) (see Scction 3.1.6.2). In control theory, the dynamics are fixed (known or unknown) and one tries to optimize the input path. In our setup, the path is fixed and the dynamics are optimized. We think, however, that our results can transfer to control theory: our theoretical framework can indeed accommodate many situations in which one learns with continuous time data. Consider, for instance, a setup with fixed dynamics in which one minimizes a criterion by optimizing over the space of paths by choosing the vector fields of neural SDE see for instance Zhang et al. (202) for a close setup. One could use our results on the continuity of the flow to bound the difference between two outputs- driven by two paths with different vector fields by the difference between the two vector fields, and hence derive an upper bound on the covering number of such a class of predictors. To perform this extension, one would however need to leverage additional tools from stochastic calculus and rough analysis.
F.2 COMPARISON WITH EARLIER RESULTS BY CHEN ET AL. (020A)
Questions about the comparison of our results with earlier results by Chen et al. (2020a) were raised during the reviewing process of this article. We provide a thorough discussion of these questions in this section. The notations used are the ones by the authors of the aforementioned article and we refer to the AISTATS 2020 version of their work.
In their work, Chen al. (2020a) consider so-called vanilla RNN which embed for every individual i i= -,..., 71. a time series (f;,))==1 through
h on (U + Wz,.).
(204)
Here, U E dh xda and W E Rd dd are learnable parameters. This latent state is then in turn used produce prediction about the true label (zz, E) i by computing
Vi.t = o, (Vha.)
(205)
where V E Xdd is again learnable parameter. In their work, the authors derive generalization bounds by leveraging similar techniques to ours. Indeed, they first proceed to show that vanilla RNN Lipschitz with respect to their parameters (see their Lemma 2, in Section 3 page 4 left column). This result is similar to our Theorem[B.6, They then proceed to derive the covering number of vanilla RNN (see their Lemma 3, Section 3, page 5 left column). This is identical to our Proposition 4.2
35

---

Published as a conference paper at ICLR 2024
Finally, in our article, Chen et al. are able to derive the Rademacher complexity using Dudley's entropy integral.
Let us discuss the generalization bound in two works. The first kind of generalization bound Chen et al. (2020a) scales with the number of parameters their model equal to
v dn + d3. + dady
(206)
which expected when using Lipschitz based complexity measures (see their Theorem 2). This is similar to our results in Theorem 4.1 where the complexity term also scales with the number of parameters.
In a second time, Chen et al. (2020a) prove more refined generalization bounds by controlling the (2, 1) norm of the weight matrices (see their Section 4). They obtain two Theorems, yielding a scaling either in
log Vd
(207)
or
Vdlogd d
(208)
using again the original notations. This bounds are stronger than the previous ones, and than our bounds, since their involve the 4-th root of the number of parameters. They do, however, require stronger assumptions on the norm of the parameters. We leave an application of this kind of results to our setup to future work.
F.3 COMPARAISON WITH EARLIER RESULTS BY GOLOWICH ET AL. 2018
Golowich et al. (2018) deploy new techniques, based on a refined control of the Schatten norms of the weight matrices, that allows to obtain depth-independent bounds. The authors consider deep non-residual neural networks of the form
I HHOO Woooo WL-10-.. 00 Wi(x)
(209)
where (W&) km are learnable parameters. The bounds they obtain scale as
Ir log [I//TS)
(210)
1/4
where 11 is an upper bound on the product of Froebenius norms of the weight matrices and lower bound the product of the spectral norms of the weight matrices. To obtain depth- independent bound, the authors require these two quantities to be depth independent. Interestingly, observe that these generalization bounds come at the price of worsened sample complexity (in the sense of dependence on the number of samples). The interested reader may also consider the discussion of this point provided Marion (2023).
To discuss these bounds and to avoid any confusion, we first need to make clear distinction between the depth and the length of our NCDE predictor:
The length of controlled ResNet (or NCDE with piecewise constant path embedding) refers the number sampling times and is equal to K. In ResNets (see Appendix F.4). this quantity is usually referred to as depth, hence a possible confusion. Contrarily to ResNets, this parameter is data dependant i.e. it depends on the number of sampling times of the time series at hand.
The depth of our NCDE refers to the depth of the neural vector field G which is assumed in our work to be standard MLP.
Our bounds are not depth independent since increasing the depth of the vector field G worsen our generalization bounds which scale with the square root of the number of parameters.
We conjecture that one could apply the bounds obtained by Golowich et al. (2018) to provide refined generalization bound on the complexity stemming from the neural vector field G However, believe that since in practice NCDEs used with neural vector fields for which g S 3, trading seppt-inddeendencee for a worsened convergence speed the number of samples will most likely not benefice our bounds.
36

---

Published as a conference paper at ICLR 2024
F.4 COMPARAISON OF RESNETS AND NCDES
vanilla ResNet architecture can be described as an embedding of x E IRd through
ho = Ur E IR
(211)
1
h h4 + L
W Wa (h+) for all & ......L
(212)
= Φ hT.
(213)
where U Rdxp, (=) E Rexp, G Rp - Rp is neural network parametrized by v and ý is the final prediction of the ResNet. The scaling factor is usually chosen between 1/2 and 1.
Consider now the controlled ResNet architecture, which can be described as mentioned earlier as an embedding of time series (Nu,..., Xtx) E (x+((+1) through
2g Uzo R"
(214)
1 = 2か + Go(zz) for all O....,K--
(215)
ÿ + TT2 2k
(216)
where we have simplified, for the sake of the analogy, the initialization to be linear initialization layer.
One can see that, from an architectural point of view, the layer-dependant weights (W of the ResNet play similar role the increments of the time series (Ax,, );² However, in ResNets these weights are learned, whereas in controlled ResNets and NCDEs they are given through the embedded time series. Also remark that while the weights of ResNet are shared between all inputs (i.e. two inputs and x' are embedded using the same weights), the increments of time series are, by definition, individual specific. This allows NCDEs to have individual-specifee dynamics for latent state.
This allows to compare experimental findings with the ones of Marion (202)), who observes that the generalization error correlates with the Lipschitz constant of the weights defined as
max OSkST
[ W
In our setting, this translates to the fact that the generalization error is correlated with the average absolute path variation
Di X X12
D.i
max ||
Remark that we have to consider the average over individuals, since the increments are individual specific. Since these inerements are not trained, and since we make the assumption that they are Our code is
bounded by Lzz [D], we should also observe correlation with L D see Figure 4 available at this link
37


Reviews:
[{"review_id": "IF7l5BBHXs", "reviewer": "Reviewer_c9Ry", "summary": "This paper makes a first-of-its-kind attempt at bounding the generalization error of NCDE.", "questions": "Q1.\nCan the authors explain why there is an exponential degradation of the bound's dependence on width as compared to the result in Chen et al. (2019a), which seems to be the closest literature to this? Can you argue that this degradation is inevitable? \n\nQ2.\nCan the authors point out as to what is the surprise or the non-triviality about the dependencies that are visible in the bound given in Theorem 4.1?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The analysis done is thorough and it is definitely an achievement to have setup the formalism required to compute a generalization bound like this.", "weaknesses": "In Section 4, Theorem 4.1 the upper bound for the generalization error varies with both the width and depth of the neural network and unlike Chen et al. (2019a), where the growth w.r.t the depth and width is logarithmic, in this case it’s polynomial. This exponential blow-up in the value of bound makes the whole exercise somewhat underwhelming. \n\nAlso its worth mentioning that the discretization dependence in Theorem 4.1 is through the constants K1^D and K2^D\n - and these two critical quantities don't seem defined anywhere in the main paper! This makes the theorem highly opaque! \n\nThe entire point why a generalization error bound is interesting is because it reveals the generalization error to be independent of some natural size parameter of the system. Like, most often the baseline achievement of a Rademacher bound for a neural system is to show that at a fixed depth and norm bound on the weights involved the generalization is width independent. In this bound, in this work, there is absolutely no such thing happening - the bound in Theorem 4.1 is worsening with every such parameter with which naively one would have expected worsening to happen! \n\nThe numerical Illustrations also seem incomplete in the absence of any demonstration of how well the bounds hold up in the cases mentioned there. In the bare minimum, in Figure 4 kind of experiments one would have expected to see a scatter plot of the theoretical bound and the true generalization error as the sampling gap, time and avg. max path variation is changed. Such a scatter plot would have revealed if there is at all any correlation between the theoretical bound and the truth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "knqjQKsB02", "reviewer": "Reviewer_QrUR", "summary": "The manuscript presents a theoretical investigation into the generalization and approximation capacities of neural controlled differential equations (NCDEs). It aims to answer critical questions about the generalization and approximation capabilities of NCDEs and how these are affected by irregular sampling. The theoretical contributions are backed by experiments. The authors provide a rigorous mathematical framework to offer insights into the capabilities and limitations of NCDEs.", "questions": "Could you clarify how your theoretical contributions on the approximation part diverge from existing works on the approximation ability of NCDE like [1] or [2]?\n\n[1] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02435, 2022.\n\n[2] Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696–6707, 2020. \n\nWhat are the potential practical applications that could benefit from the theoretical insights provided in this paper? Could these insights lead to more efficient or accurate NCDE models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper stands out for its focus on providing a theoretical framework for NCDEs. This can serve as a foundational work for further investigations.\nQuality:\nThe mathematical rigor of the paper is high, with well-laid-out proofs and theoretical discussions.\nClarity:\nDespite the mathematical complexity, the paper is organized in a way that progressively builds up the reader's understanding of the topic.\nSignificance:\nThe theoretical insights offered could be highly impactful, providing guidelines for both practical applications and future research in machine learning and control theory.", "weaknesses": "In Abstract: ``However, no theoretical analysis of their performance has been provided yet''. This is not true. \n\nThe paper could do a better job of situating its contributions within the existing literature, especially to clarify its novelty.\n\nWhile the focus is theoretical, some discussion on the practical implications of these theories could make the paper more well-rounded.\n\nThere are some grammar/format issues, such as, ``since y is the sum of a linear transformation of the endpoint of a CDE an a noise term ε bounded by …’’", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NaLjMivEhg", "reviewer": "Reviewer_V1D4", "summary": "This paper studies the generalization bound for neural-controlled differential equations(NCDE). Given an NCDE that minimizes the empirical risk from some sampled data, the author first derives a generalization bound between the empirical and expected risk given the sampling process for the input time series, and then an upper bound on the difference between the expected risk and the optimal risk from the ground truth predictor.", "questions": "1. Any assumption on the continuous $x$, or its distribution, if it is random.\n2. $|D|$ appears in Remark 3.5, but defined much later.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. novel generalization bound for neural-controlled differential equations.\n2. an upper bound on the excessive risk from sampling-induced bias and approximation bias.", "weaknesses": "1. The presentation in Theorem 4.1 could be improved, as it is hard to see how the bound explicitly depends on depth $q$, and discretization $D$. Right now it is hidden in the constant $C_q$ and $K_1^D, K_2^D$. \n2. The experiments seem very preliminary: the sampled time series only have $K=5$ points. I am not sure how interesting it is to investigate this regime as the generalization error would be poor with this few samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)