Title: Generalization bounds for neural ordinary differential equations and deep residual networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: residual neural networks

neural ODEs

generalization bound
Abstract: Neural ordinary differential equations (neural ODEs) are a popular family of continuous-depth deep learning models.  In this work, we consider a large family of parameterized ODEs with continuous-in-time parameters, which include time-dependent neural ODEs.  We derive a generalization bound for this class by a Lipschitz-based argument. By leveraging the analogy between neural ODEs and deep residual networks, our approach yields in particular a generalization bound for a class of deep residual networks. The bound involves the magnitude of the difference between successive weight matrices. We illustrate numerically how this quantity affects the generalization capability of neural networks.

Generalization bounds for neural ordinary differential equations and deep residual networks

Abstract

Neural ordinary differential equations (neural ODEs) are a popular family of continuous-depth deep learning models. In this work, we consider a large family of parameterized ODEs with continuous-in-time parameters, which include timedependent neural ODEs. We derive a generalization bound for this class by a Lipschitz-based argument. By leveraging the analogy between neural ODEs and deep residual networks, our approach yields in particular a generalization bound for a class of deep residual networks. The bound involves the magnitude of the difference between successive weight matrices. We illustrate numerically how this quantity affects the generalization capability of neural networks.

1 Introduction

Neural ordinary differential equations (neural ODEs, Chen et al., 2018) are a flexible family of neural networks used in particular to model continuous-time phenomena. Along with variants such as neural stochastic differential equations (neural SDEs, Tzen and Raginsky, 2019) and neural controlled differential equations (Kidger et al., 2020), they have been used in diverse fields such as pharmokinetics (Lu et al., 2021; Qian et al., 2021), finance (Gierjatowicz et al., 2020), and transportation (Zhou et al., 2021). We refer to Massaroli et al. (2020) for a self-contained introduction to this class of models.

Despite their empirical success, the statistical properties of neural ODEs have not yet been fully investigated. What is more, neural ODEs can be thought of as the infinite-depth limit of (properly scaled) residual neural networks (He et al., 2016a), a connection made by, e.g., E (2017); Haber and Ruthotto (2017); Lu et al. (2017). Since standard measures of statistical complexity of neural networks grow with depth (see, e.g., Bartlett et al., 2019), it is unclear why infinite-depth models, including neural ODEs, should enjoy favorable generalization properties.

To better understand this phenomenon, our goal in this paper is to study the statistical properties of a class of time-dependent neural ODEs that write

where W t ∈ R d × d is a weight matrix that depends on the time index t , and σ : R → R is an activation function applied component-wise. Time-dependent neural ODEs were first introduced by Massaroli et al. (2020) and generalize time-independent neural ODEs

as formulated in Chen et al. (2018), where W ∈ R d × d now denotes a weight matrix independent of t . There are two crucial reasons to consider time-dependent neural ODEs rather than the more

restrictive class of time-independent neural ODEs. On the one hand, the time-dependent formulation is more flexible, leading to competitive results on image classification tasks (Queiruga et al., 2020, 2021). As a consequence, obtaining generalization guarantees for this family of models is a valuable endeavor by itself. On the other hand, time dependence is required for the correspondence with general residual neural networks to hold. More precisely, the time-dependent neural ODE (1) is the limit, when the depth L goes to infinity, of the deep residual network

where ( W k ) 1 glyph[lessorequalslant] k glyph[lessorequalslant] L ∈ R d × d are weight matrices and σ is still an activation function. We refer to Marion et al. (2022, 2023); Sander et al. (2022); Thorpe and van Gennip (2022) for statements that make precise under what conditions and in which sense this limit holds, as well as its consequences for learning. These two key reasons compel us to consider the class of time-dependent ODEs (1) for our statistical study, which in turn will inform us on the properties of the models (2) and (3).

In fact, we extend our study to the larger class of parameterized ODEs , which we define as the mapping from x ∈ R d to the value at time t = 1 of the solution of the initial value problem

where H t is the variable of the ODE, θ i are functions from [0 , 1] into R that parameterize the ODE, and f i are fixed functions from R d into R d . Time-dependent neural ODEs (1) are obtained by setting a specific entrywise form for the functions f i in (4).

Since the parameters θ i belong to an infinite-dimensional space, in practice they need to be approximated in a finite-dimensional basis of functions. For example, the residual neural networks (3) can be seen as an approximation of the neural ODEs (1) on a piecewise-constant basis of function. But more complex choices are possible, such as B-splines (Yu et al., 2022). However, the formulation (4) is agnostic from the choice of finite-dimensional approximation. This more abstract point of view is fruitful to derive generalization bounds, for at least two reasons. First, the statistical properties of the parameterized ODEs (4) only depend on the characteristics of the functions θ i and not on the specifics of the approximation scheme, so it is more natural and convenient to study them at the continuous level. Second, their properties can then be transferred to any specific discretization, such as the deep residual networks (3), resulting in generalization bounds for the latter.

Regarding the characteristics of the functions θ i , we make the structural assumption that they are Lipschitz-continuous and uniformly bounded. This is a natural assumption to ensure that the initial value problem (4) has a unique solution in the usual sense of the Picard-Lindelöf theorem (Arnold, 1992). Remarkably, this assumption on the parameters also enables us to obtain statistical guarantees despite the fact that we are working with an infinite-dimensional set of parameters.

Contributions. We provide a generalization bound for the large class of parameterized ODEs (4), which include time-dependent and time-independent neural ODEs (1) and (2). To the best of our knowledge, this is the first available bound for neural ODEs in supervised learning. By leveraging on the connection between (time-dependent) neural ODEs and deep residual networks, our approach allows us to provide a depth-independent generalization bound for the class of deep residual networks (3). The bound is precisely compared with earlier results. Our bound depends in particular on the magnitude of the difference between successive weight matrices, which is, to our knowledge, a novel way of controlling the statistical complexity of neural networks. Numerical illustration is provided to show the relationship between this quantity and the generalization ability of neural networks.

Organization of the paper. Section 2 presents additional related work. In Section 3, we specify our class of parameterized ODEs, before stating the generalization bound for this class and for neural ODEs as a corollary. The generalization bound for residual networks is presented in Section 4 and compared to other bounds, before some numerical illustration. Section 5 concludes the paper. The proof technique is discussed in the main paper, but the core of the proofs is relegated to the Appendix.

2 Related work

Hybridizing deep learning and differential equations. The fields of deep learning and dynamical systems have recently benefited from sustained cross-fertilization. On the one hand, a large line of work is aimed at modeling complex continuous-time phenomena by developing specialized neural architectures. This family includes neural ODEs, but also physics-informed neural networks (Raissi et al., 2019), neural operators (Li et al., 2021) and neural flows (Biloš et al., 2021). On the other hand, successful recent advances in deep learning, such as diffusion models, are theoretically supported by ideas from differential equations (Huang et al., 2021).

Generalization for continuous-time neural networks. Obtaining statistical guarantees for continuous-time neural networks has been the topic of a few recent works. For example, Fermanian et al. (2021) consider recurrent neural networks (RNNs), a family of neural networks handling time series, which is therefore a different setup from our work that focuses on vector-valued inputs. These authors show that a class of continuous-time RNNs can be written as input-driven ODEs, which are then proved to belong to a family of kernel methods, which entails a generalization bound. Lim et al. (2021) also show a generalization bound for ODE-like RNNs, and argue that adding stochasticity (that is, replacing ODEs with SDEs) helps with generalization. Taking another point of view, Yin et al. (2021) tackle the separate (although related) question of generalization when doing transfer learning across multiple environments. They propose a neural ODE model and provide a generalization bound in the case of a linear activation function. Closer to our setting, Hanson and Raginsky (2022) show a generalization bound for parameterized ODEs for manifold learning, which applies in particular for neural ODEs. Their proof technique bears similarities with ours, but the model and task differ from our approach. In particular, they consider stacked time-independent parameterized ODEs, while we are interested in a time-dependent formulation. Furthermore, these authors do not discuss the connection with residual networks.

Lipschitz-based generalization bounds for deep neural networks. From a high-level perspective, our proof technique is similar to previous works (Bartlett et al., 2017; Neyshabur et al., 2018) that show generalization bounds for deep neural networks, which scale at most polynomially with depth. More precisely, these authors show that the network satisfies some Lipschitz continuity property (either with respect to the input or to the parameters), then exploit results on the statistical complexity of Lipschitz function classes. Under stronger norm constraints, these bounds can even be made depthindependent (Golowich et al., 2018). However, their approach differs from ours insofar as we consider neural ODEs and the associated family of deep neural networks, whereas they are solely interested in finite-depth neural networks. As a consequence, their hypotheses on the class of neural networks differ from ours. Section 4 develops a more thorough comparison. Similar Lipschitz-based techniques have also been applied to obtain generalization bounds for deep equilibrium networks (Pabbaraju et al., 2021). Going beyond statistical guarantees, Béthune et al. (2022) study approximation and robustness properties of Lipschitz neural networks.

3 Generalization bounds for parameterized ODEs

We start by recalling the usual supervised learning setup and introduce some notation in Section 3.1, before presenting our parameterized ODE model and the associated generalization bound in Section 3.2. We then apply the bound to the specific case of time-invariant neural ODEs in Section 3.3.

3.1 Learning procedure

We place ourselves in a supervised learning setting. Let us introduce the notation that are used throughout the paper (up to and including Section 4.1). The input data is a sample of n i.i.d. pairs ( x i , y i ) with the same distribution as some generic pair ( x, y ) , where x (resp. y ) takes its values in some bounded ball X = B (0 , R X ) (resp. Y = B (0 , R Y ) ) of R d , for some R X , R Y > 0 . This setting encompasses regression but also classification tasks by (one-hot) encoding labels in R d . Note that we assume for simplicity that the input and output have the same dimension, but our analysis easily extends to the case where they have different dimensions by adding (parameterized) projections at the beginning or at the end of our model. Given a parameterized class of models F Θ = { F θ , θ ∈ Θ } , the parameter θ is fitted by empirical risk minimization using a loss function glyph[lscript] : R d × R d → R + that

we assume to be Lipschitz with respect to its first argument, with a Lipschitz constant K glyph[lscript] > 0 . In the following, we write for the sake of concision that such a function is K glyph[lscript] -Lipschitz. We also assume that glyph[lscript] ( x, x ) = 0 for all x ∈ R d . The theoretical and empirical risks are respectively defined, for any θ ∈ Θ , by

where the expectation E is evaluated with respect to the distribution of ( x, y ) . Letting ̂ θ n a minimizer of the empirical risk, the generalization problem consists in providing an upper bound on the difference R ( ̂ θ n ) -̂ R n ( ̂ θ n ) .

3.2 Generalization bound

Model. We start by making more precise the parameterized ODE model introduced in Section 1. The setup presented here can easily be specialized to the case of neural ODEs, as we will see in Section 3.3. Let f 1 , . . . , f m : R d → R d be fixed K f -Lipschitz functions for some K f > 0 . Denote by M their supremum on X (which is finite since these functions are continuous). The parameterized ODE F θ is defined by the following initial value problem that maps some x ∈ R d to F θ ( x ) ∈ R d :

where the parameter θ = ( θ 1 , . . . , θ m ) is a function from [0 , 1] to R m . We have to impose constraints on θ for the model F θ to be well-defined. To this aim, we endow (essentially bounded) functions from [0 , 1] to R m with the following (1 , ∞ ) -norm

We can now define the set of parameters

for some R Θ > 0 and K Θ glyph[greaterorequalslant] 0 . Then, for θ ∈ Θ , the following Proposition, which is a consequence of the Picard-Lindelöf Theorem, shows that the mapping x ↦→ F θ ( x ) is well-defined.

Proposition 1 (Well-posedness of the parameterized ODE) . For θ ∈ Θ and x ∈ R d , there exists a unique solution to the initial value problem (5) .

An immediate consequence of Proposition 1 is that it is legitimate to consider F Θ = { F θ , θ ∈ Θ } for our model class.

When K Θ = 0 , the parameter space Θ is finite-dimensional since each θ i is constant. This setting corresponds to the time-independent neural ODEs of Chen et al. (2018). In this case, the norm (6) reduces to the ‖ · ‖ 1 norm over R m . Note that, to fit exactly the formulation of Chen et al. (2018), the time t can be added as a variable of the functions f i , which amounts to adding a new coordinate to H t . This does not change the subsequent analysis. In the richer time-dependent case where K Θ > 0 , the set Θ belongs to an infinite-dimensional space and therefore, in practice, θ i is approximated in a finite basis of functions, such as Fourier series, Chebyshev polynomials, and splines. We refer to Massaroli et al. (2020) for a more detailed discussion, including formulations of the backpropagation algorithm (a.k.a. the adjoint method) in this setting.

Note that we consider the case where the dynamics at time t are linear with respect to the parameter θ i ( t ) . Nevertheless, we emphasize that the mapping x ↦→ F θ ( x ) remains a highly non-linear function of each θ i ( t ) . To fix ideas, this setting can be seen as analogue to working with pre-activation residual networks instead of post-activation (see He et al., 2016b, for definitions of the terminology), which is a mild modification.

Statistical analysis Since Θ is a subset of an infinite-dimensional space, complexity measures based on the number of parameters cannot be used. Instead, our approach is to resort to Lipschitzbased complexity measures. More precisely, to bound the complexity of our model class, we propose two building blocks: we first show that the model F θ is Lipschitz-continuous with respect to its parameters θ . This allows us to bound the complexity of the model class depending on the complexity of the parameter class. In a second step, we assess the complexity of the class of parameters itself.

Starting with our first step, we show the following estimates for our class of parameterized ODEs. Here and in the following, ‖ · ‖ denotes the glyph[lscript] 2 norm over R d .

Proposition 2 (The parameterized ODE is bounded and Lipschitz) . Let θ and ˜ θ ∈ Θ . Then, for any x ∈ X , and

The proof, given in the Appendix, makes extensive use of Grönwall's inequality (Pachpatte and Ames, 1997), a standard tool to obtain estimates in the theory of ODEs, in order to bound the magnitude of the solution H t of (5).

The next step is to assess the magnitude of the covering number of Θ . Recall that, for ε > 0 , the ε -covering number of a metric space is the number of balls of radius ε needed to completely cover the space, with possible overlaps. More formally, considering a metric space M and denoting by B ( x, ε ) the ball of radius ε centered at x ∈ M , the ε -covering number of M is equal to inf { n glyph[greaterorequalslant] 1 |∃ x 1 , . . . , x n ∈ M , M ⊆ ⋃ n i =1 B ( x i , ε ) } .

Proposition 3 (Covering number of the ODE parameter class) . For ε > 0 , let N ( ε ) be the ε -covering number of Θ endowed with the distance associated to the (1 , ∞ ) -norm (6) . Then

Proposition 3 is a consequence of a classical result, see, e.g., Kolmogorov and Tikhomirov (1959, example 3 of paragraph 2). A self-contained proof is given in the Appendix for completeness. We also refer to Gottlieb et al. (2017) for more general results on covering numbers of Lipschitz functions.

The two propositions above and an ε -net argument allow to prove the first main result of our paper (where we recall that the notations are defined in Section 3.1).

Theorem 1 (Generalization bound for parameterized ODEs) . Consider the class of parameterized ODEs F Θ = { F θ , θ ∈ Θ } , where F θ is given by (5) and Θ by (7) . Let δ > 0 .

Then, for n glyph[greaterorequalslant] 9 max( m -2 R -2 Θ , 1) , with probability at least 1 -δ , where B is a constant depending on K glyph[lscript] , K f , R Θ , R X , R Y , M . More precisely,

Three terms appear in our upper bound of R ( ̂ θ n ) -̂ R n ( ̂ θ n ) . The first and the third ones are classical (see, e.g. Bach, 2023, Sections 4.4 and 4.5). On the contrary, the second term is more surprising with its convergence rate in O ( n -1 / 4 ) . This slower convergence rate is due to the fact that the space of parameters is infinite-dimensional. In particular, for K Θ = 0 , corresponding to a finitedimensional space of parameters, we recover the usual O ( n -1 / 2 ) convergence rate, however at the cost of considering a much more restrictive class of models. Finally, it is noteworthy that the dimensionality appearing in the bound is not the input dimension d but the number of mappings m .

Note that this result is general and may be applied in a number of contexts that go beyond deep learning, as long as the instantaneous dependence of the ODE dynamics to the parameters is linear. One such example is the predator-prey model, describing the evolution of two populations of animals, which reads dx t = x t ( α -βy t ) dt and dy t = -y t ( γ -δx t ) dt , where x t and y t are real-valued variables and α , β , γ and δ are model parameters. This ODE falls into the framework of this section,

if one were to estimate the parameters by empirical risk minimization. We refer to Deuflhard and Röblitz (2015, section 3) for other examples of parameterized biological ODE dynamics and methods for parameter identification.

Nevertheless, for the sake of brevity, we focus on applications of this result to deep learning, and more precisely to neural ODEs, which is the topic of the next section.

3.3 Application to neural ODEs

As explained in Section 1, parameterized ODEs include both time-dependent and time-independent neural ODEs. Since the time-independent model is more common in practice, we develop this case here and leave the time-dependent case to the reader. We thus consider the following neural ODE:

where W ∈ R d × d is a weight matrix, and σ : R → R is an activation function applied componentwise. We assume σ to be K σ -Lipschitz for some K σ > 0 . This assumption is satisfied by all common activation functions. To put the model in the form of Section 3.2, denote e 1 , . . . , e d the canonical basis of R d . Then the dynamics (8) can be reformulated as

where σ ij ( x ) = σ ( x j ) e i . Each σ ij is itself K σ -Lipschitz, hence we fall in the framework of Section 3.2. In other words, the functions f i of our general parameterized ODE model form a shallow neural network with pre-activation. Denote by ‖ W ‖ 1 , 1 the sum of the absolute values of the elements of W . We consider the following set of parameters, which echoes the set Θ of Section 3.2:

for some R W > 0 . We can then state the following result as a consequence of Theorem 1.

Corollary 1 (Generalization bound for neural ODEs) . Consider the class of neural ODEs F W = { F W , W ∈ W} , where F W is given by (8) and W by (9) . Let δ > 0 .

Then, for n glyph[greaterorequalslant] 9 R -1 W max( d -4 R -1 W , 1) , with probability at least 1 -δ ,

where B is a constant depending on K glyph[lscript] , K σ , R W , R X , R Y , M . More precisely,

Note that the term in O ( n -1 / 4 ) from Theorem 1 is now absent. Since we consider a time-independent model, we are left with the other two terms, recovering a standard O ( n -1 / 2 ) convergence rate.

4 Generalization bounds for deep residual networks

As highlighted in Section 1, there is a strong connection between neural ODEs and discrete residual neural networks. The previous study of the continuous case in Section 3 paves the way for deriving a generalization bound in the discrete setting of residual neural networks, which is of great interest given the pervasiveness of this architecture in modern deep learning.

We begin by presenting our model and result in Section 4.1, before detailing the comparison of our approach with other papers in Section 4.2 and giving some numerical illustration in Section 4.3.

4.1 Model and generalization bound

Model. We consider the following class of deep residual networks:

H

0

=

x

where the parameter W = ( W k ) 1 glyph[lessorequalslant] k glyph[lessorequalslant] L ∈ R L × d × d is a set of weight matrices and σ is still a K σ -Lipschitz activation function. To emphasize that W is here a third-order tensor, as opposed to the case of time-invariant neural ODEs in Section 3.3, where W was a matrix, we denote it with a bold notation. We also assume in the following that σ (0) = 0 . This assumption could be alleviated at the cost of additional technicalities. Owing to the 1 / L scaling factor, the deep limit of this residual network is a (time-dependent) neural ODE of the form studied in Section 3. We refer to Marion et al. (2022) for further discussion on the link between scaling factors and deep limits. We simply note that this scaling factor is not common practice, but preliminary experiments show it does not hurt performance and can even improve performance in a weight-tied setting (Sander et al., 2022). The space of parameters is endowed with the following (1 , 1 , ∞ ) -norm

Also denoting ‖· ‖ ∞ the element-wise maximum norm for a matrix, we consider the class of matrices

for some R W > 0 and K W glyph[greaterorequalslant] 0 , which is a discrete analogue of the set Θ defined by (7).

In particular, the upper bound on the difference between successive weight matrices is to our knowledge a novel way of constraining the parameters of a neural network. It corresponds to the discretization of the Lipschitz continuity of the parameters introduced in (7). By analogy, we refer to it as a constraint on the Lipschitz constant of the weights. Note that, for standard initialization schemes, the difference between two successive matrices is of the order O (1) and not O (1 /L ) , or, in other words, K W scales as O ( L ) . This dependence of K W on L can be lifted by adding correlations across layers at initialization. For instance, one can take, for k ∈ { 1 , . . . , L } and i, j ∈ { 1 , . . . , d } , W k,i,j = 1 √ d f i,j ( k L ) , where f i,j is a smooth function, for example a Gaussian process with the RBF kernel. Such a non-i.i.d. initialization scheme is necessary for the correspondence between deep residual networks and neural ODEs to hold (Marion et al., 2022). Furthermore, Sander et al. (2022) prove that, with this initialization scheme, the constraint on the Lipschitz constant also holds for the trained network, with K W independent of L . Finally, we emphasize that the following developments also hold in the case where K W depends on L (see also Section 4.2 for a related discussion).

Statistical analysis. At first sight, a reasonable strategy would be to bound the distance between the model (10) and its limit L →∞ that is a parameterized ODE, then apply Theorem 1. This strategy is straightforward, but comes at the cost of an additional O (1 /L ) term in the generalization bound, as a consequence of the discretization error between the discrete iterations (10) and their continuous limit. For example, we refer to Fermanian et al. (2021) where this strategy is used to prove a generalization bound for discrete RNNs and where this additional error term is incurred. We follow another way by mimicking all the proof with a finite L . This is a longer approach but it yields a sharper result since we avoid the O (1 /L ) discretization error. The proof structure is similar to Section 3: the following two Propositions are the discrete counterparts of Propositions 2 and 3.

Proposition 4 (The residual network is bounded and Lipschitz) . Let W and ˜ W ∈ W . Then, for any x ∈ X ,

Proposition 5 (Covering number of the residual network parameter class) . Let N ( ε ) be the covering number of W endowed with the distance associated to the (1 , 1 , ∞ ) -norm (11) . Then

The proof of Proposition 4 is a discrete analogous of Proposition 2. On the other hand, Proposition 5 can be proven as a consequence of Proposition 3, by showing the existence of an injective isometry from W into a set of the form (7). Equipped with these two propositions, we are now ready to state the generalization bound for our class of residual neural networks.

Theorem 2 (Generalization bound for deep residual networks) . Consider the class of neural networks F W = { F W , W ∈ W} , where F W is given by (10) and W by (12) . Let δ > 0 .

Then, for n glyph[greaterorequalslant] 9 R -1 W max( d -4 R -1 W , 1) , with probability at least 1 -δ , where B is a constant depending on K glyph[lscript] , K σ , R W , R X , R Y . More precisely,

We emphasize that this result is non-asymptotic and valid for any width d and depth L . Furthermore, the depth L does not appear in the upper bound (13). This should not surprise the reader since Theorem 1 can be seen as the deep limit L →∞ of this result, hence we expect that our bound remains finite when L →∞ (otherwise the bound of Theorem 1 would be infinite). However, L appears as a scaling factor in the definition of the neural network (10) and of the class of parameters (12). This is crucial for the depth independence to hold, as we will comment further on in the next section.

Furthermore, the depth independence comes at the price of a O ( n -1 / 4 ) convergence rate. Note that, by taking K W = 0 , we obtain a generalization bound for weight-tied neural networks with a faster convergence rate in n , since the term in O ( n -1 / 4 ) vanishes.

4.2 Comparison with other bounds

As announced in Section 2, we now compare Theorem 2 with the results of Bartlett et al. (2017) and Golowich et al. (2018). Beginning by Bartlett et al. (2017), we first state a slightly weaker version of their result to match our notations and facilitate comparison.

Corollary 2 (corollary of Theorem 1.1 of Bartlett et al. (2017)) . Consider the class of neural networks F ˜ W = { F W , W ∈ ˜ W} , where F W is given by (10) and ˜ W = { W ∈ R L × d × d , ‖ W ‖ 1 , 1 , ∞ glyph[lessorequalslant] R W } . Assume that L glyph[greaterorequalslant] R W and K σ = 1 , and let γ, δ > 0 . Consider ( x, y ) , ( x 1 , y 1 ) , . . . , ( x n , y n ) drawn i.i.d. from any probability distribution over R d 1 , . . . d such that a.s. x glyph[lessorequalslant] R .

Then, with probability at least 1 -δ , for every W ∈ ˜ W , glyph[negationslash]

×{ } ‖ ‖ X

glyph[negationslash]

We first note that the setting is slightly different from ours: they consider a large margin predictor for a multi-class classification problem, whereas we consider a general Lipschitz-continuous loss glyph[lscript] . This being said, the model class is identical to ours, except for one notable difference: the constraint on the Lipschitz constant of the weights appearing in equation (12) is not required here.

Comparing (13) and (14), we see that our bound enjoys a better dependence on the depth L but a worse dependence on the width d . Regarding the depth, our bound (13) does not depend on L , whereas the bound (14) scales as O ( √ L ) . This comes from the fact that we consider a smaller set of parameters (12), by adding the constraint on the Lipschitz norm of the weights. This constraint

allows us to control the complexity of our class of neural networks independently of depth, as long as K W is independent of L . If K W scales as O ( L ) , which is the case for i.i.d. initialization schemes, our result also features a scaling in O ( √ L ) . As for the width, Bartlett et al. (2017) achieve a better dependence by a subtle covering numbers argument that takes into account the geometry induced by matrix norms. Since our paper focuses on a depth-wise analysis by leveraging the similarity between residual networks and their infinite-depth counterpart, improving the scaling of our bound with width is left for future work. Finally, note that both bounds have a similar exponential dependence in R W .

As for Golowich et al. (2018), they consider non-residual neural networks of the form x ↦→ M L σ ( M L -1 σ ( . . . σ ( M 1 x ))) . These authors show that the generalization error of this class scales as

where Π F is an upper-bound on the product of the Frobenius norms ∏ L k =1 ‖ M k ‖ F and π S is a lower-bound on the product of the spectral norms ∏ L k =1 ‖ M k ‖ . Under the assumption that both Π F and Π F / π S are bounded independently of L , their bound is indeed depth-independent, similarly to ours. Interestingly, as ours, the bound presents a O ( n -1 / 4 ) convergence rate instead of the more usual O ( n -1 / 2 ) . However, the assumption that Π F is bounded independently of L does not hold in our residual setting, since we have M k = I + 1 L W k and thus we can lower-bound

In our setting, it is a totally different assumption, the constraint that two successive weight matrices should be close to one another, which allows us to derive depth-independent bounds.

4.3 Numerical illustration

The bound of Theorem 2 features two quantities that depend on the class of neural networks, namely R W that bounds a norm of the weight matrices and K W that bounds the maximum difference between two successive weight matrices, i.e. the Lipschitz constant of the weights. The first one belongs to the larger class of norm-based bounds that has been extensively studied (see, e.g., Neyshabur et al., 2015). We are therefore interested in getting a better understanding of the role of the second quantity, which is much less common, in the generalization ability of deep residual networks.

To this aim, we train deep residual networks (10) (of width d = 30 and depth L = 1000 ) on MNIST. We prepend the network with an initial weight matrix to project the data x from dimension 768 to dimension 30 , and similarly postpend it with another matrix to project the output F W ( x ) into dimension 10 (i.e. the number of classes in MNIST). Finally, we consider two training settings: either the initial and final matrices are trained, or they are fixed random projections. We use the initialization scheme outlined in Section 4.1. Further experimental details are postponed to the Appendix.

We report in Figure 1a the generalization gap of the trained networks, that is, the difference between the test and train errors (in terms of cross entropy loss), as a function of the maximum Lipschitz constant of the weights sup 0 glyph[lessorequalslant] k glyph[lessorequalslant] L -1 ( ‖ W k +1 -W k ‖ ∞ ) . We observe a positive correlation between these two quantities. To further analyze the relationship between the Lipschitz constant of the weights and the generalization gap, we then add the penalization term λ · ( ∑ L -1 k =0 ‖ W k +1 -W k ‖ 2 F ) 1 / 2 to the loss, for some λ glyph[greaterorequalslant] 0 . The obtained generalization gap is reported in Figure 1b as a function of λ . We observe that this penalization allows to reduce the generalization gap. These two observations go in support of the fact that a smaller Lipschitz constant improves the generalization power of deep residual networks, in accordance with Theorem 2.

However, note that we were not able to obtain an improvement on the test loss by adding the penalization term. This is not all too surprising since previous work has investigated a related penalization, in terms of the Lipschitz norm of the layer sequence ( H k ) 0 glyph[lessorequalslant] k glyph[lessorequalslant] L , and was similarly not able to report any improvement on the test loss (Kelly et al., 2020).

Finally, the proposed penalization term slightly departs from the theory that involves sup 0 glyph[lessorequalslant] k glyph[lessorequalslant] L -1 ( ‖ W k +1 -W k ‖ ∞ ) . This is because the maximum norm is too irregular to be used

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

sup

0

k

L

1

(

Wk

+1

Wk

)

-

0

.

2

-

0

.

1

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Generalization gap

Trained projections

Random projections

-

‖

-

‖∞

(a) Generalization gap as a function of the maximum Lipschitz constant of the weights. Each dot corresponds to a network trained with a varying number of epochs (between 1 and 30 ).

0.0

0.01

0.1

∞

-

0

.

025

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

Generalization gap

λ

(b) Generalization gap as a function of the penalization factor λ . The experiment is repeated 20 times for each value of λ . Each time, the network is trained for 50 epochs. The initial and final matrices are random. The value λ = ∞ corresponds to a weight-tied network.

in practice since, at any one step of gradient descent, it only impacts the maximum weights and not the others. As an illustration, Figure 2 shows the generalization gap when penalizing with the maximum max-norm sup 0 glyph[lessorequalslant] k glyph[lessorequalslant] L -1 ( ‖ W k +1 -W k ‖ ∞ ) and the L 2 norm of the max-norm ( ∑ L -1 k =0 ‖ W k +1 -W k ‖ 2 ∞ ) 1 / 2 . The factor λ is scaled appropriately to reflect the scale difference of the penalizations. The results are mixed: the L 2 norm of the max-norm is effective contrarily to the maximum max-norm. Further investigation of the properties of these norms is left for future work.

0.0

1.0

10.0

λ

0

.

09

0

.

10

0

.

11

0

.

12

0

.

13

0

.

14

0

.

15

Generalization gap

0.0

0.1

1.0

λ

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

Generalization gap

(a) Penalizing by the maximum max-norm.

(b) Penalizing by the L 2 norm of the max-norm.

5 Conclusion

We provide a generalization bound that applies to a wide range of parameterized ODEs. As a consequence, we obtain the first generalization bounds for time-independent and time-dependent neural ODEs in supervised learning tasks. By discretizing our reasoning, we also provide a bound for a class of deep residual networks. Understanding the approximation and optimization properties of this class of neural networks is left for future work. Another intriguing extension is to relax the assumption of linearity of the dynamics at time t with respect to θ i ( t ) , that is, to consider a general formulation dH t = ∑ m i =1 f i ( H t , θ i ( t )) . In the future, it should also be interesting to extend our results to the more involved case of neural SDEs, which have also been found to be deep limits of a large class of residual neural networks (Cohen et al., 2021; Marion et al., 2022).

Acknowledgments and Disclosure of Funding

The author is supported by a grant from Région Île-de-France, by a Google PhD Fellowship, and by MINES Paris - PSL. The author thanks Eloïse Berthier, Gérard Biau, Adeline Fermanian, Clément Mantoux, and Jean-Philippe Vert for inspiring discussions, thorough proofreading and suggestions on this paper.

References

Arnold. Ordinary Differential Equations . Springer Textbook. Springer Berlin Heidelberg, 1992.

Bach. Learning theory from first principles, 2023. Book draft. URL: https://www.di.ens.fr/ ~fbach/ltfp_book.pdf (version: 2023-02-05).

L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research , 20(63): 1-17, 2019.

Béthune, T. Boissin, M. Serrurier, F. Mamalet, C. Friedrich, and A. G. Sanz. Pay attention to your loss : understanding misconceptions about lipschitz neural networks. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022.

Biloš, J. Sommer, S. S. Rangapuram, T. Januschowski, and S. Günnemann. Neural flows: Efficient alternative to neural ODEs. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

A.-S. Cohen, R. Cont, A. Rossier, and R. Xu. Scaling properties of deep residual networks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 2039-2048. PMLR, 2021.

Deuflhard and S. Röblitz. Parameter identification in ODE models. In A Guide to Numerical Modelling in Systems Biology , pages 89-138. Springer International Publishing, 2015.

E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics , 5(1):1-11, 2017. ISSN 2194-671X.

Fermanian, P. Marion, J.-P. Vert, and G. Biau. Framing RNN as a kernel method: A neural ODE approach. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

Gierjatowicz, M. Sabate-Vidales, D. Šiška, L. Szpruch, and Z. Zuriˇ c. Robust pricing and hedging via neural sdes. arXiv preprint arXiv:2007.04154 , 2020.

Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In S. Bubeck, V. Perchet, and P. Rigollet, editors, Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research . PMLR, 2018.

L.-A. Gottlieb, A. Kontorovich, and R. Krauthgamer. Efficient regression in metric spaces via approximate lipschitz extension. IEEE Transactions on Information Theory , 63(8):4838-4849, 2017.

Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse problems , 34(1): 014004, 2017.

Hanson and M. Raginsky. Fitting an immersed submanifold to data via sussmann's orbit theorem. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 5323-5328, 2022.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016a.

He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision - ECCV 2016 , pages 630-645. Springer International Publishing, 2016b.

C.-W. Huang, J. H. Lim, and A. C. Courville. A variational perspective on diffusion-based generative models and score matching. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

Kelly, J. Bettencourt, M. J. Johnson, and D. K. Duvenaud. Learning differential equations that are easy to solve. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020.

Kidger, J. Morrill, J. Foster, and T. Lyons. Neural controlled differential equations for irregular time series. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020.

P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR) , 2015.

Kolmogorov and V. Tikhomirov. ε -entropy and ε -capacity of sets in functional spaces. Uspekhi Mat. Nauk , 14:3-86, 1959.

Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations , 2021.

H. Lim, N. B. Erichson, L. Hodgkinson, and M. W. Mahoney. Noisy recurrent neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

Lu, K. Deng, X. Zhang, G. Liu, and Y. Guan. Neural-ODE for pharmacokinetics modeling and its advantage to alternative machine learning models in predicting new dosing regimens. iScience , 24 (7):102804, 2021.

Lu, A. Zhong, Q. Li, and B. Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. arXiv preprint arXiv:1710.10121 , 2017.

Luk. Notes on existence and uniqueness theorems for ODEs, 2017. URL: http://web.stanford. edu/~jluk/math63CMspring17/Existence.170408.pdf (version: 2023-01-19).

Marion, A. Fermanian, G. Biau, and J.-P. Vert. Scaling ResNets in the large-depth regime. arXiv preprint arXiv:2206.06929 , 2022.

Marion, Y.-H. Wu, M. E. Sander, and B. Gérard. Implicit regularization of deep residual networks towards neural odes. arXiv preprint arXiv:2309.01213 , 2023.

Massaroli, M. Poli, J. Park, A. Yamashita, and H. Asama. Dissecting neural ODEs. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33. Curran Associates, Inc., 2020.

Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In P. Grünwald, E. Hazan, and S. Kale, editors, Proceedings of The 28th Conference on Learning Theory , volume 40 of Proceedings of Machine Learning Research , pages 1376-1401, Paris, France, 03-06 Jul 2015. PMLR.

Neyshabur, S. Bhojanapalli, and N. Srebro. A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks. In International Conference on Learning Representations , 2018.

Pabbaraju, E. Winston, and J. Z. Kolter. Estimating lipschitz constants of monotone deep equilibrium models. In International Conference on Learning Representations , 2021.

G. Pachpatte and W. Ames. Inequalities for Differential and Integral Equations . Elsevier, 1997.

Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Qian, W. Zame, L. Fleuren, P. Elbers, and M. van der Schaar. Integrating expert ODEs into neural ODEs: Pharmacology and disease progression. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

F. Queiruga, N. B. Erichson, D. Taylor, and M. W. Mahoney. Continuous-in-depth neural networks. arXiv preprint arXiv:2008.02389 , 2020.

F. Queiruga, N. B. Erichson, L. Hodgkinson, and M. W. Mahoney. Stateful ODE-nets using basis function expansions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics , 378:686-707, 2019.

E. Sander, P. Ablin, and G. Peyré. Do residual neural networks discretize neural ordinary differential equations? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 35. Curran Associates, Inc., 2022.

Thorpe and Y. van Gennip. Deep limits of residual neural networks. Research in the Mathematical Sciences , 10(1):6, 2022.

Tzen and M. Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883 , 2019.

J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge University Press, 2019.

Yin, I. Ayed, E. de Bézenac, N. Baskiotis, and P. Gallinari. LEADS: Learning dynamical systems that generalize across environments. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34. Curran Associates, Inc., 2021.

Yu, H. Miao, and H. Wu. Neural generalized ordinary differential equations with layer-varying parameters. arXiv preprint arXiv:2209.10633 , 2022.

Zhou, L. Li, K. Zhang, and G. Trajcevski. Urban flow prediction with spatial-temporal neural ODEs. Transportation Research Part C: Emerging Technologies , 124:102912, 2021.

Appendix

Organization of the Appendix Section A contains the proofs of the results of the main paper. Section B contains the details of the numerical illustrations presented in Section 4.3.

A Proofs

A.1 Proof of Proposition 1

The function

is locally Lipschitz-continuous with respect to its first variable and globally Lipschitz-continuous with respect to its second variable. Therefore, the existence and uniqueness of the solution of the initial value problem (5) for t glyph[greaterorequalslant] 0 comes as a consequence of the Picard-Lindelöf theorem (see, e.g., Luk, 2017 for a self-contained presentation and Arnold, 1992 for a textbook).

A.2 Proof of Proposition 2

For x ∈ X , let H be the solution of the initial value problem (5) with parameter θ and with the initial condition H 0 = x . Let us first upper-bound ‖ f i ( H t ) ‖ for all i ∈ { 1 , . . . , m } and t > 0 . To this aim, for t glyph[greaterorequalslant] 0 , we have

Next, Grönwall's inequality yields, for t ∈ [0 , 1] ,

Hence

yielding the first result of the proposition. Furthermore, for any i ∈ { 1 , . . . , m } ,

Now, let ˜ H be the solution of the initial value problem (5) with another parameter ˜ θ and with the same initial condition ˜ H 0 = x . Then, for any t glyph[greaterorequalslant] 0 ,

Hence

Then Grönwall's inequality implies that, for t ∈ [0 , 1] , since 1 glyph[lessorequalslant] K f R Θ exp( K f R Θ ) because K f glyph[greaterorequalslant] 1 , R Θ glyph[greaterorequalslant] 1 .

A.3 Proof of Proposition 3

We first prove the result for m = 1 . Let G x be an ε/ 2 K Θ -grid of [0 , 1] and G y an ε/ 2 -grid of [ -R Θ , R Θ ] . Formally, we can take

Our cover consists of all functions that start at a point of G y , are piecewise linear with kinks in G x , where each piece has slope + K Θ or -K Θ . Hence our cover is of size

Now take a function f : [0 , 1] → R that is uniformly bounded by R Θ and K Θ -Lipschitz. We construct a cover member at distance ε from f as follows. Choose a point y 0 in G y at distance at most ε/ 2 from f (0) . Since f (0) ∈ [ -R Θ , R Θ ] , this is clearly possible, except perhaps at the end of the interval. To verify that it is possible at the end of the interval, note that R Θ is at a distance less than ε/ 2 of the last element of the grid, since

Then, among the cover members that start at y 0 , choose the one which is closest to f at each point of G x (in case of equality, pick any one). Let us denote this cover member as ˜ f . Let us show recursively that f is at glyph[lscript] ∞ -distance at most ε from ˜ f . More precisely, let us first show by induction on k that for all k ∈ { 0 , . . . , glyph[ceilingleft] 2 K Θ ε glyph[ceilingright]} ,

First, | f (0) -˜ f (0) | glyph[lessorequalslant] ε 2 . Then, assume that (15) holds for some k . Then we have the following inequalities:

Moreover, by definition, ˜ f ( ( k +1) ε K Θ ) is the closest point to f ( ( k +1) ε K Θ ) among

The bounds above show that, among those two points, at least one is at distance no more than ε/ 2 from f ( ( k +1) ε K Θ ) . This shows (15) at rank k +1 .

To conclude, take now x ∈ [0 , 1] . There exists k ∈ { 0 , . . . , glyph[ceilingleft] 2 K Θ ε glyph[ceilingright]} such that x is at distance at most ε/ 4 K Θ from kε 2 K Θ . Again, this is clear except perhaps at the end of the interval, where it is also true since

meaning that 1 is located between two elements of the grid G x , showing that it is at distance at most ε/ 4 K Θ from one element of the grid. Then, we have

where the first and third terms are upper-bounded because f and ˜ f are K Θ -Lip, while the second term is upper bounded by (15). Hence ‖ f -˜ f ‖ ∞ glyph[lessorequalslant] ε , proving the result for m = 1 .

Finally, to prove the result for a general m , note that the Cartesian product of ε/m -covers for each coordinate of θ gives an ε -cover for θ . Indeed, consider such covers and take θ ∈ Θ . Since each coordinate of θ is uniformly bounded by R Θ and K Θ -Lipschitz, the proof above shows the existence of a cover member ˜ θ such that, for all i ∈ { 1 , . . . , m } , ‖ θ i -˜ θ i ‖ ∞ glyph[lessorequalslant] ε/m . Then

As a consequence, we conclude that

Taking the logarithm yields the result.

A.4 Proof of Theorem 1

First note that, for any θ ∈ Θ , x ∈ X and y ∈ Y ,

since, by assumption, glyph[lscript] is K glyph[lscript] -Lipschitz with respect to its first variable and glyph[lscript] ( y, y ) = 0 . Thus

by Proposition 2.

Now, taking δ > 0 , a classical computation involving McDiarmid's inequality (see, e.g., Wainwright, 2019, proof of thm 4.10) yields that, with probability at least 1 -δ ,

Denote C = 2 MK f R Θ exp(2 K f R Θ ) . Then we show that R and ̂ R n are CK glyph[lscript] -Lipschitz with respect to ( θ, ‖ · ‖ 1 , ∞ ) : for θ, ˜ θ ∈ Θ ,

according to Proposition 2. The proof for the empirical risk is very similar.

Let now ε > 0 and N ( ε ) be the covering number of Θ endowed with the (1 , ∞ ) -norm. By Proposition 3,

Take θ (1) , . . . , θ ( N ( ε )) the associated cover elements. Then, for any θ ∈ Θ , denoting θ ( i ) the cover element at distance at most ε from θ ,

Hence

Recall that a real-valued random variable X is said to be s 2 sub-Gaussian (Bach, 2023, Section 1.2.1) if for all λ ∈ R , E (exp( λ ( X -E ( X )))) glyph[lessorequalslant] exp( λ 2 s 2 / 2 ) . Since ̂ R n ( θ ) is the average of n independent random variables, which are each almost surely bounded by M , it is M 2 /n sub-Gaussian, hence we have the classical inequality on the expectation of the maximum of sub-Gaussian random variables (see, e.g., Bach, 2023, Exercise 1.13)

The remainder of the proof consists in computations to put the result in the required format. More precisely, we have

The third step is valid if 16 mR Θ ε glyph[greaterorequalslant] 2 . We will shortly take ε to be equal to 1 √ n , thus this condition holds true under the assumption from the Theorem that mR Θ √ n glyph[greaterorequalslant] 3 . Hence we obtain

Now denote ˜ B = 2 MK f exp( K f R Θ ) . Then CK glyph[lscript] glyph[lessorequalslant] ˜ B and 2 M glyph[lessorequalslant] ˜ B . Taking ε = 1 √ n , we obtain

since 2 glyph[lessorequalslant] 2 √ log(2) glyph[lessorequalslant] √ 2( m +1)log(16 mR Θ √ n ) since 16 mR Θ √ n glyph[greaterorequalslant] 2 by the Theorem's assumptions, and √ 2 log(4) glyph[lessorequalslant] 2 . We finally obtain that

by noting that n glyph[greaterorequalslant] 9 max( m -2 R -2 Θ , 1) implies that

The result unfolds since the constant B in the Theorem is equal to 3 ˜ B .

A.5 Proof of Corollary 1

The corollary is an immediate consequence of Theorem 1. To obtain the result, note that m = d 2 , thus in particular √ m +1 = √ d 2 +1 glyph[lessorequalslant] d +1 , and besides log( R W d 2 n ) glyph[lessorequalslant] 2 log( R W dn ) since R W n glyph[lessorequalslant] R 2 W n 2 by assumption on n .

A.6 Proof of Proposition 4

For x ∈ X , let ( H k ) 0 glyph[lessorequalslant] k glyph[lessorequalslant] L be the values of the layers defined by the recurrence (10) with the weights W and the input H 0 = x . We denote by ‖ · ‖ the glyph[lscript] 2 -norm for vectors and the spectral norm for matrices. Then, for k ∈ { 0 , . . . , L -1 } , we have

where the last inequality uses that the spectral norm of a matrix is upper-bounded by its (1 , 1) -norm and that σ (0) = 0 . As a consequence, for any k ∈ { 0 , . . . , L } ,

yielding the first claim of the Proposition.

Now, let ˜ H be the values of the layers (10) with another parameter ˜ W and with the same input ˜ H 0 = x . Then, for any k ∈ { 0 , . . . , L -1 } ,

Hence, using again that the spectral norm of a matrix is upper-bounded by its (1 , 1) -norm and that σ (0) = 0 ,

Then, dividing by (1 + K σ R W L ) k +1 and using the method of differences, we obtain that

Finally note that

We conclude that

A.7 Proof of Proposition 5

For two integers a and b , denote respectively a//b and a % b the quotient and the remainder of the Euclidean division of a by b . Then, for W ∈ R L × d × d , let φ ( W ) : [0 , 1] → R d 2 the piecewise-affine function defined as follows: φ ( W ) is affine on every interval [ k L , k +1 L ] for k ∈ { 0 , . . . , L -1 } ; for k ∈ { 1 , . . . , L } and i ∈ { 1 , . . . , d 2 } ,

and φ ( W ) i (0) = φ ( W ) i (1 /L ) . Then φ ( W ) satisfies two properties. First, it is a linear function of W . Second, for W ∈ R L × d × d ,

because, for x ∈ [0 , 1] , φ ( W )( x ) is a convex combination of two vectors that are bounded in glyph[lscript] 1 -norm by ‖ W ‖ 1 , 1 , ∞ , so it is itself bounded in glyph[lscript] 1 -norm by ‖ W ‖ 1 , 1 , ∞ , implying that ‖ φ ( W ) ‖ 1 , ∞ glyph[lessorequalslant] ‖ W ‖ 1 , 1 , ∞ . Reciprocally,

Now, take W ∈ W . The second property of φ implies that ‖ φ ( W ) ‖ 1 , ∞ glyph[lessorequalslant] R W . Moreover, each coordinate of φ ( W ) is K W -Lipschitz, since the slope of each piece of φ ( W ) i is at most K W . As a consequence, φ ( W ) belongs to

Therefore φ ( W ) is a subset of Θ W , thus its covering number is less than the one of Θ W . Moreover, φ is clearly injective, thus we can define φ -1 on its image. Consider an ε -cover ( θ 1 , . . . , θ N ) of ( φ ( W ) , ‖ · ‖ 1 , ∞ ) . Let us show that ( φ -1 ( θ 1 ) , . . . , φ -1 ( θ N )) is an ε -cover of ( W , ‖ · ‖ 1 , 1 , ∞ ) : take W ∈ W and consider θ i a cover member at distance less than ε from φ ( W ) . Then

where the second equality holds by linearity of φ . Therefore, the covering number of ( W , ‖ · ‖ 1 , 1 , ∞ ) is upper bounded by the one of ( φ ( W ) , ‖ · ‖ 1 , ∞ ) , which itself is upper bounded by the one of (Θ W , ‖ · ‖ 1 , ∞ ) , yielding the result by Proposition 3.

A.8 Proof of Theorem 2

The proof structure is the same as the one of Theorem 1, but some constants change. Similarly to (16), we obtain that, if 16 d 2 R W ε glyph[greaterorequalslant] 2 (which holds true for ε = 1 / √ n and under the assumption of the Theorem),

with and

Finally denote

Then CK glyph[lscript] glyph[lessorequalslant] ˜ B and 2 M glyph[lessorequalslant] ˜ B . Taking ε = 1 √ n , we obtain as in the proof of Theorem 1 that

for n glyph[greaterorequalslant] 9 R -1 W max( d -4 R -1 W , 1) . Thus since √ d 2 +1 glyph[lessorequalslant] d + 1 and R W n glyph[lessorequalslant] R 2 W n 2 by assumption on n . The result unfolds since the constant B in the Theorem is equal to 3 √ 2 ˜ B .

A.9 Proof of Corollary 2

Let where ‖ · ‖ 2 , 1 denotes the (2 , 1) -norm defined as the glyph[lscript] 1 -norm of the glyph[lscript] 2 -norms of the columns, and I is the identity matrix (and we recall that ‖ · ‖ denotes the spectral norm). We apply Theorem 1.1 from Bartlett et al. (2017) by taking as reference matrices the identity matrix. The theorem shows that, under the assumptions of the corollary,

glyph[negationslash]

glyph[negationslash]

where, as in the corollary, ̂ R n ( W ) glyph[lessorequalslant] n -1 ∑ n i =1 1 F W ( x i ) y i glyph[lessorequalslant] γ +max j = y i f ( x i ) j and C is a universal constant. Let us upper bound A ( W ) to conclude. On the one hand, we have

On the other hand, for any k ∈ { 1 , . . . , L } , while

under the assumption that L glyph[greaterorequalslant] R W . All in all, we obtain that

which yields the result.

B Experimental details

Our code is available at https://github.com/PierreMarion23/generalization-ode-resnets .

We use the following model, corresponding to model (10) with additional projections at the beginning and at the end:

where x ∈ R 768 is a vectorized MNIST image, A ∈ R d × 768 , and B ∈ R 10 × d . Table 1 gives the value of the hyperparameters.

We use the initialization scheme outlined in Section 4.1: we initialize, for k ∈ { 1 , . . . , L } and i, j ∈ { 1 , . . . , d } ,

where f i,j are independent Gaussian processes with the RBF kernel (with bandwidth equal to 0 . 1 ). We refer to Marion et al. (2022) and Sander et al. (2022) for further discussion on this initialization scheme. However, A and B are initialized with a more usual scheme, namely with i.i.d. N (0 , 1 /c ) random variables, where c denotes the number of columns of A (resp. B ).

In Figure 1a, we repeat training 10 times independently. Each time, we perform 30 epochs, and compute after each epoch both the Lipschitz constant of the weights and the generalization gap. This gives 300 pairs (Lipschitz constant, generalization gap), which each corresponds to one dot in the figure. Furthermore, we report results for two setups: when A and B are trained or when they are fixed random matrices.

In Figure 1b, A and B are not trained. The reason is to assess the effect of the penalization on W for a fixed scale of A and B . If we allow A and B to vary, then it is possible that the effect of the penalization might be neutralized by a scale increase of A and B during training.

For all experiments, we use the standard MNIST datasplit (60k training samples and 10k testing samples). We train using the cross entropy loss, mini-batches of size 128 , and the optimizer Adam (Kingma and Ba, 2015) with default parameters and a learning rate of 0 . 02 .

We use PyTorch (Paszke et al., 2019) and PyTorch Lightning for our experiments.

The code takes about 60 hours to run on a standard laptop (no GPU).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n【Post-rebuttal Comments】\n\n I thank the authors for the discussions after the authors' rebuttal. My questions are appropriately answered. So, I want to keep my score and vote for acceptance.\n\n\n【Original Comments】\n\nThis paper evaluates the generalization performance of the class of functions represented as solutions of time-dependent parametrized ODEs (including time-dependent Neural ODEs) and their discretized variants, that is, Residual Network, that smoothly changes weight parameters with respect to layer index. The obtained generalization bound is $O(1/n^4)$ for the general case and $O(1/n^2)$ for the time-independent case (for sample size $n$.) Numerical experiments were conducted to examine the relationship between the smoothness of weights between successive layers and generalization performance. Also, a learning method was proposed to add the difference in weights between successive layers as a regularization.\n\nSTRENGTHS:\n- The derived results are consistent with classical statistical learning theory. Specifically, setting $K_\\Theta=0$ yields the classical $O(1/n^2)$ rate.\n- Numerical experiments verify the relationship between the smoothness of weights and generalization performance. These results align with the theory.\n- Comparisons with existing results (Bartlett et al. (2017), Golowich et al. (2018)) are carefully considered. In particular, this study is novel in that it provides generalization bounds for a class of depth-independent models to which Golowich et al. (2018) are not applicable.\n- The paper is well-written. The organization and mathematical description of the paper are appropriate, and it was easy to understand the paper's main point.\n\nWEAKNESSES:\n- To the best of my knowledge, it is rare to impose smoothness between the weights of successive layers in a discretized ResNet model. In addition, the prediction accuracy of those models has yet to be confirmed to be sufficient. Therefore, it is difficult to say that this paper provides a theoretical guarantee for practical models.\n\nQUESTIONS:\nIn l.84--91, this paper refers to existing studies of the generalization bound of continuous-time NNs. However, their relationship with this study has yet to be discussed extensively. Is the novelty of this compared with them that the former is a time-independent ODE while the latter is a time-dependent one? Also, is there any study that dealt with time-independent ODEs?\n\nLIMITATIONS:\nLimitations are discussed in the experiment section (l.338) and conclusion section (l.346.)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors present a generalization bound for a large class of ODEs in this work. They connect ODEs to residual architectures to control the generalization with the differences between weight matrices.

STRENGTHS:
The authors present a theoretical bound, which is something worth highlighting when everybody is just building ad hoc pipelines and then testing them against some benchmark. From this point of view, this work represents a clear advance over other results.

WEAKNESSES:
Even though the results are strong, the consequences and how they can be used are not done in a way that can help the reader to fully understand the applications of these results.

QUESTIONS:
Have you measured how the bounds fare in time series problems?

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the generalization ability of neural ordinary differential equations and deep residual networks through Lipschitz-based complexity arguments. The bound, specifically for the discretized version, involves the maximum magnitude of the difference between successive weight matrices, which is not commonly seen. The paper also uses numerical experiments to investigate how this quantity relates to the generalization capability of deep residual networks. The generalization gap is smaller when the quantity is smaller, but adding the quantity as a penalization term did not improve the prediction performance.\n\nSTRENGTHS:\n[originality]\n\n- To the best of my knowledge, this paper is the first to present a generalization error bound for neural ODEs. The proof techniques used in this work are rather standard, but they are combined in new ways to derive the results.\n- The potential importance of the Lipschitzness of the weights (i.e., the maximal magnitude of the differences between successive weight matrices) is newly proposed.\n\n[quality]\n\n- The analysis and results seem reasonable, and the mathematical setup is adequately explained.\n\n[clarity]\n\n- The paper is well-organized and articulately written. Its concise abstract enhances the paper's accessibility and ease of comprehension.\n\n[significance]\n\n- Neural ODEs and deep residual networks are now among the standard tools in the machine learning community. Therefore, theoretical results that advance our understanding of the behavior of these models are relevant and can improve their reliability for use in various applications.\n- The analysis in Theorem 1 reveals a term whose convergence rate is O(n^{-1/4}). Corollary 1 exemplifies that the slow-rate term can be eliminated by making the coefficient parameter time-independent in the infinite-dimensional case, partly elucidating the consequences of having time-dependent component in the model.\n\nWEAKNESSES:\n[originality]\n\n- None in particular\n\n[quality]\n\n- The paper would benefit from additional citations, particularly where named theorems are mentioned in the main text. For instance, it would be helpful to add citations to the Picard-Lindelof theorem at Line 59 and to Gronwall's inequality at Line 166 for the convenience of the reader.\n\n[clarity]\n\n- None in particular\n\n[significance]\n\n- None in particular\n\nQUESTIONS:\nHave you conducted any experiments of Figure 1b using the maximum max-norm of the weights as a penalty term (or a sum of the max-norm of the weights) instead of the squared sum of Frobenius norms? While I acknowledge the equivalence between matrix norms, and although it may not be strictly necessary for the paper, I believe that readers (including myself) would be interested in seeing whether the behavior is different when the quantity that appeared directly in deriving the theoretical guarantee is used.\n\n[minor suggestions]\n\n- Line 112 “takes values into” → “takes values in”\n- Line 285 “Corollary 2 (of Theorem 1.1 of Bartlett et al. (2017))” → “Corollary 2 (corollary of Theorem 1.1 of Bartlett et al. (2017))”\n- Line 319 “a.k.a.” → “i.e.,” may be more appropriate?\n\nLIMITATIONS:\nNone in particular", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a generalization bound for the large class of time-dependent and time-independent neural ODEs. In addition, by leveraging on the connection between neural ODEs and deep residual networks, the paper provides a depth-independent generalization bound for the class of deep residual networks. The bound is compared with some earlier results, showing its novelty. The paper introduces a novel way of controlling the statistical complexity of neural networks and the bound depends on the magnitude of the difference between successive weight matrices. Numerical experiments are provided to show the relationship between this bound and the generalization ability of neural networks.\n\nSTRENGTHS:\nThe paper provides a generalization bound for the large class of time-dependent, time-independent neural ODEs and deep residual networks. The bound is compared with some earlier results, showing its novelty. The paper introduces a novel way of controlling the statistical complexity of neural networks and the bound depends on the magnitude of the difference between successive weight matrices. Numerical experiments are provided to show the relationship between this bound and the generalization ability of neural networks.\n\nWEAKNESSES:\nSome expressions are not very clear, and some results have too strong conditions. Please see Questions below.\n\nQUESTIONS:\n1. In Proposition 2 and Theorem 1, what do $R_X$ and $R_Y$ mean? The reviewer guesses that $R_X$ is the bound of the set of initial values $x$, but these two notations as well as $X$ and $Y$ do not have clear definitions.\n\n2. A clear definition of $n$ in Theorem 1 is missing. If the data is any $n$ samples, the conclusion of Theorem 1 is obviously wrong.\n\n3. The condition in Corollary 1 is too strong. The reviewer think Neural ODE should contains at least one hidden layer. And the reviewer think that the generalization of Neural ODE with one hidden layer can be obtained by using Theorem 1, by studying the Lipschitz constant of $\\sigma(Wx)$. \n\n4. The reviewer thinks that some existing definitions or symbols, such as covering number, subGaussian, should be briefly explained.\n\n5. The numerical experiments show the correlation between the generalization gap and the maximum Lipschitz constant of the weight. However, the generalization gap proved in this paper mainly depends on the number of samples $n$. Could the paper provide experiments to show the correlation between these two quantities as well as the convergence order?\n\n6. Some abnormal result points can be found in the experimental results, such as the point in the bottom left corner of Figure 1. Is there any explanation for this？\n\n7. Theory and numerical experiments show small Lipschitz constant of the weights reduce the generalization gap. Does this indicate that time independent coefficients ($K_{\\Theta}=0$?) have better generalization?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}