SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pierre Marion
Sorbonne Université, CNRS,
Laboratoire de Probabilités, Statistique et Modélisation, LPSM,
F-75005 Paris, France
pierre.marion@sorbonne-universite.fr
Neural ordinary differential equations (neural ODEs) are a popular family of continuous-depth deep learning models. In this work, we consider a large family of parameterized ODEs with continuous-in-time parameters, which include time-dependent neural ODEs. We derive a generalization bound for this class by a Lipschitz-based argument. By leveraging the analogy between neural ODEs and deep residual networks, our approach yields in particular a generalization bound for a class of deep residual networks. The bound involves the magnitude of the difference between successive weight matrices. We illustrate numerically how this quantity affects the generalization capability of neural networks.
Neural ordinary differential equations (neural ODEs, Chen et al., 2018) are a flexible family of neural networks used in particular to model continuous-time phenomena. Along with variants such as neural stochastic differential equations (neural SDEs, Tzen and Raginsky, 2019) and neural controlled differential equations (Kidger et al., 2020), they have been used in diverse fields such as pharmokinetics (Lu et al., 2021; Qian et al., 2021), finance (Gierjatowicz et al., 2020), and transportation (Zhou et al., 2021). We refer to Massaroli et al. (2020) for a self-contained introduction to this class of models.
Despite their empirical success, the statistical properties of neural ODEs have not yet been fully investigated. What is more, neural ODEs can be thought of as the infinite-depth limit of (properly scaled) residual neural networks (He et al., 2016a), a connection made by, e.g., E (2017); Haber and Ruthotto (2017); Lu et al. (2017). Since standard measures of statistical complexity of neural networks grow with depth (see, e.g., Bartlett et al., 2019), it is unclear why infinite-depth models, including neural ODEs, should enjoy favorable generalization properties.
To better understand this phenomenon, our goal in this paper is to study the statistical properties of a class of time-dependent neural ODEs that write
$$
d H _ {t} = W _ {t} \sigma \left(H _ {t}\right) d t, \tag {1}
$$
where  $W_{t} \in \mathbb{R}^{d \times d}$  is a weight matrix that depends on the time index  $t$ , and  $\sigma: \mathbb{R} \to \mathbb{R}$  is an activation function applied component-wise. Time-dependent neural ODEs were first introduced by Massaroli et al. (2020) and generalize time-independent neural ODEs
$$
d H _ {t} = W \sigma \left(H _ {t}\right) d t, \tag {2}
$$
as formulated in Chen et al. (2018), where  $W \in \mathbb{R}^{d \times d}$  now denotes a weight matrix independent of  $t$ . There are two crucial reasons to consider time-dependent neural ODEs rather than the more
restrictive class of time-independent neural ODEs. On the one hand, the time-dependent formulation is more flexible, leading to competitive results on image classification tasks (Queiruga et al., 2020, 2021). As a consequence, obtaining generalization guarantees for this family of models is a valuable endeavor by itself. On the other hand, time dependence is required for the correspondence with general residual neural networks to hold. More precisely, the time-dependent neural ODE (1) is the limit, when the depth  $L$  goes to infinity, of the deep residual network
$$
H _ {k + 1} = H _ {k} + \frac {1}{L} W _ {k + 1} \sigma \left(H _ {k}\right), \quad 0 \leqslant k \leqslant L - 1, \tag {3}
$$
where  $(W_{k})_{1\leqslant k\leqslant L}\in \mathbb{R}^{d\times d}$  are weight matrices and  $\sigma$  is still an activation function. We refer to Marion et al. (2022, 2023); Sander et al. (2022); Thorpe and van Gennip (2022) for statements that make precise under what conditions and in which sense this limit holds, as well as its consequences for learning. These two key reasons compel us to consider the class of time-dependent ODEs (1) for our statistical study, which in turn will inform us on the properties of the models (2) and (3).
In fact, we extend our study to the larger class of parameterized ODEs, which we define as the mapping from  $x \in \mathbb{R}^d$  to the value at time  $t = 1$  of the solution of the initial value problem
$$
H _ {0} = x, \quad d H _ {t} = \sum_ {i = 1} ^ {m} \theta_ {i} (t) f _ {i} \left(H _ {t}\right) d t, \tag {4}
$$
where  $H_{t}$  is the variable of the ODE,  $\theta_{i}$  are functions from [0, 1] into  $\mathbb{R}$  that parameterize the ODE, and  $f_{i}$  are fixed functions from  $\mathbb{R}^d$  into  $\mathbb{R}^d$ . Time-dependent neural ODEs (1) are obtained by setting a specific entrywise form for the functions  $f_{i}$  in (4).
Since the parameters  $\theta_{i}$  belong to an infinite-dimensional space, in practice they need to be approximated in a finite-dimensional basis of functions. For example, the residual neural networks (3) can be seen as an approximation of the neural ODEs (1) on a piecewise-constant basis of function. But more complex choices are possible, such as B-splines (Yu et al., 2022). However, the formulation (4) is agnostic from the choice of finite-dimensional approximation. This more abstract point of view is fruitful to derive generalization bounds, for at least two reasons. First, the statistical properties of the parameterized ODEs (4) only depend on the characteristics of the functions  $\theta_{i}$  and not on the specifics of the approximation scheme, so it is more natural and convenient to study them at the continuous level. Second, their properties can then be transferred to any specific discretization, such as the deep residual networks (3), resulting in generalization bounds for the latter.
Regarding the characteristics of the functions  $\theta_{i}$ , we make the structural assumption that they are Lipschitz-continuous and uniformly bounded. This is a natural assumption to ensure that the initial value problem (4) has a unique solution in the usual sense of the Picard-Lindelöf theorem (Arnold, 1992). Remarkably, this assumption on the parameters also enables us to obtain statistical guarantees despite the fact that we are working with an infinite-dimensional set of parameters.
Contributions. We provide a generalization bound for the large class of parameterized ODEs (4), which include time-dependent and time-independent neural ODEs (1) and (2). To the best of our knowledge, this is the first available bound for neural ODEs in supervised learning. By leveraging on the connection between (time-dependent) neural ODEs and deep residual networks, our approach allows us to provide a depth-independent generalization bound for the class of deep residual networks (3). The bound is precisely compared with earlier results. Our bound depends in particular on the magnitude of the difference between successive weight matrices, which is, to our knowledge, a novel way of controlling the statistical complexity of neural networks. Numerical illustration is provided to show the relationship between this quantity and the generalization ability of neural networks.
Organization of the paper. Section 2 presents additional related work. In Section 3, we specify our class of parameterized ODEs, before stating the generalization bound for this class and for neural ODEs as a corollary. The generalization bound for residual networks is presented in Section 4 and compared to other bounds, before some numerical illustration. Section 5 concludes the paper. The proof technique is discussed in the main paper, but the core of the proofs is relegated to the Appendix.
Hybridizing deep learning and differential equations. The fields of deep learning and dynamical systems have recently benefited from sustained cross-fertilization. On the one hand, a large line of work is aimed at modeling complex continuous-time phenomena by developing specialized neural architectures. This family includes neural ODEs, but also physics-informed neural networks (Raissi et al., 2019), neural operators (Li et al., 2021) and neural flows (Biloš et al., 2021). On the other hand, successful recent advances in deep learning, such as diffusion models, are theoretically supported by ideas from differential equations (Huang et al., 2021).
Generalization for continuous-time neural networks. Obtaining statistical guarantees for continuous-time neural networks has been the topic of a few recent works. For example, Germanian et al. (2021) consider recurrent neural networks (RNNs), a family of neural networks handling time series, which is therefore a different setup from our work that focuses on vector-valued inputs. These authors show that a class of continuous-time RNNs can be written as input-driven ODEs, which are then proved to belong to a family of kernel methods, which entails a generalization bound. Lim et al. (2021) also show a generalization bound for ODE-like RNNs, and argue that adding stochasticity (that is, replacing ODEs with SDEs) helps with generalization. Taking another point of view, Yin et al. (2021) tackle the separate (although related) question of generalization when doing transfer learning across multiple environments. They propose a neural ODE model and provide a generalization bound in the case of a linear activation function. Closer to our setting, Hanson and Raginsky (2022) show a generalization bound for parameterized ODEs for manifold learning, which applies in particular for neural ODEs. Their proof technique bears similarities with ours, but the model and task differ from our approach. In particular, they consider stacked time-independent parameterized ODEs, while we are interested in a time-dependent formulation. Furthermore, these authors do not discuss the connection with residual networks.
Lipschitz-based generalization bounds for deep neural networks. From a high-level perspective, our proof technique is similar to previous works (Bartlett et al., 2017; Neyshabur et al., 2018) that show generalization bounds for deep neural networks, which scale at most polynomially with depth. More precisely, these authors show that the network satisfies some Lipschitz continuity property (either with respect to the input or to the parameters), then exploit results on the statistical complexity of Lipschitz function classes. Under stronger norm constraints, these bounds can even be made depth-independent (Golowich et al., 2018). However, their approach differs from ours insofar as we consider neural ODEs and the associated family of deep neural networks, whereas they are solely interested in finite-depth neural networks. As a consequence, their hypotheses on the class of neural networks differ from ours. Section 4 develops a more thorough comparison. Similar Lipschitz-based techniques have also been applied to obtain generalization bounds for deep equilibrium networks (Pabbaraju et al., 2021). Going beyond statistical guarantees, Béthane et al. (2022) study approximation and robustness properties of Lipschitz neural networks.
3 Generalization bounds for parameterized ODEs
We start by recalling the usual supervised learning setup and introduce some notation in Section 3.1, before presenting our parameterized ODE model and the associated generalization bound in Section 3.2. We then apply the bound to the specific case of time-invariant neural ODEs in Section 3.3.
3.1 Learning procedure
We place ourselves in a supervised learning setting. Let us introduce the notation that are used throughout the paper (up to and including Section 4.1). The input data is a sample of  $n$  i.i.d. pairs  $(x_i, y_i)$  with the same distribution as some generic pair  $(x, y)$ , where  $x$  (resp.  $y$ ) takes its values in some bounded ball  $\mathcal{X} = B(0, R_{\mathcal{X}})$  (resp.  $\mathcal{Y} = B(0, R_{\mathcal{Y}})$ ) of  $\mathbb{R}^d$ , for some  $R_{\mathcal{X}}, R_{\mathcal{Y}} > 0$ . This setting encompasses regression but also classification tasks by (one-hot) encoding labels in  $\mathbb{R}^d$ . Note that we assume for simplicity that the input and output have the same dimension, but our analysis easily extends to the case where they have different dimensions by adding (parameterized) projections at the beginning or at the end of our model. Given a parameterized class of models  $\mathcal{F}_{\Theta} = \{F_{\theta}, \theta \in \Theta\}$ , the parameter  $\theta$  is fitted by empirical risk minimization using a loss function  $\ell : \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}^+$  that
we assume to be Lipschitz with respect to its first argument, with a Lipschitz constant  $K_{\ell} > 0$ . In the following, we write for the sake of concision that such a function is  $K_{\ell}$ -Lipschitz. We also assume that  $\ell(x, x) = 0$  for all  $x \in \mathbb{R}^d$ . The theoretical and empirical risks are respectively defined, for any  $\theta \in \Theta$ , by
$$
\mathcal {R} (\theta) = \mathbb {E} [ \ell (F _ {\theta} (x), y) ] \quad \text {a n d} \quad \widehat {\mathcal {R}} _ {n} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(F _ {\theta} \left(x _ {i}\right), y _ {i}\right),
$$
where the expectation  $\mathbb{E}$  is evaluated with respect to the distribution of  $(x,y)$ . Letting  $\widehat{\theta}_n$  a minimizer of the empirical risk, the generalization problem consists in providing an upper bound on the difference  $\mathcal{R}(\widehat{\theta}_n) - \widehat{\mathcal{R}}_n(\widehat{\theta}_n)$ .
3.2 Generalization bound
Model. We start by making more precise the parameterized ODE model introduced in Section 1. The setup presented here can easily be specialized to the case of neural ODEs, as we will see in Section 3.3. Let  $f_{1},\ldots ,f_{m}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  be fixed  $K_{f}$ -Lipschitz functions for some  $K_{f} > 0$ . Denote by  $M$  their supremum on  $\mathcal{X}$  (which is finite since these functions are continuous). The parameterized ODE  $F_{\theta}$  is defined by the following initial value problem that maps some  $x\in \mathbb{R}^d$  to  $F_{\theta}(x)\in \mathbb{R}^{d}$ :
$$
H _ {0} = x
$$
$$
d H _ {t} = \sum_ {i = 1} ^ {m} \theta_ {i} (t) f _ {i} \left(H _ {t}\right) d t \tag {5}
$$
$$
F _ {\theta} (x) = H _ {1},
$$
where the parameter  $\theta = (\theta_{1},\dots,\theta_{m})$  is a function from  $[0,1]$  to  $\mathbb{R}^m$ . We have to impose constraints on  $\theta$  for the model  $F_{\theta}$  to be well-defined. To this aim, we endow (essentially bounded) functions from  $[0,1]$  to  $\mathbb{R}^m$  with the following  $(1,\infty)$ -norm
$$
\left\| \theta \right\| _ {1, \infty} = \sup  _ {0 \leqslant t \leqslant 1} \sum_ {i = 1} ^ {m} \left| \theta_ {i} (t) \right|. \tag {6}
$$
We can now define the set of parameters
$$
\Theta = \left\{\theta : [ 0, 1 ] \rightarrow \mathbb {R} ^ {m}, \| \theta \| _ {1, \infty} \leqslant R _ {\Theta} \text {a n d} \theta_ {i} \text {i s} K _ {\Theta} \text {- L i p s c h i t z f o r} i \in \{1, \dots , m \} \right\}, \tag {7}
$$
for some  $R_{\Theta} > 0$  and  $K_{\Theta} \geqslant 0$ . Then, for  $\theta \in \Theta$ , the following Proposition, which is a consequence of the Picard-Lindelöf Theorem, shows that the mapping  $x \mapsto F_{\theta}(x)$  is well-defined.
Proposition 1 (Well-posedness of the parameterized ODE). For  $\theta \in \Theta$  and  $x\in \mathbb{R}^d$ , there exists a unique solution to the initial value problem (5).
An immediate consequence of Proposition 1 is that it is legitimate to consider  $\mathcal{F}_{\Theta} = \{F_{\theta}, \theta \in \Theta\}$  for our model class.
When  $K_{\Theta} = 0$ , the parameter space  $\Theta$  is finite-dimensional since each  $\theta_i$  is constant. This setting corresponds to the time-independent neural ODEs of Chen et al. (2018). In this case, the norm (6) reduces to the  $\| \cdot \|_1$  norm over  $\mathbb{R}^m$ . Note that, to fit exactly the formulation of Chen et al. (2018), the time  $t$  can be added as a variable of the functions  $f_i$ , which amounts to adding a new coordinate to  $H_t$ . This does not change the subsequent analysis. In the richer time-dependent case where  $K_{\Theta} > 0$ , the set  $\Theta$  belongs to an infinite-dimensional space and therefore, in practice,  $\theta_i$  is approximated in a finite basis of functions, such as Fourier series, Chebyshev polynomials, and splines. We refer to Massaroli et al. (2020) for a more detailed discussion, including formulations of the backpropagation algorithm (a.k.a. the adjoint method) in this setting.
Note that we consider the case where the dynamics at time  $t$  are linear with respect to the parameter  $\theta_{i}(t)$ . Nevertheless, we emphasize that the mapping  $x \mapsto F_{\theta}(x)$  remains a highly non-linear function of each  $\theta_{i}(t)$ . To fix ideas, this setting can be seen as analogue to working with pre-activation residual networks instead of post-activation (see He et al., 2016b, for definitions of the terminology), which is a mild modification.
Statistical analysis Since  $\Theta$  is a subset of an infinite-dimensional space, complexity measures based on the number of parameters cannot be used. Instead, our approach is to resort to Lipschitz-based complexity measures. More precisely, to bound the complexity of our model class, we propose two building blocks: we first show that the model  $F_{\theta}$  is Lipschitz-continuous with respect to its parameters  $\theta$ . This allows us to bound the complexity of the model class depending on the complexity of the parameter class. In a second step, we assess the complexity of the class of parameters itself.
Starting with our first step, we show the following estimates for our class of parameterized ODEs. Here and in the following,  $\| \cdot \|$  denotes the  $\ell_2$  norm over  $\mathbb{R}^d$ .
Proposition 2 (The parameterized ODE is bounded and Lipschitz). Let  $\theta$  and  $\tilde{\theta} \in \Theta$ . Then, for any  $x \in \mathcal{X}$ ,
$$
\left\| F _ {\theta} (x) \right\| \leqslant R _ {\mathcal {X}} + M R _ {\Theta} \exp \left(K _ {f} R _ {\Theta}\right)
$$
$$
\left\| F _ {\theta} (x) - F _ {\tilde {\theta}} (x) \right\| \leqslant 2 M K _ {f} R _ {\Theta} \exp (2 K _ {f} R _ {\Theta}) \| \theta - \tilde {\theta} \| _ {1, \infty}.
$$
The proof, given in the Appendix, makes extensive use of Gronwall's inequality (Pachpatte and Ames, 1997), a standard tool to obtain estimates in the theory of ODEs, in order to bound the magnitude of the solution  $H_{t}$  of (5).
The next step is to assess the magnitude of the covering number of  $\Theta$ . Recall that, for  $\varepsilon > 0$ , the  $\varepsilon$ -covering number of a metric space is the number of balls of radius  $\varepsilon$  needed to completely cover the space, with possible overlaps. More formally, considering a metric space  $\mathcal{M}$  and denoting by  $B(x, \varepsilon)$  the ball of radius  $\varepsilon$  centered at  $x \in \mathcal{M}$ , the  $\varepsilon$ -covering number of  $\mathcal{M}$  is equal to  $\inf \{n \geqslant 1 | \exists x_1, \ldots, x_n \in \mathcal{M}, \mathcal{M} \subseteq \bigcup_{i=1}^{n} B(x_i, \varepsilon)\}$ .
Proposition 3 (Covering number of the ODE parameter class). For  $\varepsilon >0$ , let  $\mathcal{N}(\varepsilon)$  be the  $\varepsilon$ -covering number of  $\Theta$  endowed with the distance associated to the  $(1,\infty)$ -norm (6). Then
$$
\log \mathcal {N} (\varepsilon) \leqslant m \log \left(\frac {1 6 m R _ {\Theta}}{\varepsilon}\right) + \frac {m ^ {2} K _ {\Theta} \log (4)}{\varepsilon}.
$$
Proposition 3 is a consequence of a classical result, see, e.g., Kolmogorov and Tikhomirov (1959, example 3 of paragraph 2). A self-contained proof is given in the Appendix for completeness. We also refer to Gottlieb et al. (2017) for more general results on covering numbers of Lipschitz functions.
The two propositions above and an  $\varepsilon$ -net argument allow to prove the first main result of our paper (where we recall that the notations are defined in Section 3.1).
Theorem 1 (Generalization bound for parameterized ODEs). Consider the class of parameterized ODEs  $\mathcal{F}_{\Theta} = \{F_{\theta}, \theta \in \Theta\}$ , where  $F_{\theta}$  is given by (5) and  $\Theta$  by (7). Let  $\delta > 0$ .
Then, for  $n \geqslant 9\max(m^{-2}R_{\Theta}^{-2},1)$ , with probability at least  $1 - \delta$
$$
\mathcal {R} (\widehat {\theta} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + B \sqrt {\frac {(m + 1) \log (R _ {\Theta} m n)}{n}} + B \frac {m \sqrt {K _ {\Theta}}}{n ^ {1 / 4}} + \frac {B}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$
where  $B$  is a constant depending on  $K_{\ell}, K_{f}, R_{\Theta}, R_{\mathcal{X}}, R_{\mathcal{Y}}, M$ . More precisely,
$$
B = 6 K _ {\ell} K _ {f} \exp (K _ {f} R _ {\Theta}) \big (R _ {\mathcal {X}} + M R _ {\Theta} \exp (K _ {f} R _ {\Theta}) + R _ {\mathcal {Y}} \big).
$$
Three terms appear in our upper bound of  $\mathcal{R}(\widehat{\theta}_n) - \widehat{\mathcal{R}}_n(\widehat{\theta}_n)$ . The first and the third ones are classical (see, e.g. Bach, 2023, Sections 4.4 and 4.5). On the contrary, the second term is more surprising with its convergence rate in  $\mathcal{O}(n^{-1/4})$ . This slower convergence rate is due to the fact that the space of parameters is infinite-dimensional. In particular, for  $K_{\Theta} = 0$ , corresponding to a finite-dimensional space of parameters, we recover the usual  $\mathcal{O}(n^{-1/2})$  convergence rate, however at the cost of considering a much more restrictive class of models. Finally, it is noteworthy that the dimensionality appearing in the bound is not the input dimension  $d$  but the number of mappings  $m$ .
Note that this result is general and may be applied in a number of contexts that go beyond deep learning, as long as the instantaneous dependence of the ODE dynamics to the parameters is linear. One such example is the predator-prey model, describing the evolution of two populations of animals, which reads  $dx_{t} = x_{t}(\alpha - \beta y_{t})dt$  and  $dy_{t} = -y_{t}(\gamma - \delta x_{t})dt$ , where  $x_{t}$  and  $y_{t}$  are real-valued variables and  $\alpha, \beta, \gamma$  and  $\delta$  are model parameters. This ODE falls into the framework of this section,
if one were to estimate the parameters by empirical risk minimization. We refer to Deuflhard and Röblitz (2015, section 3) for other examples of parameterized biological ODE dynamics and methods for parameter identification.
Nevertheless, for the sake of brevity, we focus on applications of this result to deep learning, and more precisely to neural ODEs, which is the topic of the next section.
3.3 Application to neural ODEs
As explained in Section 1, parameterized ODEs include both time-dependent and time-independent neural ODEs. Since the time-independent model is more common in practice, we develop this case here and leave the time-dependent case to the reader. We thus consider the following neural ODE:
$$
d H _ {t} = W \sigma \left(H _ {t}\right) d t \tag {8}
$$
$$
F _ {W} (x) = H _ {1},
$$
where  $W \in \mathbb{R}^{d \times d}$  is a weight matrix, and  $\sigma : \mathbb{R} \to \mathbb{R}$  is an activation function applied component-wise. We assume  $\sigma$  to be  $K_{\sigma}$ -Lipschitz for some  $K_{\sigma} > 0$ . This assumption is satisfied by all common activation functions. To put the model in the form of Section 3.2, denote  $e_1, \ldots, e_d$  the canonical basis of  $\mathbb{R}^d$ . Then the dynamics (8) can be reformulated as
$$
d H _ {t} = \sum_ {i, j = 1} ^ {d} W _ {i j} \sigma_ {i j} (H _ {t}) d t,
$$
where  $\sigma_{ij}(x) = \sigma (x_j)e_i$ . Each  $\sigma_{ij}$  is itself  $K_{\sigma}$ -Lipschitz, hence we fall in the framework of Section 3.2. In other words, the functions  $f_{i}$  of our general parameterized ODE model form a shallow neural network with pre-activation. Denote by  $\| W\|_{1,1}$  the sum of the absolute values of the elements of  $W$ . We consider the following set of parameters, which echoes the set  $\Theta$  of Section 3.2:
$$
\mathcal {W} = \{W \in \mathbb {R} ^ {d \times d}, \| W \| _ {1, 1} \leqslant R _ {\mathcal {W}} \}, \tag {9}
$$
for some  $R_{\mathcal{W}} > 0$ . We can then state the following result as a consequence of Theorem 1.