SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 1 (Generalization bound for neural ODEs). Consider the class of neural ODEs  $\mathcal{F}_{\mathcal{W}} = \{F_W, W \in \mathcal{W}\}$ , where  $F_W$  is given by (8) and  $\mathcal{W}$  by (9). Let  $\delta > 0$ .
Then, for  $n \geqslant 9R_{\mathcal{W}}^{-1}\max(d^{-4}R_{\mathcal{W}}^{-1}, 1)$ , with probability at least  $1 - \delta$
$$
\mathcal {R} (\widehat {W} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {W} _ {n}) + B (d + 1) \sqrt {\frac {\log (R _ {\mathcal {W}} d n)}{n}} + \frac {B}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$
where  $B$  is a constant depending on  $K_{\ell}, K_{\sigma}, R_{\mathcal{W}}, R_{\mathcal{X}}, R_{\mathcal{Y}}, M$ . More precisely,
$$
B = 6 \sqrt {2} K _ {\ell} K _ {\sigma} \exp (K _ {\sigma} R _ {\mathcal {W}}) \bigl (R _ {\mathcal {X}} + M R _ {\mathcal {W}} \exp (K _ {\sigma} R _ {\mathcal {W}}) + R _ {\mathcal {Y}} \bigr).
$$
Note that the term in  $\mathcal{O}(n^{-1/4})$  from Theorem 1 is now absent. Since we consider a time-independent model, we are left with the other two terms, recovering a standard  $\mathcal{O}(n^{-1/2})$  convergence rate.
4 Generalization bounds for deep residual networks
As highlighted in Section 1, there is a strong connection between neural ODEs and discrete residual neural networks. The previous study of the continuous case in Section 3 paves the way for deriving a generalization bound in the discrete setting of residual neural networks, which is of great interest given the pervasiveness of this architecture in modern deep learning.
We begin by presenting our model and result in Section 4.1, before detailing the comparison of our approach with other papers in Section 4.2 and giving some numerical illustration in Section 4.3.
4.1 Model and generalization bound
Model. We consider the following class of deep residual networks:
$$
H _ {k + 1} = H _ {k} + \frac {1}{L} W _ {k + 1} \sigma \left(H _ {k}\right), \quad 0 \leqslant k \leqslant L - 1 \tag {10}
$$
$$
F _ {\mathbf {W}} (x) = H _ {L},
$$
where the parameter  $\mathbf{W} = (W_k)_{1\leqslant k\leqslant L}\in \mathbb{R}^{L\times d\times d}$  is a set of weight matrices and  $\sigma$  is still a  $K_{\sigma}$ -Lipschitz activation function. To emphasize that  $\mathbf{W}$  is here a third-order tensor, as opposed to the case of time-invariant neural ODEs in Section 3.3, where  $W$  was a matrix, we denote it with a bold notation. We also assume in the following that  $\sigma(0) = 0$ . This assumption could be alleviated at the cost of additional technicalities. Owing to the  $1 / L$  scaling factor, the deep limit of this residual network is a (time-dependent) neural ODE of the form studied in Section 3. We refer to Marion et al. (2022) for further discussion on the link between scaling factors and deep limits. We simply note that this scaling factor is not common practice, but preliminary experiments show it does not hurt performance and can even improve performance in a weight-tied setting (Sander et al., 2022). The space of parameters is endowed with the following  $(1,1,\infty)$ -norm
$$
\| \mathbf {W} \| _ {1, 1, \infty} = \sup  _ {1 \leqslant k \leqslant L} \sum_ {i, j = 1} ^ {d} | W _ {k, i, j} |. \tag {11}
$$
Also denoting  $\| \cdot \|_{\infty}$  the element-wise maximum norm for a matrix, we consider the class of matrices
$$
\mathcal {W} = \left\{\mathbf {W} \in \mathbb {R} ^ {L \times d \times d}, \| \mathbf {W} \| _ {1, 1, \infty} \leqslant R _ {\mathcal {W}} \quad \text {a n d} \right. \tag {12}
$$
$$
\left. \left\| W _ {k + 1} - W _ {k} \right\| _ {\infty} \leqslant \frac {K _ {\mathcal {W}}}{L} \text {f o r} 1 \leqslant k \leqslant L - 1 \right\},
$$
for some  $R_{\mathcal{W}} > 0$  and  $K_{\mathcal{W}} \geqslant 0$ , which is a discrete analogue of the set  $\Theta$  defined by (7).
In particular, the upper bound on the difference between successive weight matrices is to our knowledge a novel way of constraining the parameters of a neural network. It corresponds to the discretization of the Lipschitz continuity of the parameters introduced in (7). By analogy, we refer to it as a constraint on the Lipschitz constant of the weights. Note that, for standard initialization schemes, the difference between two successive matrices is of the order  $\mathcal{O}(1)$  and not  $\mathcal{O}(1 / L)$ , or, in other words,  $K_{\mathcal{W}}$  scales as  $\mathcal{O}(L)$ . This dependence of  $K_{\mathcal{W}}$  on  $L$  can be lifted by adding correlations across layers at initialization. For instance, one can take, for  $k \in \{1, \dots, L\}$  and  $i, j \in \{1, \dots, d\}$ ,  $\mathbf{W}_{k,i,j} = \frac{1}{\sqrt{d}} f_{i,j}\left(\frac{k}{L}\right)$ , where  $f_{i,j}$  is a smooth function, for example a Gaussian process with the RBF kernel. Such a non-i.i.d. initialization scheme is necessary for the correspondence between deep residual networks and neural ODEs to hold (Marion et al., 2022). Furthermore, Sander et al. (2022) prove that, with this initialization scheme, the constraint on the Lipschitz constant also holds for the trained network, with  $K_{\mathcal{W}}$  independent of  $L$ . Finally, we emphasize that the following developments also hold in the case where  $K_{\mathcal{W}}$  depends on  $L$  (see also Section 4.2 for a related discussion).
Statistical analysis. At first sight, a reasonable strategy would be to bound the distance between the model (10) and its limit  $L \to \infty$  that is a parameterized ODE, then apply Theorem 1. This strategy is straightforward, but comes at the cost of an additional  $\mathcal{O}(1 / L)$  term in the generalization bound, as a consequence of the discretization error between the discrete iterations (10) and their continuous limit. For example, we refer to Fermanian et al. (2021) where this strategy is used to prove a generalization bound for discrete RNNs and where this additional error term is incurred. We follow another way by mimicking all the proof with a finite  $L$ . This is a longer approach but it yields a sharper result since we avoid the  $\mathcal{O}(1 / L)$  discretization error. The proof structure is similar to Section 3: the following two Propositions are the discrete counterparts of Propositions 2 and 3.
Proposition 4 (The residual network is bounded and Lipschitz). Let  $\mathbf{W}$  and  $\tilde{\mathbf{W}}\in \mathcal{W}$ . Then, for any  $x\in \mathcal{X}$ ,
$$
\left\| F _ {\mathbf {W}} (x) \right\| \leqslant R _ {\mathcal {X}} \exp \left(K _ {\sigma} R _ {\mathcal {W}}\right)
$$
$$
\| F _ {\mathbf {W}} (x) - F _ {\tilde {\mathbf {W}}} (x) \| \leqslant \frac {R _ {\mathcal {X}}}{R _ {\mathcal {W}}} \exp (2 K _ {\sigma} R _ {\mathcal {W}}) \| \mathbf {W} - \tilde {\mathbf {W}} \| _ {1, 1, \infty}.
$$
Proposition 5 (Covering number of the residual network parameter class). Let  $\mathcal{N}(\varepsilon)$  be the covering number of  $\mathcal{W}$  endowed with the distance associated to the  $(1,1,\infty)$ -norm (11). Then
$$
\log \mathcal {N} (\varepsilon) \leqslant d ^ {2} \log \left(\frac {1 6 d ^ {2} R _ {\mathcal {W}}}{\varepsilon}\right) + \frac {d ^ {4} K _ {\mathcal {W}} \log (4)}{\varepsilon}.
$$
The proof of Proposition 4 is a discrete analogous of Proposition 2. On the other hand, Proposition 5 can be proven as a consequence of Proposition 3, by showing the existence of an injective isometry from  $\mathcal{W}$  into a set of the form (7). Equipped with these two propositions, we are now ready to state the generalization bound for our class of residual neural networks.
Theorem 2 (Generalization bound for deep residual networks). Consider the class of neural networks  $\mathcal{F}_{\mathcal{W}} = \{F_{\mathbf{W}}, \mathbf{W} \in \mathcal{W}\}$ , where  $F_{\mathbf{W}}$  is given by (10) and  $\mathcal{W}$  by (12). Let  $\delta > 0$ .
$$
\mathcal {R} \left(\widehat {\mathbf {W}} _ {n}\right) \leqslant \widehat {\mathcal {R}} _ {n} \left(\widehat {\mathbf {W}} _ {n}\right) + B (d + 1) \sqrt {\frac {\log \left(R _ {\mathcal {W}} d n\right)}{n}} + B \frac {d ^ {2} \sqrt {K _ {\mathcal {W}}}}{n ^ {1 / 4}} + \frac {B}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}, \tag {13}
$$
where  $B$  is a constant depending on  $K_{\ell}, K_{\sigma}, R_{\mathcal{W}}, R_{\mathcal{X}}, R_{\mathcal{Y}}$ . More precisely,
$$
B = 6 \sqrt {2} K _ {\ell} \max \left(\frac {\exp (K _ {\sigma} R _ {\mathcal {W}})}{R _ {\mathcal {W}}}, 1\right) (R _ {\mathcal {X}} \exp (K _ {\sigma} R _ {\mathcal {W}}) + R _ {\mathcal {Y}}).
$$
We emphasize that this result is non-asymptotic and valid for any width  $d$  and depth  $L$ . Furthermore, the depth  $L$  does not appear in the upper bound (13). This should not surprise the reader since Theorem 1 can be seen as the deep limit  $L \to \infty$  of this result, hence we expect that our bound remains finite when  $L \to \infty$  (otherwise the bound of Theorem 1 would be infinite). However,  $L$  appears as a scaling factor in the definition of the neural network (10) and of the class of parameters (12). This is crucial for the depth independence to hold, as we will comment further on in the next section.
Furthermore, the depth independence comes at the price of a  $\mathcal{O}(n^{-1/4})$  convergence rate. Note that, by taking  $K_{\mathcal{W}} = 0$ , we obtain a generalization bound for weight-tied neural networks with a faster convergence rate in  $n$ , since the term in  $\mathcal{O}(n^{-1/4})$  vanishes.
4.2 Comparison with other bounds
As announced in Section 2, we now compare Theorem 2 with the results of Bartlett et al. (2017) and Golowich et al. (2018). Beginning by Bartlett et al. (2017), we first state a slightly weaker version of their result to match our notations and facilitate comparison.
Corollary 2 (corollary of Theorem 1.1 of Bartlett et al. (2017)). Consider the class of neural networks  $\mathcal{F}_{\tilde{\mathcal{W}}} = \{F_{\mathbf{W}},\mathbf{W}\in \tilde{\mathcal{W}}\}$ , where  $F_{\mathbf{W}}$  is given by (10) and  $\tilde{\mathcal{W}} = \{\mathbf{W}\in \mathbb{R}^{L\times d\times d},\| \mathbf{W}\|_{1,1,\infty}\leqslant R_{\mathcal{W}}\}$ .
Assume that  $L \geqslant R_{\mathcal{W}}$  and  $K_{\sigma} = 1$ , and let  $\gamma, \delta > 0$ . Consider  $(x,y), (x_1,y_1), \ldots, (x_n,y_n)$  drawn i.i.d. from any probability distribution over  $\mathbb{R}^d \times \{1, \ldots, d\}$  such that a.s.  $\| x \| \leqslant R_{\mathcal{X}}$ .
Then, with probability at least  $1 - \delta$ , for every  $\mathbf{W} \in \tilde{\mathcal{W}}$ ,
$$
\mathbb {P} \left(\underset {1 \leqslant j \leqslant d} {\arg \max } F _ {\mathbf {W}} (x) _ {j} \neq y\right) \leqslant \widehat {\mathcal {R}} _ {n} (\mathbf {W}) + C \frac {R _ {\mathcal {X}} R _ {\mathcal {W}} \exp \left(R _ {\mathcal {W}}\right) \log (d) \sqrt {L}}{\gamma \sqrt {n}} + \frac {C}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}, \tag {14}
$$
where  $\widehat{\mathcal{R}}_n(\mathbf{W})\leqslant n^{-1}\sum_{i = 1}^n\mathbf{1}_{F_{\mathbf{W}}(x_i)_{y_i}\leqslant \gamma +\max_{j\neq y_i}f(x_i)_j}$  and  $C$  is a universal constant.
We first note that the setting is slightly different from ours: they consider a large margin predictor for a multi-class classification problem, whereas we consider a general Lipschitz-continuous loss  $\ell$ . This being said, the model class is identical to ours, except for one notable difference: the constraint on the Lipschitz constant of the weights appearing in equation (12) is not required here.
Comparing (13) and (14), we see that our bound enjoys a better dependence on the depth  $L$  but a worse dependence on the width  $d$ . Regarding the depth, our bound (13) does not depend on  $L$ , whereas the bound (14) scales as  $\mathcal{O}(\sqrt{L})$ . This comes from the fact that we consider a smaller set of parameters (12), by adding the constraint on the Lipschitz norm of the weights. This constraint
allows us to control the complexity of our class of neural networks independently of depth, as long as  $K_{\mathcal{W}}$  is independent of  $L$ . If  $K_{\mathcal{W}}$  scales as  $\mathcal{O}(L)$ , which is the case for i.i.d. initialization schemes, our result also features a scaling in  $\mathcal{O}(\sqrt{L})$ . As for the width, Bartlett et al. (2017) achieve a better dependence by a subtle covering numbers argument that takes into account the geometry induced by matrix norms. Since our paper focuses on a depth-wise analysis by leveraging the similarity between residual networks and their infinite-depth counterpart, improving the scaling of our bound with width is left for future work. Finally, note that both bounds have a similar exponential dependence in  $R_{\mathcal{W}}$ .
As for Golowich et al. (2018), they consider non-residual neural networks of the form  $x \mapsto M_L \sigma(M_{L-1} \sigma(\ldots \sigma(M_1 x)))$ . These authors show that the generalization error of this class scales as
$$
\mathcal {O} \bigg (R _ {\mathcal {X}} \frac {\Pi_ {F} \sqrt {\log \left(\frac {\Pi_ {F}}{\pi_ {S}}\right)}}{n ^ {1 / 4}} \bigg),
$$
where  $\Pi_F$  is an upper-bound on the product of the Frobenius norms  $\prod_{k=1}^{L}\|M_k\|_F$  and  $\pi_S$  is a lower-bound on the product of the spectral norms  $\prod_{k=1}^{L}\|M_k\|$ . Under the assumption that both  $\Pi_F$  and  $\Pi_F/\pi_S$  are bounded independently of  $L$ , their bound is indeed depth-independent, similarly to ours. Interestingly, as ours, the bound presents a  $\mathcal{O}(n^{-1/4})$  convergence rate instead of the more usual  $\mathcal{O}(n^{-1/2})$ . However, the assumption that  $\Pi_F$  is bounded independently of  $L$  does not hold in our residual setting, since we have  $M_k = I + \frac{1}{L}W_k$  and thus we can lower-bound
$$
\prod_ {k = 1} ^ {L} \| M _ {k} \| _ {F} \geqslant \prod_ {k = 1} ^ {L} \left(\| I \| _ {F} - \frac {1}{L} \| M _ {k} \| _ {F}\right) \geqslant \left(\sqrt {d} - \frac {R _ {\mathcal {W}}}{L}\right) ^ {L} \approx d ^ {\frac {L}{2}} e ^ {- \frac {R _ {\mathcal {W}}}{\sqrt {d}}}.
$$
In our setting, it is a totally different assumption, the constraint that two successive weight matrices should be close to one another, which allows us to derive depth-independent bounds.
4.3 Numerical illustration
The bound of Theorem 2 features two quantities that depend on the class of neural networks, namely  $R_{\mathcal{W}}$  that bounds a norm of the weight matrices and  $K_{\mathcal{W}}$  that bounds the maximum difference between two successive weight matrices, i.e. the Lipschitz constant of the weights. The first one belongs to the larger class of norm-based bounds that has been extensively studied (see, e.g., Neyshabur et al., 2015). We are therefore interested in getting a better understanding of the role of the second quantity, which is much less common, in the generalization ability of deep residual networks.
To this aim, we train deep residual networks (10) (of width  $d = 30$  and depth  $L = 1000$ ) on MNIST. We prepend the network with an initial weight matrix to project the data  $x$  from dimension 768 to dimension 30, and similarly postpend it with another matrix to project the output  $F_{\mathbf{W}}(x)$  into dimension 10 (i.e. the number of classes in MNIST). Finally, we consider two training settings: either the initial and final matrices are trained, or they are fixed random projections. We use the initialization scheme outlined in Section 4.1. Further experimental details are postponed to the Appendix.
We report in Figure 1a the generalization gap of the trained networks, that is, the difference between the test and train errors (in terms of cross entropy loss), as a function of the maximum Lipschitz constant of the weights  $\sup_{0\leqslant k\leqslant L - 1}(\| W_{k + 1} - W_k\|_\infty)$ . We observe a positive correlation between these two quantities. To further analyze the relationship between the Lipschitz constant of the weights and the generalization gap, we then add the penalization term  $\lambda \cdot \left(\sum_{k = 0}^{L - 1}\| W_{k + 1} - W_k\| _F^2\right)^{1 / 2}$  to the loss, for some  $\lambda \geqslant 0$ . The obtained generalization gap is reported in Figure 1b as a function of  $\lambda$ . We observe that this penalization allows to reduce the generalization gap. These two observations go in support of the fact that a smaller Lipschitz constant improves the generalization power of deep residual networks, in accordance with Theorem 2.
However, note that we were not able to obtain an improvement on the test loss by adding the penalization term. This is not all too surprising since previous work has investigated a related penalization, in terms of the Lipschitz norm of the layer sequence  $(H_{k})_{0\leqslant k\leqslant L}$ , and was similarly not able to report any improvement on the test loss (Kelly et al., 2020).
Finally, the proposed penalization term slightly departs from the theory that involves  $\sup_{0\leqslant k\leqslant L - 1}(\| W_{k + 1} - W_k\|_\infty)$ . This is because the maximum norm is too irregular to be used
in practice since, at any one step of gradient descent, it only impacts the maximum weights and not the others. As an illustration, Figure 2 shows the generalization gap when penalizing with the maximum max-norm  $\sup_{0\leqslant k\leqslant L - 1}(\| W_{k + 1} - W_k\|_\infty)$  and the  $L_{2}$  norm of the max-norm  $\left(\sum_{k = 0}^{L - 1}\| W_{k + 1} - W_k\|_\infty^2\right)^{1 / 2}$ . The factor  $\lambda$  is scaled appropriately to reflect the scale difference of the penalizations. The results are mixed: the  $L_{2}$  norm of the max-norm is effective contrarily to the maximum max-norm. Further investigation of the properties of these norms is left for future work.
We provide a generalization bound that applies to a wide range of parameterized ODEs. As a consequence, we obtain the first generalization bounds for time-independent and time-dependent neural ODEs in supervised learning tasks. By discretizing our reasoning, we also provide a bound for a class of deep residual networks. Understanding the approximation and optimization properties of this class of neural networks is left for future work. Another intriguing extension is to relax the assumption of linearity of the dynamics at time  $t$  with respect to  $\theta_{i}(t)$ , that is, to consider a general formulation  $dH_{t} = \sum_{i=1}^{m} f_{i}(H_{t}, \theta_{i}(t))$ . In the future, it should also be interesting to extend our results to the more involved case of neural SDEs, which have also been found to be deep limits of a large class of residual neural networks (Cohen et al., 2021; Marion et al., 2022).
The author is supported by a grant from Région Île-de-France, by a Google PhD Fellowship, and by MINES Paris - PSL. The author thanks Eloise Berthier, Gérard Biau, Adeline Germanian, Clément Mantoux, and Jean-Philippe Vert for inspiring discussions, thorough proofreading and suggestions on this paper.
V. Arnold. Ordinary Differential Equations. Springer Textbook. Springer Berlin Heidelberg, 1992.
F. Bach. Learning theory from first principles, 2023. Book draft. URL: https://www.di.ens.fr/~fbach/ltfp_book.pdf (version: 2023-02-05).
P. L. Bartlett, D. J. Foster, and M. J. Telgarsky. Spectrally-normalized margin bounds for neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
P. L. Bartlett, N. Harvey, C. Liaw, and A. Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63): 1-17, 2019.
L. Béthane, T. Boissin, M. Serrurier, F. Mamalet, C. Friedrich, and A. G. Sanz. Pay attention to your loss : understanding misconceptions about lipschitz neural networks. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022.
M. Biloš, J. Sommer, S. S. Rangapuram, T. Januschowski, and S. Gunnemann. Neural flows: Efficient alternative to neural ODEs. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
A.-S. Cohen, R. Cont, A. Rossier, and R. Xu. Scaling properties of deep residual networks. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 2039-2048. PMLR, 2021.
P. Deuflhard and S. Röblitz. Parameter identification in ODE models. In A Guide to Numerical Modelling in Systems Biology, pages 89-138. Springer International Publishing, 2015.
W. E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017. ISSN 2194-671X.
A. Fermanian, P. Marion, J.-P. Vert, and G. Biau. Framing RNN as a kernel method: A neural ODE approach. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
P. Gierjatowicz, M. Sabate-Vidales, D. Šiška, L. Szpruch, and Z. Zurič. Robust pricing and hedging via neural sdes. arXiv preprint arXiv:2007.04154, 2020.
N. Golowich, A. Rakhlin, and O. Shamir. Size-independent sample complexity of neural networks. In S. Bubeck, V. Perchet, and P. Rigollet, editors, Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research. PMLR, 2018.
L.-A. Gottlieb, A. Kontorovich, and R. Krauthgamer. Efficient regression in metric spaces via approximate lipschitz extension. IEEE Transactions on Information Theory, 63(8):4838-4849, 2017.
Organization of the Appendix Section A contains the proofs of the results of the main paper. Section B contains the details of the numerical illustrations presented in Section 4.3.
The function
$$
(t, h) \mapsto \sum_ {i = 1} ^ {m} \theta_ {i} (t) f _ {i} (h)
$$
is locally Lipschitz-continuous with respect to its first variable and globally Lipschitz-continuous with respect to its second variable. Therefore, the existence and uniqueness of the solution of the initial value problem (5) for  $t \geq 0$  comes as a consequence of the Picard-Lindelöf theorem (see, e.g., Luk, 2017 for a self-contained presentation and Arnold, 1992 for a textbook).
For  $x \in \mathcal{X}$ , let  $H$  be the solution of the initial value problem (5) with parameter  $\theta$  and with the initial condition  $H_0 = x$ . Let us first upper-bound  $\| f_i(H_t)\|$  for all  $i \in \{1, \dots, m\}$  and  $t > 0$ . To this aim, for  $t \geqslant 0$ , we have
$$
\begin{array}{l} \left\| H _ {t} - H _ {0} \right\| = \left\| \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} \theta_ {i} (s) f _ {i} (H _ {s}) d s \right\| \\ \leqslant \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \theta_ {i} (s) | \| f _ {i} (H _ {0}) \| d s + \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \theta_ {i} (s) | \| f _ {i} (H _ {s}) - f _ {i} (H _ {0}) \| d s \\ \leqslant M \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \theta_ {i} (s) | d s + K _ {f} \int_ {0} ^ {t} \left(\| H _ {s} - H _ {0} \| \sum_ {i = 1} ^ {m} | \theta_ {i} (s) |\right) d s \\ \leqslant t M R _ {\Theta} + K _ {f} R _ {\Theta} \int_ {0} ^ {t} \| H _ {s} - H _ {0} \| d s. \\ \end{array}
$$
Next, Gronwall's inequality yields, for  $t \in [0,1]$ ,
$$
\left\| H _ {t} - H _ {0} \right\| \leqslant t M R _ {\Theta} \exp (t K _ {f} R _ {\Theta}) \leqslant M R _ {\Theta} \exp (K _ {f} R _ {\Theta}).
$$
$$
\left\| H _ {t} \right\| \leqslant \left\| H _ {0} \right\| + \left\| H _ {t} - H _ {0} \right\| \leqslant R _ {\mathcal {X}} + M R _ {\Theta} \exp \left(K _ {f} R _ {\Theta}\right),
$$
yielding the first result of the proposition. Furthermore, for any  $i \in \{1, \dots, m\}$ ,
$$
\left\| f _ {i} \left(H _ {t}\right) \right\| \leqslant \left\| f _ {i} \left(H _ {t}\right) - f _ {i} \left(H _ {0}\right) \right\| + \left\| f _ {i} \left(H _ {0}\right) \right\| \leqslant M \left(K _ {f} R _ {\Theta} \exp \left(K _ {f} R _ {\Theta}\right) + 1\right) =: C.
$$