SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: In episode  $n$  we split the horizon  $T$  into intervals of  $\Delta_{n}$  time. We hallucinate the trajectory in every interval and select time  $t_{n,i}$  in the interval  $i$  where the uncertainty on the hallucinated trajectory is the highest.
$$
t_{n,i}\stackrel {\mathrm{def}}{=}(i - 1)\Delta_{n} + \operatorname *{argmax}_{0\leq t\leq \Delta_{n}}\| \boldsymbol{\sigma}_{n - 1}(\widehat{\boldsymbol{z}}_{n}(t))\|
$$
$$
\begin{array}{l} \text {s . t .} \quad \dot {\hat {\boldsymbol {x}}} _ {n} = \boldsymbol {f} _ {n} (\hat {\boldsymbol {x}} _ {n}, \boldsymbol {\pi} _ {n} (\hat {\boldsymbol {x}} _ {n})) \end{array}
$$
$$
\widehat {\boldsymbol {x}} _ {n} (0) = \boldsymbol {x} _ {n} ((i - 1) \Delta_ {n})
$$
For this MSS, we show in Appendix A the upper bound
$$
\mathcal {I} _ {N} \left(\boldsymbol {f} ^ {*}, S _ {n} ^ {\mathrm {A D P}}\right) \leq \max  _ {\substack {\pi_ {1}, \dots , \pi_ {N} \\ \pi_ {n} \in \Pi}} 9 \sum_ {n = 1} ^ {N} \frac {T}{m _ {n}} \sum_ {i = 1} ^ {m _ {n}} \| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} \left(t _ {n, i}\right)\right) \| ^ {2}. \tag{7}
$$
The model complexity upper bound for the adaptive MSS in Equation (7) does not have the  $L_{\sigma^2}$ -dependent term from the equidistant MSS (6) and only depends on the sum of uncertainties at the collected points. Further, the number of points we collect in episode  $n$  is  $\mathcal{O}(\beta_n)$ , and  $\beta_{n}$ , e.g., if we model the dynamics with GPs with radial basis function (RBF) kernel, is of order polylog(n) (c.f., Lemma 2).
3.2 Modeling dynamics with a Gaussian Process
We can prove sublinear regret for the proposed three MSSs coupled with optimistic exploration when we model the dynamics using GPs with standard kernels such as RBF, linear, etc. We consider the vector-valued dynamics model  $\pmb{f}^{*}(\pmb{z}) = (f_{1}^{*}(\pmb{z}),\dots,f_{d_{x}}^{*}(\pmb{z}))$ , where scalar-valued functions  $f_{j}^{*}\in \mathcal{H}_{k}$  are elements of a Reproducing Kernel Hilbert Space (RKHS)  $\mathcal{H}_k$  with kernel function  $k$  and their norm is bounded  $\| f_j^*\| _k\leq B$ . We write  $\pmb{f}^{*}\in \mathcal{H}_{k,B}^{d_x}\stackrel {\mathrm{def}}{=}\{(f_1,\ldots ,f_{d_x})\mid \| f_j\| _k\leq B\}$ .
To learn  $f^{*}$  we fit a GP model with a zero mean and kernel  $k$  to the collected data  $\mathcal{D}_{1:n}$ . For ease of notation, we denote by  $\dot{\pmb{y}}_{1:n}^{j}$  the concatenated  $j$ -th dimension of the state derivative observations from  $\mathcal{D}_{1:n}$ . The posterior means and variances of the GP model have a closed form (Williams and Rasmussen (2006)):
$$
\mu_ {n, j} (\boldsymbol {z}) = \boldsymbol {k} _ {n} ^ {\top} (\boldsymbol {z}) \left(\boldsymbol {K} _ {n} + \sigma^ {2} \boldsymbol {I}\right) ^ {- 1} \dot {\boldsymbol {y}} _ {1: n} ^ {j}
$$
$$
\sigma_ {n, j} ^ {2} (\pmb {z}) = k (\pmb {z}, \pmb {z}) - \pmb {k} _ {n} ^ {\top} (\pmb {z}) \left(\pmb {K} _ {n} + \sigma^ {2} \pmb {I}\right) ^ {- 1} \pmb {k} _ {n} (\pmb {z}),
$$
where we write  $\pmb{K}_n = [k(\pmb{z}_l, \pmb{z}_m)]_{\pmb{z}_l, \pmb{z}_m \in \mathcal{D}_{1:n}}$ , and  $\pmb{k}_n(\pmb{z}) = [k(\pmb{z}_l, \pmb{z})]_{\pmb{z}_l \in \mathcal{D}_{1:n}}$ . The posterior means and variances with the right scaling factor  $\beta_n(\delta)$  satisfy the all-time well-calibrated Assumption 2.
Lemma 2 (Lemma 3.6 from Rothfuss et al. (2023)). Let  $\pmb{f}^{*} \in \mathcal{H}_{k,B}^{d_{x}}$  and  $\delta \in (0,1)$ . Then there exist  $\beta_{n}(\delta) \in \mathcal{O}(\sqrt{\gamma_{n} + \log(1 / \delta)})$  such that the confidence sets  $\mathcal{M}_n$  built from the triplets  $(\pmb{\mu}_n, \pmb{\sigma}_n, \beta_n(\delta))$  form an all-time well-calibrated statistical model.
Here,  $\gamma_{n}$  is the maximum information gain after observing  $n$  points (Srinivas et al., 2009), as defined in Appendix A, where we also provide the rates for common kernels. For example, for the RBF kernel,  $\gamma_{n} = \mathcal{O}\left(\log (n)^{d_{x} + d_{u} + 1}\right)$ .
Finally, we show sublinear regret for the proposed MSSs for the case when we model dynamics with GPs. The proof of Theorem 3 is provided in Appendix A.
Theorem 3. Assume that  $\pmb{f}^{*} \in \mathcal{H}_{k,B}^{d_{x}}$ , the observation noise is i.i.d.  $\mathcal{N}(0; \sigma^2 I)$ , and  $\| \cdot \|$  is the Euclidean norm. We model  $\pmb{f}^{*}$  with the GP model. The regret for different MSSs is with probability at least  $1 - \delta$  bounded by
$$
R _ {N} \left(S ^ {O R A}\right) \leq \mathcal {O} \left(\beta_ {N} T ^ {2} e ^ {L _ {f} (1 + L _ {\pi}) T} \sqrt {N \gamma_ {N}}\right), \quad m _ {n} ^ {O R A} = 1
$$
$$
R _ {N} \left(S ^ {E Q I}\right) \leq \mathcal {O} \left(\beta_ {N} T ^ {2} e ^ {L _ {\boldsymbol {f}} (1 + L _ {\pi}) T} \sqrt {N (\gamma_ {N} + \log (N))}\right), \quad m _ {n} ^ {E Q I} = n
$$
$$
R _ {N} (S ^ {A D P}) \leq \mathcal {O} \left(\beta_ {N} T ^ {2} e ^ {L _ {f} (1 + L _ {\pi}) T} \sqrt {N \gamma_ {N}}\right), \quad m _ {n} ^ {A D P} = \mathcal {O} (\beta_ {n})
$$
The optimistic exploration coupled with any of the proposed MSSs achieves regret that depends on the maximum information gain. All regret bounds from Theorem 3 are sublinear for common kernels, like linear and RBF. The bound for the adaptive MSS with RBF kernel is  $\mathcal{O}\left(\sqrt{N\log(N)^{2(d_x + d_u + 1)}}\right)$ , where we hide the dependence on  $T$  in the  $\mathcal{O}$  notation. We reiterate that while the number of observations per episode of the oracle MSS is optimal, we cannot implement the oracle MSS in practice. In contrast, the equidistant MSS is easy to implement, but the number of measurements grows linearly with episodes. Finally, adaptive MSS is practical, i.e., easy to implement, and requires only a few measurements per episode, i.e., polylog(n) in episode  $n$  for RBF.
Summary OCoRL consists of two key and orthogonal components;  $(i)$  optimistic policy selection and  $(ii)$  measurement selection strategies (MSSs). In optimistic policy selection, we optimistically, w.r.t. plausible dynamics, plan a trajectory and rollout the resulting policy. We study different MSSs, such as the typical equidistant MSS, for data collection within the framework of continuous time modeling. Furthermore, we propose an adaptive MSS that measures data where we have the highest uncertainty on the planned (hallucinated) trajectory. We show that OCoRL suffers no regret for the equidistant, adaptive, and oracle MSSs.
Model-based Reinforcement Learning Model-based reinforcement learning (MBRL) has been an active area of research in recent years, addressing the challenges of learning and exploiting environment dynamics for improved decision-making. Among the seminal contributions, Deisenroth and Rasmussen (2011) proposed the PILCO algorithm which uses Gaussian processes for learning the system dynamics and policy optimization. Chua et al. (2018) used deep ensembles as dynamics models. They coupled MPC (Morari and Lee, 1999) to efficiently solve high dimensional tasks with considerably better sample complexity compared to the state-of-the-art model-free methods SAC (Haarnoja et al., 2018), PPO (Schulman et al., 2017), and DDPG (Lillicrap et al., 2015). The aforementioned model-based methods use more or less greedy exploitation that is provably optimal only in the case of the linear dynamics (Simchowitz and Foster, 2020). Exploration methods based on Thompson sampling (Dearden et al., 2013; Chowdhury and Gopalan, 2019) and Optimism (Auer et al., 2008; Abbasi-Yadtori and Szepesvári, 2011; Luo et al., 2018; Curi et al., 2020), however, provably converge to the optimal policy also for a large class of nonlinear dynamics if modeled with GPs. Among the discrete-time RL algorithms, our work is most similar to the work of Curi et al. (2020) where they use the reparametrization trick to explore optimistically among all statistically plausible discrete dynamical models.
Continuous-time Reinforcement Learning Reinforcement learning in continuous-time has been around for several decades (Doya, 2000; Vrabie and Lewis, 2008, 2009; Vamvoudakis et al., 2009). Recently, the field has gained more traction following the work on Neural ODEs by Chen et al. (2018). While physics biases such as Lagrangian (Cranmer et al., 2020) or Hamiltonian (Greydanus et al., 2019) mechanics can be enforced in continuous-time modeling, different challenges such
as vanishing  $Q$ -function (Tallec et al., 2019) need to be addressed in the continuous-time setting. Yildiz et al. (2021) introduce a practical episodic model-based algorithm in continuous time. In each episode, they use the learned mean estimate of the ODE model to solve the optimal control task with a variant of a continuous-time actor-critic algorithm. Compared to Yildiz et al. (2021) we solve the optimal control task using the optimistic principle. Moreover, we thoroughly motivate optimistic planning from a theoretical standpoint. Lutter et al. (2021) introduce a continuous fitted value iteration and further show a successful hardware application of their continuous-time algorithm. A vast literature exists for continuous-time RL with linear systems (Modares and Lewis, 2014; Wang et al., 2020), but few, only for linear systems, provide theoretical guarantees (Mohammadi et al., 2021; Basei et al., 2022). To the best of our knowledge, we are the first to provide theoretical guarantees of convergence to the optimal cost in continuous time for a large class of RKHS functions.
Aperiodic strategies Aperiodic MSSs and controls (event and self-triggered control) are mostly neglected in RL since most RL works predominantly focus on discrete-time modeling. There exists a considerable amount of literature on the event and self-triggered control (Astrom and Bernhardsson, 2002; Anta and Tabuada, 2010; Heemels et al., 2012). However, compared to periodic control, its theory is far from being mature (Heemels et al., 2021). In our work, we assume that we can control the system continuously, and rather focus on when to measure the system instead. The closest to our adaptive MSS is the work of Du et al. (2020), where they empirically show that by optimizing the number of interaction times, they can achieve similar performance (in terms of cost) but with fewer interactions. Compared to us, they do not provide any theoretical guarantees. Umlauft and Hirche (2019); Lederer et al. (2021) consider the non-episodic setting where they can continuously monitor the system. They suggest taking measurements only when the uncertainty of the learned model on the monitored trajectory surpasses the boundary ensuring stability. They empirically show, for feedback linearizable systems, that by applying their strategy the number of measurements reduces drastically and the tracking error remains bounded. Compared to them, we consider general dynamical systems and also don't assume continuous system monitoring.
We now empirically evaluate the performance of OCoRL on several environments. We test OCoRL on Cancer Treatment and Glucose in blood systems from Howe et al. (2022), Pendulum, Mountain Car and Cart Pole from Brockman et al. (2016), Bicycle from Polack et al. (2017), Furuta Pendulum from Lutter et al. (2021) and Quadrotor in 2D and 3D from Nonami et al. (2010). The details of the systems' dynamics and tasks are provided in Appendix C.
Comparison methods To make the comparison fair, we adjust methods so that they all collect the same number of measurements per episode. For the equidistant setting, we collect  $M$  points per episode (we provide values of  $M$  for different systems in Appendix C). For the adaptive MSS, we assume  $\Delta_{n} \geq T$ , and instead of one measurement per episode we collect a batch of  $M$  measurements such that they (as a batch) maximize the variance on the hallucinated trajectory. To this end, we consider the Greedy Max Determinant and Greedy Max Kernel Distance strategies of Holzmuller et al. (2022). We provide details of the adaptive strategies in Appendix C. We compare OCoRL with the described MSSs to the optimal discrete-time zero hold control, where we assume the access to the true discretized dynamics  $\pmb{f}_d^*(\pmb{x}, \pmb{u}) = \pmb{x} + \int_0^{T/(M-1)} \pmb{f}^*(\pmb{x}(t), \pmb{u}) dt$ . We further also compare with the best continuous-time control policy, i.e., the solution of Equation (1).
Does the continuous-time control policy perform better than the discrete-time control policy? In the first experiment, we test whether learning a continuous-time model from the finite data coupled with a continuous-time control policy on the learned model can outperform the discrete-time zero-order hold control on the true system. We conduct the experiment on all environments and report the cost after running OCoRL for a few tens of episodes (the exact experimental details are provided in Appendix C). From Table 1, we conclude that the OCoRL outperforms the discrete-time zero-order hold control on the true model on every system if we use the adaptive MSS, while achieving lower cost on 7 out of 9 systems if we measure the system equidistantly.
Table 1: OCoRL with adaptive MSSs achieves lower final cost  $C(\pi_N, \pmb{f}^*)$  compared to the discrete-time control on the true system on all tested environments while converging towards the best continuous-time control policy. While equidistant MSS achieves higher cost compared to the adaptive MSS, it still outperforms the discrete-time zero-order hold control on the true model for most systems.
System	Known True Model	Optimistic Exploration with different MSS
Continuous time OC	Discrete zero-order hold OC	Max Kernel Distance	Max Determinant	Equidistant
Cancer Treatment	20.57	21.05	20.70 ± 0.06	20.68 ± 0.05	21.05 ± 1.60
Glucose in Blood	15.23	15.30	15.23 ± 0.01	15.24 ± 0.01	15.25 ± 0.01
Pendulum	20.16	20.59	20.20 ± 0.02	20.20 ± 0.02	20.29 ± 0.03
Mountain Car	34.63	35.04	34.63 ± 0.01	34.63 ± 0.01	34.64 ± 0.01
Cart Pole	17.49	19.96	17.52 ± 0.04	17.53 ± 0.04	17.63 ± 0.05
Bicycle	9.45	10.24	9.53 ± 0.02	9.53 ± 0.03	9.67 ± 0.05
Furuta Pendulum	23.31	25.11	23.64 ± 0.22	23.52 ± 0.18	314.77 ± 411.01
Quadrotor 2D	3.54	4.01	3.54 ± 0.01	3.54 ± 0.01	3.57 ± 0.01
Quadrotor 3D	7.38	7.84	7.51 ± 0.21	7.54 ± 0.28	9.41 ± 1.43
Does the adaptive MSS perform better than equidistant MSS? We compare the adaptive and equidistant MSSs on all systems and observe that the adaptive MSSs consistently perform better than the equidistant MSS. To better illustrate the difference between the adaptive and equidistant MSSs we study a 2-dimensional Pendulum system (c.f., Figure 2). First, we see that if we use adaptive MSSs we consistently achieve lower per-episode costs during the training. Second, we observe that while equidistant MSS spreads observations equidistantly in time and collects lots of measurements with almost the same state-action input of the dynamical system, the adaptive MSS spreads the measurements to have diverse state-action input pairs for the dynamical system on the executed trajectory and collects higher quality data.
Figure 2: All MSSs coupled with continuous-time control achieve lower cost than the optimal discrete zero-order hold control on the true model. Adaptive MSSs (Greedy Max Kernel Distance and Greedy Max Determinant) reduce the suffered cost considerably faster than equidistant MSS and are converging towards the best possible continuous-time control. Whereas equidistant MSS spreads the measurements uniformly over time, the adaptive MSSs spread the data over the state-action space (dynamical system's input) and collect higher quality data.
Does optimism help? For the final experiment, we examine whether planning optimistically helps. In particular, we compare our planning strategy to the mean planner that is also used by Yildiz et al. (2021). The mean planner solves the optimal control problem greedily with the learned mean model  $\mu_{n}$  in every episode  $n$ . We evaluate the performance of this model on the Pendulum system for all MSSs. We observe that planning optimistically results in reducing the cost faster and achieves better performance for all MSSs.
Figure 3: Both greedy and optimistic continuous-time control strategies outperform the discrete-time zero-order hold control on the true model. However, the optimistic strategy is particularly notable as it expedites cost reduction across all MSSs.
We introduce a model-based continuous-time RL algorithm OCoRL that uses the optimistic paradigm to provably achieve sublinear regret, with respect to the best possible continuous-time performance, for several MSSs if modeled with GPs. Further, we develop a practical adaptive MSS that, compared to the standard equidistant MSS, drastically reduces the number of measurements per episode while retaining the regret guarantees. Finally, we showcase the benefits of continuous-time compared to discrete-time modeling in several environments (c.f. Figure 1, Table 1), and demonstrate the benefits of planning with optimism compared to greedy planning.
In this work, we considered the setting with deterministic dynamics where we obtain a noisy measurement of the state's derivative. We leave the more practical setting, where only noisy measurements of the state, instead of its derivative, are available as well as stochastic, delayed differential equations, and partially observable systems to future work.
Our aim with this work is to catalyze further research within the RL community on continuous-time modeling. We believe that this shift in perspective could lead to significant advancements in the field and look forward to future contributions.
We would like to thank Yarden As, Scott Sussex and Dominik Baumann for their feedback. This project has received funding from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, and the Microsoft Swiss Joint Research Center.
Doya, K. (2000). Reinforcement learning in continuous time and space. Neural computation, 12(1):219-245.
Du, J., Futoma, J., and Doshi-Velez, F. (2020). Model-based reinforcement learning for semimarkov decision processes with neural odes. Advances in Neural Information Processing Systems, 33:19805-19816.
Engquist, B., Li, X., Ren, W., Vanden-Eijnden, E., et al. (2007). Heterogeneous multiscale methods: a review. Communications in Computational Physics, 2(3):367-450.
Gäfvert, M. (2016). Modelling the furuta pendulum. Department of Automatic Control, Lund Institute of Technology (LTH).
Ghysels, E., Santa-Clara, P., and Valkanov, R. (2006). Predicting volatility: getting the most out of return data sampled at different frequencies. Journal of Econometrics, 131(1-2):59-95.
Greydanus, S., Dzamba, M., and Yosinski, J. (2019). Hamiltonian neural networks. Advances in neural information processing systems, 32.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905.
Hartman, P. (2002). Ordinary differential equations. SIAM.
Heemels, W., Johansson, K. H., and Tabuada, P. (2021). Event-triggered and self-triggered control. In Encyclopedia of Systems and Control, pages 724-730. Springer.
Heemels, W. P., Johansson, K. H., and Tabuada, P. (2012). An introduction to event-triggered and self-triggered control. In 2012 IEEE 51st IEEE conference on decision and control (cdc), pages 3270-3285. IEEE.
Holzmuller, D., Zaverkin, V., Kastner, J., and Steinwart, I. (2022). A framework and benchmark for deep batch active learning for regression. arXiv preprint arXiv:2203.09410.
Howe, N., Dufort-Labbé, S., Rajkumar, N., and Bacon, P.-L. (2022). Myriad: a real-world testbed to bridge trajectory optimization and deep learning. Advances in Neural Information Processing Systems, 35:29801-29815.
Jacobson, D. H. and Mayne, D. Q. (1970). Differential dynamic programming. Number 24. Elsevier Publishing Company.
Jones, D. S., Plank, M., and Sleeman, B. D. (2009). Differential equations and mathematical biology. CRC press.
Kaandorp, G. C. and Koole, G. (2007). Optimal outpatient appointment scheduling. *Health care management science*, 10:217-229.
Kakade, S., Krishnamurthy, A., Lowrey, K., Ohnishi, M., and Sun, W. (2020). Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems, 33:15312-15325.
Kelly, M. (2017). An introduction to trajectory optimization: How to do your own direct collocation. SIAM Review, 59(4):849-904.
Khalil, H. K. (2015). Nonlinear control, volume 406. Pearson New York.
Kirschner, J. and Krause, A. (2018). Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory, pages 358-384. PMLR.
Knowles, I. and Renka, R. J. (2014). Methods for numerical differentiation of noisy data. *Electron. J. Differ. Equ.*, 21:235-246.
Knowles, I. and Wallace, R. (1995). A variational method for numerical differentiation. Numerische Mathematik, 70:91-110.
Wagner, J., Mazurek, P., Miekina, A., and Morawski, R. Z. (2018). Regularised differentiation of measurement data in systems for monitoring of human movements. Biomedical Signal Processing and Control, 43:265-277.
Wang, H., Zariphopoulou, T., and Zhou, X. Y. (2020). Reinforcement learning in continuous time and space: A stochastic control approach. The Journal of Machine Learning Research, 21(1):8145-8178.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA.
Williams, G., Aldrich, A., and Theodorou, E. A. (2017). Model predictive path integral control: From theory to parallel computation. Journal of Guidance, Control, and Dynamics, 40(2):344-357.
Yildiz, C., Heinonen, M., and Lahdesmäki, H. (2021). Continuous-time model-based reinforcement learning. In International Conference on Machine Learning, pages 12009-12018. PMLR.
A Theory 17
A.1 Bounding Regret with the Model Complexity 17
A.2 Bounding Model Complexities 18
A.3 Bounding Measurement Uncertainty of GPs 21
A.4 Useful Facts and Inequalities 22
B Solving Optimistic Optimal Control Problem 23
B.1 Application to Arbitrary Discretizations 23
C Experimental Setup 26
C.1 System's dynamics 26
C.2 System's tasks 27
C.3 Adaptive MSSs 27
C.4 Episodes Hyperparameters 28
C.5 Practical Implementation 28
D Additional Experiments on Number of Measurements 29
A Theory
We begin in Appendix A.1 by deriving the cumulative regret bound of Proposition 1 which depends on the model complexity. In Appendix A.2, we bound the model complexities of the presented MSSs. Finally, in Appendix A.3, we bound the measurement uncertainty when a GP is used as the statistical model, yielding sublinear regret guarantees for many commonly used kernels. We include useful facts and inequalities in Appendix A.4.
A.1 Bounding Regret with the Model Complexity
The main goal of this section is to derive Proposition 1 which relates the cumulative regret  $R_{N}(S)$  to the model complexity  $\mathcal{I}_N(\pmb {f}^*,S)$ . In what follows, the quantifier "with high probability" refers to "with probability at least  $1 - \delta$  as in Definition 2".
Lemma 4. Assuming  $\pmb{x}_n(0) = \widehat{\pmb{x}}_n(0)$ , the distance between the true and the hallucinated trajectory at any time  $t \geq 0$  is bounded with high probability by
$$
\left\| \widehat {\boldsymbol {x}} _ {n} (t) - \boldsymbol {x} _ {n} (t) \right\| \leq 2 \beta_ {n} e ^ {L _ {f} (1 + L _ {\pi}) t} \int_ {0} ^ {t} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\boldsymbol {z} _ {n} (s)\right) \right\| d s \tag {8}
$$
$$
\left\| \widehat {\boldsymbol {x}} _ {n} (t) - \boldsymbol {x} _ {n} (t) \right\| \leq 2 \beta_ {n} e ^ {L _ {f} (1 + L _ {\pi}) t} \int_ {0} ^ {t} \left\| \boldsymbol {\sigma} _ {n - 1} \left(\widehat {\boldsymbol {z}} _ {n} (s)\right) \right\| d s. \tag {9}
$$