SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Y. Jennifer Sun *
ys7849@princeton.edu
Stephen Newman *
sn9581@princeton.edu
Princeton University & Google DeepMind
Linear Quadratic Regulator (LQR) and Linear Quadratic Gaussian (LQG) control are foundational and extensively researched problems in optimal control. We investigate LQR and LQG problems with semi-adversarial perturbations and time-varying adversarial bandit loss functions. The best-known sublinear regret algorithm of Gradu et al. [2020] has a  $T^{\frac{3}{4}}$  time horizon dependence, and the authors posed an open question about whether a tight rate of  $\sqrt{T}$  could be achieved. We answer in the affirmative, giving an algorithm for bandit LQR and LQG which attains optimal regret (up to logarithmic factors) for both known and unknown systems. A central component of our method is a new scheme for bandit convex optimization with memory, which is of independent interest.
Linear-Quadratic Regulator (LQR) and the more general Linear-Gaussian (LQG) control problems have been extensively studied in the field of control theory due to their wide range of applications and admittance of analytical solutions by the seminal works of Bellman [1954] and Kalman [1960]. LQR and LQG control problems study the design of a feedback control policy for a linear dynamical system with the goal of minimizing cumulative, possibly time-varying quadratic costs. The discrete version of the problem studies the control of the following linear dynamical system governed by dynamics  $(A,B,C)^{1}$ :
$$
\mathbf {x} _ {t + 1} = A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}, \mathbf {y} _ {t} = C \mathbf {x} _ {t} + \mathbf {e} _ {t},
$$
where at time  $t$ ,  $\mathbf{x}_t$  represents the system's state,  $\mathbf{u}_t$  represents the control exerted on the system, and  $\{\mathbf{w}_t\}_{t=1}^T$  represents a sequence of i.i.d. centered Gaussian perturbations injected to the system. For a simple example of the model, consider the problem of controlling an elevator to reach the desired floors while minimizing the discomfort (due to acceleration or deceleration) and the energy consumption. In this case, the state  $\mathbf{x}_t$  would be a vector consisting of the elevator's position and velocity at time  $t$ . The control input  $\mathbf{u}_t$  is the force applied by the elevator motor at time  $t$ .  $\mathbf{w}_t$  represents process noise at time  $t$ , caused possibly by unexpected mechanical disturbances. In the generality of LQG, the system's states are not accessible. In the elevator example, one may not have the access to the measurement of the elevator's velocity or the precise position of the elevator. Instead, the algorithm has access to an observation  $\mathbf{y}_t$ , which is a linear transformation of state perturbed by a sequence  $\{\mathbf{e}_t\}_{t=1}^T$  of i.i.d. centered Gaussian noises, i.e. the elevator's observed position. The cost is
<sup>1</sup>The LQR/LQG dynamics can be generalized to time-varying linear dynamical systems. Here we restrict ourselves to linear time-invariant systems for simplicity.
a quadratic function of both the observation and the control exerted. The goal in LQR/LQG problems is to find a control policy  $\pi$  in some policy class  $\Pi$  that minimizes the cumulative cost over a finite time horizon  $T$ . With  $\mathbf{y}_t^\pi$ ,  $\mathbf{u}_t^\pi$  denoting the observation and control at time  $t$  resulted from executing policy  $\pi$ , the objective is formally given by
$$
\underset {\pi \in \Pi} {\text {m i n i m i z e}} J _ {T} (\pi) \stackrel {{\text {d e f}}} {{=}} \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {y} _ {t} ^ {\pi}, \mathbf {u} _ {t} ^ {\pi}\right) = \sum_ {t = 1} ^ {T} \mathbf {y} _ {t} ^ {\pi^ {\top}} Q _ {t} \mathbf {y} _ {t} ^ {\pi} + \mathbf {u} _ {t} ^ {\pi^ {\top}} R _ {t} \mathbf {u} _ {t} ^ {\pi},
$$
where  $Q_{t}, R_{t}$  are some possibly time-varying state-weighting and control-weighting matrices, respectively. In the elevator example, if our goal is to control the elevator to reach the desired floors while minimizing the discomfort and the energy consumption, the cost function at every time step  $t$  may take the form  $c_{t}(\mathbf{y}_{t},\mathbf{u}_{t}) = c_{1}\| \mathbf{y}_{t} - z_{\mathrm{goal}}\|_{2}^{2} + c_{2}\| \mathbf{u}_{t}\|_{2}^{2}$ , where  $z_{\mathrm{goal}}$  is the target position of the elevator,  $c_{1},c_{2} > 0$ .
Variations of the LQR/LQG problem have garnered considerable interest in the machine learning community. In the recent literature of online nonstochastic control, several setting-based generalizations to the aforementioned linear-control framework have been explored, including:
- Adversarial cost functions (Cohen et al. [2018], Agarwal et al. [2019a]): In this setting, the cost functions are adversarily chosen in an unpredictable way to the learner. This generalized formulation allows for applications such as building climate control in the presence of time-varying energy costs, as the energy costs may depend on unexpected demand fluctuation (Cohen et al. [2018]). More recently, the model of adversially chosen cost functions, along with adversarial perturbations in the subsequent paragraph, give rise to building connection between control theory and meta optimization (Chen and Hazan [2023]), where the nonconvex problem of hyperparameter tuning (e.g. learning rate tuning) can be reduced to online nonstochastic control, which often admits a convex formulation.
- Adversarial perturbations (Agarwal et al. [2019a], etc.): In this setting, differing from the usual Gaussian assumption on the perturbation distribution  $(\mathbf{w}_t \stackrel{i.i.d.}{\sim} N(0, \Sigma))$ , non-stochastic control often considers arbitrary disturbances  $\mathbf{w}_t$  in the dynamics. The relaxed assumption on the perturbations allows for the control of nonlinear systems under model inaccuracies by approximating them as linear systems (Ghai et al. [2022]).
- Bandit feedback: Together with the aforementioned two generalizations, control under bandit feedback further generalizes the control problem and suits many real-world applications. In the bandit setting, the cost functions is unknown to the learner. The learner has access only to the cost incurred by their chosen control. In addition, the learner has access to the system's state  $(\mathbf{x}_t$  in the fully observed LQR setting) or a noisy, possibly and usually low-dimensional observation  $(\mathbf{y}_t$  in the partially observed LQG setting). The bandit LQG problem can be illustrated with a motivating example. In the problem of controlling medical device such as a ventilator, the state usually represents a number of health status indicators of the patient, such as heart rate, blood pressure, and other related metrics. However, this state may not be fully observable. After the treatment is provided, the learner can observe the patient's response, e.g. changes in some vital indicators, with some observation noise and a cost function, e.g. deviation from the targeted breathing pattern. In this limited information setting, can a learner hope to learn meaning controls under minimal assumptions? Previously, the work of Gradu et al. [2020] and Cassel and Koren [2020] both studied the bandit nonstochastic control problem of fully observed systems and provided provable guarantees.
Taken together, these settings give rise to a general setting in differentiable reinforcement learning that strictly contains a variety of classical problems in optimal and robust control. Naturally, when adversarial costs and perturbations are considered, an optimal solution is not defined a priori. Instead, the primary performance metric is regret: the difference between the total cost of a control algorithm and that of the best controller from a specific policy class in hindsight.
As mentioned above, this general setting of bandit nonstochastic control was considered in the recent work of Gradu et al. [2020], whose proposed Bandit Perturbation Controller (BPC) algorithm has a provable regret guarantee of  $\tilde{O}(T^{\frac{3}{4}})$  when compared to a rich policy class called disturbance action controllers (DAC) for fully observed systems. Similar setting has also been studied by Cassel and Koren [2020], who established an optimal regret up to logarithmic factor of  $\tilde{O}(\sqrt{T})$  for fully
observable systems under stochastic perturbations and adversarially chosen cost functions. Recently Ghai et al. [2023] improves the dimension dependence of regret bounds using exploration in action-space rather than parameter-space. However, bandit control for partially observable systems (e.g. LQG) is less understood. Thus, these developments in the search for efficient, low-regret bandit online control algorithms leave a central open question (also stated by by Gradu et al. [2020]):
Can we achieve optimal regret  $O(\sqrt{T})$  with bandit LQG and nonstochastic noise?
Our work answers this question up to logarithmic factors. Our novel Ellipsoidal Bandit Perturbation Controller (EBPC) achieves a  $\tilde{O} (\sqrt{T})$  regret guarantee in the presence of semi-adversarial perturbations in bandit LQG problems with strongly convex cost functions, with the additional generality of possibly unknown system dynamics. By Shamir [2013], this is asymptotically optimal up to logarithmic factors, as bandit optimization over quadratics reduces to bandit control with quadratic losses under  $A = 0$ ,  $B = I$ . Our work therefore resolves the upper-bound/ lower-bound gap for this generalization of LQR/LQG. The following table gives a comprehensive comparison between the regret guarantee of EBPC and existing results in literature.
Table 1: Comparison of previous results to our contributions.
Algorithm	Noise	Observation	Feedback	System	Regret
Agarwal et al. [2019a]	Adversarial	full	full	known	O(√T)
Agarwal et al. [2019b]	Stochastic	full	full	known	O(1)
Foster and Simchowitz [2020]	Adversarial	full	full	known	O(1)
Simchowitz et al. [2020]	Semi-Adv.	partial	full	known	O(1)
Simchowitz et al. [2020]	Semi-Adv.	partial	full	unknown	O(√T)
Gradu et al. [2020]	Adversarial	full	bandit	unknown	O(Tr/4)
Cassel and Koren [2020]	Stochastic	full	bandit	known	O(√T)
Cassel and Koren [2020]	Adversarial	full	bandit	known	O(T2/3)
Theorem 4.1	Semi-Adv.	partial	bandit	known	O(√T)
Theorem 4.2	Semi-Adv.	partial	bandit	unknown	O(√T)
Online Nonstochastic Control and Online LQR. In the last decade, much research has been devoted to the intersection of learning and control. Abbasi-Yadkori and Szepesvári [2011] and Ibrahimi et al. [2012] considered the problem of learning a controller in LQR for known quadratic cost functions and stochastic/martingale difference perturbation sequence when the dynamics of the system is unknown, and achieved  $\tilde{O} (\sqrt{T})$ -regret in this case. Dean et al. [2018] provided the first provable low-regret, efficient algorithm to solve LQR problems with known cost functions and stochastic perturbations. Cohen et al. [2018] extended this result to changing quadratic costs with stochastic perturbations and provided a regret guarantee of  $O(\sqrt{T})$ . Lale et al. [2021] consider the LQG problem with stochastic noise and unknown systems.
More recently, interest has turned to nonstochastic control, in which the cost functions and the perturbations can be adversarially chosen Agarwal et al. [2019a]. A broad spectrum of control problems were reconsidered from the nonstochastic perspective, and several different generalizations were derived. See Hazan and Singh [2022] for a comprehensive text detailing these results.
Online Bandit Convex Optimization with Memory. A classical approach to nonstochastic control of stable/stabilizable linear dynamical systems is to reduce control problems to online convex optimization with bounded memory. This reduction stems from the observation that the effects of past controls decay exponentially in stable/stabilizable systems. In the bandit online convex optimization with bounded memory setting, the learner iteratively plays a decision  $x_{t}$  in a convex set  $\mathcal{K} \subseteq \mathbb{R}^{d}$  and suffers an adversarially chosen loss  $F_{t}(x_{t-H+1:t})$ , where  $x_{t-H+1:t}$  is the sequence of points  $x_{t-H+1}, \ldots, x_{t}$ . In particular, the loss depends on the last  $H$  points played by the algorithm, and the only information revealed to the learner is the scalar loss that they incurred. The goal is to minimize
regret, the difference between the loss actually suffered and the loss suffered under the best single play in hindsight:
$$
\operatorname {R e g r e t} _ {T} \stackrel {\text {d e f}} {=} \sum_ {t = H} ^ {T} F _ {t} (x _ {t - H + 1: t}) - \min  _ {x \in \mathcal {K}} \sum_ {t = H} ^ {T} F _ {t} (x, \dots , x).
$$
Since the loss function is unknown to the learner in the bandit setting, online bandit convex optimization algorithms make use of low-bias estimators of the true gradient or Hessian. Therefore, it is standard to measure the algorithm's performance by regret over the expectation of losses, where the expectation is taken over the randomness injected when creating such estimators.
Online convex optimization with memory in the full information setting, where the loss function is known to the learner, was proposed by Anava et al. [2015]. The work of Agarwal et al. [2019a], was the first to connect this to control, and to give a regret bound for online control with adversarial perturbations.
The  $\tilde{O}(T^{\frac{3}{4}})$  regret bound of Gradu et al. [2020] leveraged the bandit convex optimization method of Flaxman et al. [2005]. We focus on the bandit LQR/LQG setting with strongly convex and smooth loss functions. It is thus natural to use the techniques of Hazan and Levy [2014], who obtained a  $\tilde{O}(\sqrt{T})$  regret guarantee for bandit convex optimization without memory.
Online Learning with Delay. One technical difficulty in extending OCO with memory to the bandit setting arises from the requirement of independence between every play and the noises injected from the recent  $H$  steps. We resolve this issue by adapting online learning with delay to the subroutine algorithm used in our BCO algorithm. Online learning with delay was introduced by Quandrud and Khashabi [2015]. In particular, Flaspohler et al. [2021] relates online learning with delay to online learning with optimism and established a sublinear regret guarantee for mirror descent algorithms. A similar delay scheme was seen in [Gradu et al., 2020].
1.2 Notations and organization
Notation. For convenience and succinctness, we denote  $\bar{H} \stackrel{\mathrm{def}}{=} H - 1$ . We use lowercase bold letters (e.g. x, y, u, w, e) to denote the states, observations, controls, and noises of the dynamical system  $^2$ , and  $d_{\mathbf{x}}, d_{\mathbf{u}}, d_{\mathbf{y}}$  to denote their corresponding dimensions. We use  $S^{n-1}$  to denote the unit sphere in  $\mathbb{R}^n$ , as  $S^{n-1} \cong \mathbb{R}^{n-1}$ .  $\mathbb{R}_+$ ,  $\mathbb{R}_{++}$  denote all non-negative, positive real numbers, respectively. For a differentiable function  $F: (\mathbb{R}^n)^H \to \mathbb{R}$ , we denote the gradient of  $F$  with respect to its  $i$ th argument vector by  $\nabla_i F(\cdot)$ .  $\rho(\cdot)$  acting on a square matrix measures the spectral radius of the matrix. For a sequence  $M = (M^{[i]})_{i \in I}$ , we use  $\|M\|_{\ell_1,\mathrm{op}}$  to denote the sum of the operator norm:  $\|M\|_{\ell_1,\mathrm{op}} \stackrel{\mathrm{def}}{=} \sum_{i \in I} \|M^{[i]}\|_{\mathrm{op}}$ .  $\sigma(A)$  denotes the  $\sigma$ -algebra generated by a random variable  $A$ . We use  $O(\cdot)$  to hide all universal constants,  $\tilde{O}(\cdot)$  to hide poly(log T) terms, and  $\mathcal{O}(\cdot)$  to hide all natural parameters.
Organization. Our method has two main components: a novel algorithm for BCO with memory (EBCO-M), and its application to building a novel bandit perturbation controller (EBPC). Section 2 describes our problem setting. Section 3 gives EBCO-M and its near-optimal regret guarantee. Section 4 introduces EBPC and its regret guarantees to both known and unknown systems.
2 The Bandit LQG Problem
In this section we provide necessary background and describe and formalize the main problem of interest. We consider control of linear time-invariant dynamical systems of the form
$$
\mathbf {x} _ {t + 1} = A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}, \mathbf {y} _ {t} = C \mathbf {x} _ {t} + \mathbf {e} _ {t}. \tag {2.1}
$$
with dynamics matrices  $A \in \mathbb{R}^{d_{\mathbf{x}} \times d_{\mathbf{x}}}$ ,  $B \in \mathbb{R}^{d_{\mathbf{x}} \times d_{\mathbf{u}}}$ ,  $C \in \mathbb{R}^{d_{\mathbf{y}} \times d_{\mathbf{x}}}$ . Here, consistent with previous notations,  $\mathbf{x}_t \in \mathbb{R}^{d_{\mathbf{x}}}$  is the state of the system at time  $t$ ,  $\mathbf{u}_t \in \mathbb{R}^{d_{\mathbf{u}}}$  is the control applied at time  $t$ , and
It shall be noted that unbolded  $x, y, u$  are used in the EBCO-M algorithm (Algorithm 1) and is unrelated to the linear dynamical system.
$\mathbf{w}_t \in \mathbb{R}^{d_{\mathbf{x}}}, \mathbf{e}_t \in \mathbb{R}^{d_{\mathbf{y}}}$  are the system and measurement perturbations. At each timestep, the learner may observe  $\mathbf{y}_t \in \mathbb{R}^{d_{\mathbf{y}}}$ , which usually represents a possibly noisy projection of the state  $\mathbf{x}_t$  onto some (possibly and usually) low-dimensional space.
In the online bandit setting, the learner is asked to perform a control  $\mathbf{u}_t$  at time  $t$ . After the control is performed, the adversary chooses a quadratic cost function  $c_{t}(\mathbf{y}_{t},\mathbf{u}_{t})$ . The learner observes the scalar  $c_{t}(\mathbf{y}_{t},\mathbf{u}_{t})\in \mathbb{R}_{+}$  and the signal  $\mathbf{y}_t$ , but no additional information about  $c_{t}(\cdot ,\cdot)$ ,  $\mathbf{x}_t$ ,  $\mathbf{w}_t$ , or  $\mathbf{e}_t$  is given. The goal of the learner is to minimize expected regret, where regret against the controller class  $\Pi$  is defined as
$$
\operatorname {R e g r e t} _ {T} \stackrel {\text {d e f}} {=} \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {y} _ {t}, \mathbf {u} _ {t}\right) - \min  _ {\pi \in \Pi} \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {y} _ {t} ^ {\pi}, \mathbf {u} _ {t} ^ {\pi}\right), \tag {2.2}
$$
where  $\mathbf{u}_t^\pi$  is the control exerted at time  $t$  by policy  $\pi$ , and  $\mathbf{y}_t^\pi$  is the time- $t$  observation that would have occurred against the same costs/noises if the control policy  $\pi$  were carried out from the beginning.
In controlling linear dynamical systems with partial observations, we often make use of the system's counterfactual signal had no controls been performed since the beginning of the instance:
Definition 2.1 (Nature's y). Nature's y at time  $t$ , denoted by  $\mathbf{y}_t^{\mathrm{nat}}$ , is the signal that the system would have generated at time  $t$  under  $\mathbf{u}_{1:t} = 0$ . We may compute this as
$$
\mathbf {x} _ {t + 1} ^ {\mathbf {n a t}} = A \mathbf {x} _ {t} ^ {\mathbf {n a t}} + \mathbf {w} _ {t}, \quad \mathbf {y} _ {t} ^ {\mathbf {n a t}} = C \mathbf {x} _ {t} ^ {\mathbf {n a t}},
$$
or, equivalently,  $\mathbf{y}_t^{\mathbf{nat}} = \mathbf{e}_t + \sum_{i=1}^{t-1} CA^{t-i-1}\mathbf{w}_i$ .
Critically, this may be calculated via the Markov operator:
Definition 2.2 (Markov operator). The Markov operator  $G = [G^{[i]}]_{i \in \mathbb{N}}$  corresponding to a linear system parametrized by  $(A, B, C)$  as in Eq.(2.1) is a sequence of matrices in  $\mathbb{R}^{d_{\mathbf{y}} \times d_{\mathbf{u}}}$  such that  $G^{[i]} \stackrel{\text{def}}{=} CA^{i-1}B$ ,  $G^{[0]} \stackrel{\text{def}}{=} \mathbf{0}_{d_{\mathbf{y}} \times d_{\mathbf{u}}}$ .
It follows immediately that  $\mathbf{y}_t^{\mathrm{nat}}$  may be computed from observations as  $\mathbf{y}_t^{\mathrm{nat}} = \mathbf{y}_t - \sum_{i=1}^{t} G^{[i]}\mathbf{u}_{t-i}$ .
We impose four core assumptions on the problem:
Assumption 2.3 (Stable system). We assume the system is stable: the spectral radius  $\rho(A) < 1^3$ .
This assumption has the following important consequence:
Remark 2.4 (Decay of stable systems). That the system is stable implies that  $\exists P\succ \mathbf{0}_{d_{\mathbf{x}}\times d_{\mathbf{x}}}$ $P\in \operatorname {Sym}(d_{\mathbf{x}})$  such that  $rP\succeq A^{\top}PA$  for some  $0\leq r < 1$ , and therefore  $\exists \kappa$  depending on  $\| B\|_{\mathrm{op}},\| C\|_{\mathrm{op}},\sigma_{\min}(P)$  such that  $\| G^{[i]}\|_{\mathrm{op}}\leq \kappa r^{i - 1}$ . Then with  $H = O(\log T)$ , we can assume that  $\| G\|_{\ell_1,\mathrm{op}} = \sum_{i = 0}^{\infty}\| G^{[i]}\|_{\mathrm{op}}\leq R_G$  and  $\psi_G(H)\stackrel {\text{def}}{=}\sum_{i = H}^{\infty}\| G^{[i]}\|_{\mathrm{op}}\leq \frac{R_G}{T}$ .
Together, Assumption 2.3 and its consequence Remark 2.4 allows modeling the nonstochastic control problem as an online convex optimization problem against adversary with bounded memory, as the effect of past controls decays exponentially. Assumption 2.5-2.7 introduce the assumptions on the adversarial cost functions and perturbations.
Assumption 2.5 (Noise model). The perturbations  $\{\mathbf{w}_t,\mathbf{e}_t\}_{t = 1}^T$  are assumed to be semi-adversarial:  $\mathbf{w}_t,\mathbf{e}_t$  decompose as sums of adversarial and stochastic components  $\mathbf{w}_t = \mathbf{w}_t^{\mathrm{adv}} + \mathbf{w}_t^{\mathrm{stoch}}$  and  $\mathbf{e}_t = \mathbf{e}_t^{\mathrm{adv}} + \mathbf{e}_t^{\mathrm{stoch}}$ . The stochastic components of the perturbations are assumed to come from distributions satisfying  $\mathbb{E}[\mathbf{w}_t^{\mathrm{stoch}}] = \mathbb{E}[\mathbf{e}_t^{\mathrm{stoch}}] = 0$ ,  $\mathbb{E}\left[\mathbf{w}_t^{\mathrm{stoch}}\mathbf{w}_t^{\mathrm{stoch}}\right] \succeq \sigma_{\mathbf{w}}^2 I$ ,  $\mathbb{E}\left[\mathbf{e}_t^{\mathrm{stoch}}\mathbf{e}_t^{\mathrm{stoch}}\right] \succeq \sigma_{\mathbf{e}}^2 I$ ,  $\sigma_{\mathbf{e}} > 0$ .  $\{\mathbf{w}_t,\mathbf{e}_t\}_{t = 1}^T$  are bounded such that  $\| \mathbf{y}_t^{\mathrm{nat}}\| _2 \leq R_{\mathrm{nat}}$ ,  $\forall t$ , for some parameter  $R_{\mathrm{nat}}$ .
<sup>3</sup>Although out of scope of this work, we refer the readers to Appendix G of Simchowitz et al. [2020] for a possible and expected extension of out results to the relaxed assumption of stability: the learner is given access to a stabilizing controller of the system.
The bound on  $\mathbf{y}^{\mathrm{nat}}$  is implied by bounded noise, which is a standard assumption in literature, and the stability of the system. The semi-adversarial assumption is also seen in prior work [Simchowitz et al., 2020], and is a necessary condition for our analysis: we depend on the regret guarantee of a bandit online convex optimization with memory algorithm which requires the strong convexity of the expected loss functions conditioned on all but the  $\Theta(\mathrm{poly}(\log T))$  most recent steps of history. This assumption is essentially equivalent to the adversarial assumption in applications: in almost all systems, noise is either endemic or may be injected. We also emphasize that this assumption is much weaker than that of previous optimal-rate work: even in the known-state, known-dynamic case, the previous optimal guarantee in the bandit setting depended on no adversarial perturbation (see Table 1).
Assumption 2.6 (Cost model). The cost functions  $c_{t}(\cdot, \cdot)$  are assumed to be quadratic,  $\sigma_{c}$ -strongly convex,  $\beta_{c}$ -smooth, i.e.,  $c_{t}(\mathbf{y}, \mathbf{u}) = \mathbf{y}^{\top} Q_{t} \mathbf{y} + \mathbf{u}^{\top} R_{t} \mathbf{u}$  with  $\beta_{c} I \succeq Q_{t} \succeq \sigma_{c} I, \beta_{c} I \succeq R_{t} \succeq \sigma_{c} I \forall t$ . They are also assumed to obey the following Lipschitz condition:  $\forall (\mathbf{y}, \mathbf{u}), (\mathbf{y}', \mathbf{u}') \in \mathbb{R}^{d_{\mathbf{y}} + d_{\mathbf{u}}}$ ,
$$
\left| c _ {t} (\mathbf {y}, \mathbf {u}) - c _ {t} \left(\mathbf {y} ^ {\prime}, \mathbf {u} ^ {\prime}\right) \right| \leq L _ {c} \left(\left\| (\mathbf {y}, \mathbf {u}) \right\| _ {2} \vee \left\| \left(\mathbf {y} ^ {\prime}, \mathbf {u} ^ {\prime}\right) \right\| _ {2}\right) \| (\mathbf {y} - \mathbf {y} ^ {\prime}, \mathbf {u} - \mathbf {u} ^ {\prime}) \| _ {2}. \tag {2.3}
$$