# Mean-Covariance Steering of a Linear Stochastic System with Input Delay and Additive Noise

Gabriel Velho

Université Paris-Saclay,
CentraleSupélec, CNRS
Laboratoire des Signaux
et Systèmes,
Gif-sur-Yvette, France

gabriel.velho@centralesupelec.fr

Riccardo Bonalli

Université Paris-Saclay,
CentraleSupélec, CNRS
Laboratoire des Signaux
et Systèmes,
Gif-sur-Yvette, France

riccardo.bonalli@centralesupelec.fr

Jean Auriol

Université Paris-Saclay,
CentraleSupélec, CNRS
Laboratoire des Signaux
et Systèmes,
Gif-sur-Yvette, France

jean.auriol@centralesupelec.fr

Islam Boussaada

Université Paris-Saclay,
CentraleSupélec, CNRS
Laboratoire des Signaux
et Systèmes,
Institut Polytechnique
des Sciences Avancées,
Gif-sur-Yvette, France

islam.boussaada@centralesupelec.fr

**Abstract**—In this paper, we introduce a novel approach to solve the (mean-covariance) steering problem for a fairly general class of linear continuous-time stochastic systems subject to input delays. Specifically, we aim at steering delayed linear stochastic differential equations to a final desired random variable with given mean and covariance. We first establish a controllability result for these control systems, revealing the existence of a lower bound under which the covariance of the control system cannot be steered. This structural threshold covariance stems from a unique combined effect due to stochastic diffusions and delays. Next, we propose a numerically cheap approach to reach any neighbor of this threshold covariance in finite time. Via an optimal control-based strategy, we enhance the aforementioned approach to keep the system covariance small at will in the whole control horizon. Under some additional assumptions on the dynamics, we give theoretical guarantees on the efficiency of our method. Finally, numerical simulations are provided to ground our theoretical findings, showcasing the ability of our methods in optimally approaching the covariance threshold.

**Index Terms**—Stochastic systems, Delay systems, Linear systems, Covariance steering.

## I. INTRODUCTION

Dynamical processes are often affected by disturbances stemming from various factors, such as imprecise measurements, parameter uncertainties, and external disturbances. Such disturbances may considerably alter the dynamics, as in several applications such as automated vehicle steering, traffic network control, or building heat regulation. Therefore, effectively mitigating these uncertainties is crucial to establish the reliability and security of such controlled systems. Stochastic Differential Equations (SDEs) provide broad and accurate modelization of a large class of uncertain systems. Stochastic control enables the effective design of stabilizing controllers for SDEs, which are also robust against random fluctuations. In particular, such robustness may be reliably achieved by seeking controllers that keep the state variance relatively small, see [1]–[3], and references therein.

Another crucial consideration in the modeling of dynamical systems is the integration of delays into the dynamics. Delays in the system state or the control input stem from various sources, including physical constraints or transmission times [4]. When delays take large values, neglecting them may lead to critical stability issues, hindering robust control of the system dynamics. For deterministic systems, the generation of predictive state models has been suggested to handle delays [5], [6]. However, these methods often depend on prior knowledge of the system dynamics, a challenging limitation when dealing with stochastic systems. Certain methods have been proposed to handle systems with unknown perturbations [7], [8]. However, their implementation still requires some prior knowledge of the noise structure. Consequently, such approaches can hardly be adjusted to stabilize delayed SDEs.

All the aforementioned hindrances show the urgency in developing novel methods to efficiently and robustly control stochastic systems with delays. In this regard, some stabilization methods have been proposed [9], [10]. Yet, state covariance minimization, often key to mitigating uncertainty, is generally disregarded. To effectively compute strategies seeking minimal covariance, optimal control methods have been alternatively investigated. These approaches are supported by necessary conditions for optimality, which are however efficiently implementable only in specific settings [11]–[13]. Notably, in Linear Quadratic (LQ) settings, i.e., linear dynamics and quadratic costs, conditions for optimality may be efficiently solved by seeking solutions to Riccati-type ordinary differential equations [14]–[16], a numerically cheap method. Nevertheless, to the best of our knowledge, as efficient as they may be, these approaches do not support final state constraints. Importantly, estimates bounding the state covariance are generally underrated and not investigated, although these are crucial to establish the system's overall safety [17].

In this paper, by merging methodologies from deterministic delayed control and stochastic control, we start bridging these gaps. Specifically, for the first time, we
introduce a novel approach to control linear SDEs with delays under guarantees ensuring the state covariance is kept small throughout the control horizon. Our controls are uniquely cheap to numerically implement. Our contribution is threefold:

1) First, we investigate the controllability, in mean and covariance, of delayed SDEs. For this, we extend Arstein-type transformations [5], [18]. Unlike non-delayed settings, our analysis shows the covariance of linear delayed SDEs can not be steered to any symmetric definite positive matrix. Instead, due to the combined presence of diffusions and delays, the system can never be stalled beyond some minimal covariance<fnref n="1" />. This is a structural obstruction exclusive to delayed SDEs. Still, under classical rank-type controllability conditions, we prove an open-loop control exists enabling to steer delayed SDEs close to this minimal covariance at will.

2) The implementation of open-loop controllers is notably impractical in the stochastic framework. To achieve the design of numerically tractable control laws, we leverage covariance steering techniques [1], [3]. This will enable computing feedback controls that steer linear delayed SDEs from an initial covariance to any final covariance as close as wanted to the aforementioned minimal covariance while minimizing the control effort.

3) The previous class of controllers does not guarantee the covariance remains small throughout the whole control horizon. To bridge this gap, we leverage optimal control techniques to minimize both the final state covariance and the covariance along the whole trajectory. Importantly, under some additional assumptions on the dynamics, we provide estimates of these minimal covariances in the autonomous case. These bounds show the covariance of delayed SDEs can be forced to evolve within any neighbor of the minimal covariance.

The paper is organized as follows. In Section II, we outline the problem formulation. Our first controllability result is stated and proved in Section III. In Section IV, we introduce a numerically efficient methodology to steer the system covariance. Upon these results, in Section V we develop a technique to minimize the covariance throughout the control horizon, developing error estimates. Finally, in Section VI, we present numerical simulations on a real-world system model for building temperature control, subject to realistic temperature transmission delays and random external temperature fluctuations.

## II. PROBLEM FORMULATION

### A. Notations

Let us consider $n$ and $m$ two positive integers. We assume state variables take values in $\mathbb{R}^n$, while control variables take values in $\mathbb{R}^m$. We assume we are given a filtered probability space $(\Omega, \mathcal{F} \triangleq (\mathcal{F}_t)_{t \in [0, \infty)}, \mathbb{P})$. For the sake of clarity in the exposition and without loss of generality, from now on, we assume stochastic perturbations are due to a one-dimensional Wiener process $W_t$, which is adapted to the filtration $\mathcal{F}$. Let $T > 0$ be some given time horizon, while $0 < h < T$ is some fixed delay. For any $r \in \mathbb{N}$, we denote by $L^2_{\mathcal{F}}([0, T], \mathbb{R}^r)$ the set of square integrable processes $P: [0, T] \times \Omega \to \mathbb{R}^r$ that are adapted to $\mathcal{F}$. The spaces of semi-definite and definite positive symmetric matrices in $\mathbb{R}^n$ are denoted by $S_n^+$ and $S_n^{++}$, respectively. If $M(t)$ is an $L^\infty([0, T], \mathbb{R}^{n \times n})$ matrix function, we denote $\Phi_M(t, s)$ the fundamental matrix associated to it, which is by definition the unique solution to the system

$$ \left\{ \begin{array}{l} \frac{d\Phi_M}{dt}(t, s) = M(t)\Phi_M(t, s), \\ \Phi_M(s, s) = I. \end{array} \right. $$

If $X \in L^2_{\mathcal{F}}([0, T], \mathbb{R}^r)$, we denote by $\Sigma_X(\cdot)$ its covariance (matrix), which is defined as

$$ \Sigma_X(t) \triangleq \mathbb{E}[(X(t) - \mathbb{E}[X(t)])(X(t) - \mathbb{E}[X(t)])^T] \in S_n^+. $$

### B. Control system and assumptions

In this paper, we consider delayed-input SDEs of the form

$$ \left\{ \begin{array}{l} dX(t) = (A(t)X(t) + B(t)U(t-h) + r(t))dt \\ \qquad + \sigma(t)dW_t, \\ X(0) = X_0, \\ U(s) = 0 \quad \text{for } s \in [-h, 0], \end{array} \right. \qquad (1) $$

where $X_0 \in \mathbb{R}^n$ is a fixed initial condition, whereas the control $U$ lies in the control space $\mathcal{U} \triangleq L^2_{\mathcal{F}}([0, T], \mathbb{R}^m)$. The constant $h > 0$ is a positive delay acting on the control input. Given $X_T \in \mathbb{R}^n$ and $\Sigma_T \in S_n^{++}$, our goal is to find $U \in \mathcal{U}$ that steers the corresponding solution $X$ of (1) to

$$ \left( \begin{array}{c} \mathbb{E}[X(T)] \\ \Sigma_X(T) \end{array} \right) = \left( \begin{array}{c} X_T \\ \Sigma_T \end{array} \right). $$

We call this problem the (mean-covariance) steering problem. It boils down to finding the control that displaces the initial probability distribution of the state to a more desirable final distribution. To fulfil this control objective, we make the following assumption.

**Assumption 1.** Throughout the paper, we assume the following properties hold true:

* $A \in L^\infty([0, T], \mathbb{R}^{n \times n})$, $B \in L^\infty([0, T], \mathbb{R}^{n \times m})$, $r \in L^\infty([0, T], \mathbb{R}^n)$ and $\sigma \in L^\infty([0, T], \mathbb{R}^n)$. In particular, these mappings are deterministic.

<footnote n="1">¹with respect to the partial order in the space of symmetric definite matrices.</footnote>
• The Grammian associated with the non-delayed deterministic system, defined as

$$G_{\tau}^{T+h} \triangleq \int_{\tau}^{T+h} \Phi_{A}(T+h, s)B(s)B(s)^{T}\Phi_{A}(s, T+h)^{T}dt, \quad (2)$$

is invertible for all $0 < \tau < T + h$.

The first assumption on the dynamics is classical as it guarantees the well-posedness of the system [19]. The second assumption allows for the total controllability of the deterministic non-delayed system, meaning it can be controlled in any sub-interval of $[0, T + h]$ [20]. Furthermore, we assume that the initial state $X_0$ is deterministic. While our approach can be extended straightforwardly to Gaussian random variables (as in [1]), we maintain a null initial covariance for the sake of simplicity. The above assumptions are generic and usually satisfied by a wide class of systems.

### C. System reduction

We can reduce the steering problem to an easier one, where $X_0$ and $X_T$ are both 0 and where the drift $r(t)$ is the zero function. For this, let $X_r(t) \triangleq X_T \frac{t}{T} + X_0 \frac{T-t}{T}$ and $\bar{X}(t) \triangleq X(t) - X_r(t)$. The difference $\bar{X}(t)$ follows the dynamic:

$$\left\{ \begin{aligned} d\bar{X}(t) &= (A(t)\bar{X}(t) + B(t)U(t-h) + \bar{r}(t)) dt \\ &\quad + \sigma(t)dW_t, \\ \bar{X}(0) &= 0, \\ U(s) &= 0 \quad \text{for } s \in [-h, 0]. \end{aligned} \right. \quad (3)$$

where $\bar{r}(t) \triangleq r(t) + \dot{X}_r(t) - A(t)X_r(t)$. Steering the mean and the covariance of $X$ from $\begin{pmatrix} X_0 \\ 0 \end{pmatrix}$ to $\begin{pmatrix} X_T \\ \Sigma_T \end{pmatrix}$ is thus equivalent to steering the mean and covariance of $\bar{X}(t)$ from $\begin{pmatrix} 0 \\ 0 \end{pmatrix}$ to $\begin{pmatrix} 0 \\ \Sigma_T \end{pmatrix}$. Indeed, adding a deterministic term to the controller compensates for the effect induced by the drift and the change of variables [3]. More precisely, we consider

$$U(t) = U_{\text{feedback}}(t) + U_{\text{drift}}(t),$$

with

$$U_{\text{drift}}(t) = -B(t+h)^T \Phi_A(t+h)^T G^{-1} \\ \times \left( \int_0^T \Phi_A(T,s) \bar{r}(s) ds \right).$$

By leveraging similar computations to the ones detailed in [20], it can be verified that controlling system (3) as required above boils down to steering the mean and covariance of the reduced system

$$\left\{ \begin{aligned} dX(t) &= (A(t)X(t) + B(t)U_{\text{feedback}}(t-h)) dt \\ &\quad + \sigma(t)dW_t, \\ X(0) &= 0, \\ U(s) &= 0 \quad \text{for } s \in [-h, 0], \end{aligned} \right. \quad (4)$$

from $\begin{pmatrix} 0 \\ 0 \end{pmatrix}$ to $\begin{pmatrix} 0 \\ \Sigma_T \end{pmatrix}$ with a control $U = U_{\text{feedback}} \in \mathcal{U}$ of zero mean. Therefore, from now on, we only focus on this latter problem. Of particular interest is the case where $\Sigma_T$ is the smallest possible.

## III. COVARIANCE STEERING: OPEN-LOOP CONTROLS

In this section, we study which state covariances can be reached by the solution of (4) within a finite time frame. In particular, we show that there exists a lower bound on the state covariance under which (4) can not be steered. We then establish the existence of open-loop controls that achieve state covariances that are arbitrarily close to this lower bound.

### A. Extending the Artstein transformation

We achieve the aforementioned goal by leveraging the Artstein transformation [5] to our stochastic setting.

**Definition 2 (Artstein transform).** Let $X$ be the process solving equation (3). The Artstein transform of $X$ is the following process, which is adapted to $\mathcal{F}$,

$$Y(t) \triangleq X(t) + \int_{t-h}^{t} \Phi_{A}(t, s+h)B(s+h)U(s)ds. \quad (5)$$

It can be easily checked that the Artstein transform solves the following non-delayed stochastic equation

$$\left\{ \begin{aligned} dY(t) &= (A(t)Y(t) + \bar{B}U(t)) dt + \sigma(t)dW_t, \\ Y(0) &= 0, \end{aligned} \right. \quad (6)$$

where $\bar{B}(t) \triangleq \Phi_A(t, t+h)B(t+h)$. We can now apply known results from non-delayed stochastic control from [3], [19] to the Artstein transform $Y$ to steer (6) as desired. However, it is unclear how to compute the covariance of $X$, which we aim to estimate once the covariance of $Y$ is available. For this, we make use of the following:

**Lemma 3.** Let $X$ be the process solving equation (3), and $Y$ its Artstein transform. Then for all $t \in [h, T]$

$$X(t) = \Phi_A(t, t-h)Y(t-h) + \int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s. \quad (7)$$

*Proof.* Using the analytic formula for linear SDEs [19], we can establish the following relation between $X(t)$ and $X(t-h)$:

$$X(t) = \Phi_A(t, t-h)X(t-h) \\ + \int_{t-h}^{t} \Phi_A(t, s)B(s)U(s-h)ds \\ + \int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s.$$
By expressing $X(t-h)$ in terms of $Y(t-h)$, a change of variable in the regular integral yields

$$X(t) = \Phi_A(t, t-h)Y(t-h) + \int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s \\ - \Phi_A(t, t-h)\int_{t-2h}^{t-h} \Phi_A(t-h, s+h)B(s+h)U(s)ds \\ + \int_{t-2h}^{t-h} \Phi_A(t, s+h)B(s+h)U(s)ds,$$

from which we infer the conclusion thanks to the identity $\Phi_A(t, t-h)\Phi_A(t-h, s+h) = \Phi_A(t, s+h)$. $\square$

Equation (7) provides important insights for the controllability of the process $X$. In particular, it states $X(t)$ can only be controlled through $Y(t-h)$, in that the additional noise term $\int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s$ can not be controlled. Let $\Sigma_Y(t) \triangleq \mathbb{E}[(Y(t) - \mathbb{E}[Y(t)])(Y(t) - \mathbb{E}[Y(t)])^T]$ denote the covariance of the process $Y$. Equation (7) can be further manipulated to express $\Sigma_X$ as a function of $\Sigma_Y$ as explained in the following lemma.

**Lemma 4.** Let $X$ be the process solving equation (3), and let $Y$ be its Artstein transform. For all $t \in [h, T]$,

$$\Sigma_X(t) = \Phi_A(t, t-h)\Sigma_Y(t-h)\Phi_A(t, t-h)^T + \Sigma_{\min}(t), \quad (8)$$

where

$$\Sigma_{\min}(t) \triangleq \int_{t-h}^{t} \Phi_A(t, s)\sigma(s)\sigma(s)^T \Phi_A(t, s)^T ds.$$

*Proof.* Fix $t \in [h, T]$. We first utilize a standard result from stochastic calculus that states that, if $\gamma_1$ and $\gamma_2$ two deterministic functions in $L^2([0, T], \mathbb{R})$, then:

$$\mathbb{E} \left[ \left( \int_{0}^{t} \gamma_{1}(s) dW_{s} \right) \left( \int_{0}^{t} \gamma_{2}(s) dW_{s} \right) \right] = \int_{0}^{t} \gamma_{1}(s) \gamma_{2}(s) ds \quad (9)$$

From this result, we have that

$$\mathbb{E} \left[ \left( \int_{0}^{t} \Phi_{A}(t, s) \sigma(s) dW_{s} \right) \left( \int_{0}^{t} \Phi_{A}(t, s) \sigma(s) dW_{s} \right)^{T} \right] \\ = \int_{0}^{t} \Phi_{A}(t, s) \sigma(s) \sigma(s)^{T} \Phi_{A}(t, s)^{T} ds = \Sigma_{\min}(t)$$

Since the covariance of the sum of two independent random variables is the sum of their covariances, what is left to prove is that $\Phi_A(t, t-h)Y(t-h)$ is independent of $\int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s$. Since $X$ is adapted to $\mathcal{F}$, by definition $Y$ is also adapted to $\mathcal{F}$, and in particular $Y(t-h)$ is $\sigma(W_s: 0 \le s \le t-h)$-measurable. To conclude, it is sufficient to prove that $\int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s$ is independent from $W_r$, for $r$ in $[0, t-h]$. Now, if $t-h = s_0 < \dots < s_N = t$ is a sequence of partitions of $[t-h, t]$ whose size tends

to zero, thanks to the properties of the $\hat{\text{Ito}}$ integral and Assumption 1 we have that

$$\int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s = \\ \lim_{N \to \infty} \sum_{i=0}^{N-1} \Phi_A(t, s_i)\sigma(s_i)(W_{s_{i+1}} - W_{s_i}), \text{ in } L^2.$$

For every $r \in [0, t-h]$, it thus follows that

$$\mathbb{E} \left[ W_r \int_{t-h}^{t} \Phi_A(t, s) \sigma(s) dW_s \right] = \\ = \lim_{N \to \infty} \sum_{i=0}^{N-1} \Phi_A(t, s_i) \sigma(s_i) \mathbb{E} \left[ W_r (W_{s_{i+1}} - W_{s_i}) \right] = 0,$$

due to the independence properties of Wiener processes. This concludes the proof. $\square$

The covariance of the state $X$ comprises two components: one determined by the covariance of $Y$, which is under our complete control, and another determined by the covariance of $\int_{t-h}^{t} \Phi_A(t, s)\sigma(s)dW_s$ which remains unaffected by the control term. Consequently, the state covariance is inherently greater than the latter covariance, presenting a limitation as there is no means to reduce it.

## B. Controllability of the Artstein transform

Our goal is now to minimize the covariance as much as possible and propose control strategies that can approach the lower covariance bound as closely as wanted. The following result is classical [21, Theorem 13]:

**Theorem 5** (Controllability of non-delayed SDEs). If the Grammian $\overline{G}_{\tau}^T$ associated to the deterministic part of equation (6), defined as

$$\overline{G}_{\tau}^{T} \triangleq \int_{\tau}^{T} \Phi_{A}(T, s) \overline{B}(s) \overline{B}(s)^{T} \Phi_{A}(T, s)^{T} ds \quad (10)$$

is invertible for all $\tau$ in $[0, T]$, then for every $\Sigma_T \in S_n^{++}$ there exists $U \in \mathcal{U}$ such that the solution $Y$ to (6) associated with $U$ is such that $\mathbb{E}[Y(T-h)] = 0$ and $\Sigma_Y(T-h) = \Sigma_T$.

We may combine Lemma 4 with Theorem 5 to infer open-loop controllability of the original control system (4).

**Theorem 6** (Controllability of delayed SDEs). Let $\Sigma_T \in S_n^{++}$. Under Assumption 1, there exists $U \in \mathcal{U}$ such that the solution $X$ to (3) associated with $U$ verifies $\mathbb{E}[X(T)] = 0$ and $\Sigma_X(T) = \Sigma_T + \Sigma_{\min}(T)$.

*Proof.* To apply Theorem 5, we need to prove that Assumption 1 guarantees the invertibility of the modified
Grammian $\overline{G}_{\tau}^{T}$. For this, we may compute

$$
\begin{align*}
\overline{G}_{\tau}^{T} &= \int_{\tau}^{T} \Phi_{A}(T, s) \overline{B}(s) \overline{B}(s)^{T} \Phi_{A}(T, s)^{T} ds \\
&= \int_{\tau}^{T} \Phi_{A}(T, s) \Phi_{A}(s, s+h) B(s+h) B(s+h)^{T} (\Phi_{A}(T, s) \Phi_{A}(s, s+h))^{T} ds \\
&= \int_{\tau}^{T} \Phi_{A}(T, s+h) B(s+h) B(s+h)^{T} \Phi_{A}(T, s+h)^{T} ds \\
&= \Phi_{A}(T, T+h) G_{\tau+h}^{T+h} \Phi_{A}(T, T+h)^{T},
\end{align*}
$$

which is invertible thanks to Assumption 1. Therefore, there exists $U \in \mathcal{U}$ such that the solution $Y$ to (6) associated with $U$ is such that $\mathbb{E}[Y(T)] = 0$ and $\Sigma_Y(T-h) = \Phi_A(t, t-h)^{-1}\Sigma_T \Phi_A(t, t-h)^{-T}$, and we conclude. $\square$

Thanks to Theorem 6, we can steer the state $X$ to a final covariance that can be arbitrarily close to the limit $\Sigma_{\min}(T)$, by selecting $\Sigma_T$ arbitrarily small. The control derived in this section to guide the system is an open-loop stochastic control, posing challenges for numerical implementation [22]. Consequently, we need to explore the possibility of addressing the steering problem using a feedback controller.

## IV. COVARIANCE STEERING: CLOSE-LOOP CONTROLS

In the previous section, we solved the steering problem by means of open-loop controls. Such control laws are known to be very expensive to compute. To fix this hindrance, we show in this section, the existence of tractable feedback controls that solve the steering problem. We achieve this result by leveraging and combining recent advances in covariance steering techniques, e.g., [1]–[3], with the previous Artstein transformation. In particular, the covariance of solutions to equation (4) can be steered to $\Sigma_T + \Sigma_{\min}$, with $\Sigma_T \in S_n^{++}$, via feedback controls.

In [3], the authors prove that under the controllability conditions of Theorem 5, there exists a unique feedback control $U \in \mathcal{U}$ that steers the covariance of $Y$ solution to (6) to a desired covariance, while minimizing the functional cost

$$J(U) \triangleq \mathbb{E} \left[ \int_{0}^{T} U(t)^{T} RU(t) dt \right],$$

where $R \in S_n^{++}$. We can show the existence of tractable feedback controls that solve the steering problem by combining [3] with our previous results as follows:

**Theorem 7.** Let $\Sigma_T \in S_n^{++}$ and $Y$ denote solutions to (6). Under Assumption 1, there exists a unique feedback control $U^*$ of the form

$$U^*(t) = -R(t)^{-1} \overline{B}(t)^T \Pi(t) Y(t), \quad (11)$$

that minimizes the functional cost $J$ under the constraint $\Sigma_Y(T-h) = \Sigma_T$. Moreover the gain $\Pi(t)$ is the solution to the following Riccati-type ordinary differential equation

$$\left\{ \begin{array}{l} \dot{\Pi} = -A^T \Pi - \Pi A + \Pi B R^{-1} B^T \Pi \\ \Pi(0) = \Pi_0, \end{array} \right. \qquad (12)$$

where $\Pi_0$ is the unique solution of the system

$$\Sigma_T = f(\Pi_0), \qquad (13)$$

where $f$ is a homeomorphism$^2$.

*Proof.* The proof of this theorem is a direct consequence of [3, Lemma 6], whose assumptions are satisfied thanks to Theorem 6. $\square$

Similarly to what has been done in the previous section, once we can control the state covariance of the process $Y$, we can leverage the relation given by lemma 4 and steer the state covariance of $X$ to any symmetric positive definite matrix above the lower bound $\Sigma_M$.

This method still presents some limitations, mainly:

* Computing the controller requires solving equation (13), which, although it can be done through root-finding algorithms, can be expensive as the function $f$ is not easy to compute.

* There are no guarantees on the value of the covariance along the trajectory.

In the next section, we therefore introduce an optimal control-based approach to minimizing the covariance along the trajectory.

## V. GLOBAL-IN-TIME COVARIANCE STEERING VIA CLOSE-LOOP CONTROLS

In the previous section, we showed the existence of tractable feedback controls that solve the steering problem. Such control laws may not guarantee that the covariance remains small during the whole control horizon $[0, T]$. In this section, our objective is to establish an efficiently implementable control strategy that keeps the state covariance consistently low along its trajectory, and not just at the final time. For this, we propose to compute controllers that enable approaching any neighbor of the covariance threshold via techniques from Linear Quadratic (LQ) control [19]. The goal consists of minimizing the quadratic functional cost $J_R$, defined as

$$J_R(U, X) \triangleq \mathbb{E} \left[ \int_{h}^{T} X(t)^T Q(t) X(t) dt + X(T)^T G X(T) \right] \\ + \mathbb{E} \left[ \int_{0}^{T-h} U(t)^T R(t) U(t) dt \right], \qquad (14)$$

<footnote>$^2$For conciseness, we do not provide the full formula of $f$, see [3] for more details.</footnote>
where $Q \in L^\infty([0, T], S_n^+)$, $G \in S_n^+$ and $R \in L^\infty([0, T], S_n^{++})$. This functional cost penalizes the covariance along the trajectory, the final covariance, as well as the control effort. Our problem thus states:

$$ \min_{U \in \mathcal{U}} J_R(U, X), \quad X \text{ solves SDE (3).} \qquad (15) $$

To effectively solve (15), using equation (7) we replace the process $X$ with the process $Y$ in $J_R$. The benefit of this transformation is that optimal control techniques for non-delayed systems may be leveraged, e.g., [19]. Under some additional assumptions on the dynamics, we claim that, when the penalization on the control effort via $R$ approaches zero, the controls solutions to (15) enable the covariance of the delayed SDE to track any neighbor of the threshold covariance.

### A. Optimal control for delayed SDEs.

We may change variable in (15) as follows. For $t > h$, let us denote $V_{min,Q}(t)$ the following positive real number:

$$ V_{min,Q}(t) \triangleq \int_{t-h}^{t} \sigma(s)^{T} \Phi_{A}(t,s)^{T} Q(t) \Phi_{A}(t,s) \sigma(s) ds. $$

**Lemma 8.** The cost function $J_R$ can be written in terms of the state $Y$ as follows:

$$ J_R(U, X) = \mathbb{E} \left[ \int_0^{T-h} Y(t)^T \overline{Q}(t) Y(t) dt \right] \\ + \mathbb{E} \left[ Y(T-h)^T \overline{G} Y(T-h) \right] \\ + \mathbb{E} \left[ \int_0^{T-h} U(t)^T R(t) U(t) dt \right] \\ + \int_h^T V_{min,Q}(t) dt + V_{min,G}(T), \quad (16) $$

where

$$ \overline{Q}(t) \triangleq \Phi_A(t+h, t)^T Q(t+h) \Phi_A(t+h, t) $$

and

$$ \overline{G} \triangleq \Phi_A(T+h, T)^T G \Phi_A(T+h, T) $$

are symmetric positive matrices.

**Proof.** Since the processes $\Phi_A(t, t-h)Y(t-h)$ and $\int_{t-h}^t \Phi_A(t, s)\sigma(s)dW_s$ are independent, similar computations to the ones in the proof of Lemma 4 yield, for $t > h$

$$ \begin{aligned} \mathbb{E} [X(t)^T Q(t) X(t)] = & \\ \mathbb{E} [Y(t-h)^T \Phi_A(t, t-h)^T Q(t) \Phi_A(t, t-h) Y(t-h)] & \\ + \mathbb{E} \left[ \left( \int_{t-h}^{t} \sigma(s) \Phi_A(t, s) dW_s \right)^T Q(t) \left( \int_{t-h}^{t} \sigma(s) \Phi_A(t, s) dW_s \right) \right]. & \end{aligned} $$

Therefore, by leveraging (9) we may compute

$$ \mathbb{E} [X(t)^T Q(t) X(t)] = \mathbb{E} [Y(t-h)^T \overline{Q}(t-h) Y(t-h)] + V_{min,Q}(t). $$

Replacing the latter formula in (14) finally yields (16). □

At this step, we can apply the results from [19, Chapter 6.2] to obtain optimal controls that minimize (14) in the form of feedback controls:

**Theorem 9.** The control $U^*$ solution to (15) is given by

$$ U^* = -R(t)^{-1} \overline{B}(t)^T P(t) Y(t), $$

with $P(\cdot)$ solution to the deterministic Riccati equation

$$ \left\{ \begin{array}{l} \dot{P} = -A^T P - PA - \overline{Q} + P \overline{B} R^{-1} \overline{B}^T P \\ P(0) = \overline{G}. \end{array} \right. \qquad (17) $$

**Proof.** Since $V_{min,Q}(t)$ is independent from the control variable, Lemma 8 shows that the minimization problem (15) is equivalent to the following:

$$ \min_{u \in \mathcal{U}} \bar{J}_R(U, Y), \quad X \text{ solves SDE (6).} \qquad (18) $$

where $\bar{J}_R$ is defined as

$$ \bar{J}_R(U, Y) \triangleq \mathbb{E} \left[ \begin{aligned} & \int_0^{T-h} Y(t)^T \overline{Q}(t) Y(t) dt \\ & + \mathbb{E} \left[ Y(T-h)^T \overline{G} Y(T-h) \right] \\ & + \mathbb{E} \left[ \int_0^{T-h} U(t)^T R(t) U(t) dt \right]. \end{aligned} \right] \qquad (19) $$

This problem is LQ, and thus we may apply the results in [19, Chapter 6.2] to conclude. □

Thanks to Lemma 8, we may note that (14) is lower-bounded by the following quantity:

$$ V_{min} \triangleq \int_{h}^{T} V_{min,Q}(t) dt + V_{min,G}(T), \qquad (20) $$

We will prove that, under some additional assumptions, when $R$ tends to zero it holds that

$$ \min_{U \in \mathcal{U}} J_R(U, X) \xrightarrow{R \to 0} V_{min}, $$

meeting our desired goal.

### B. Convergence equivalence

From now on, we select the specific control weight $R(t) = \rho I$, where $\rho > 0$ is a user-defined parameter that characterizes the control effort in (14). We accordingly denote the associated cost functional $J_R$ by $J_\rho$, which corresponds to

$$ J_{\rho}(U, X) \triangleq \mathbb{E} \left[ \begin{aligned} & \int_{h}^{T} X(t)^{T} Q(t) X(t) dt + X(T)^{T} G X(T) \\ & + \rho \mathbb{E} \left[ \int_{0}^{T-h} U(t)^{T} U(t) dt \right]. \end{aligned} \right] \qquad (21) $$

We denote by $U_\rho$ the corresponding optimal control, whose existence is granted by Theorem 9. Finally, we denote $X_\rho$ the corresponding state trajectory. Our objective is
to give some conditions on the dynamics under which the weighted variance of $X_\rho$ approaches the threshold quantity (20) as $\rho$ approaches zero. The following lemma plays a crucial role in achieving this goal:

**Lemma 10.** Let $U_n$ be a sequence of controls in $\mathcal{U}$ such that, for any matrix $S$ in $S_n^+$, the associated state $Y_n$ obtained through equation (6) verifies

$$ \forall t \in [0, T-h], \quad \mathbb{E} [Y_n(t)^T S Y_n(t)] \underset{n \to \infty}{\to} 0. \quad (22) $$

Then,

$$ J_{\rho}(U_{\rho}) \underset{\rho \to 0}{\to} \int_{h}^{T} V_{min,Q}(t)dt + V_{min,G}(T). \quad (23) $$

*Proof.* Let us first note that the function $\rho \mapsto J_\rho(U_\rho)$ is an increasing function of $\rho$. Indeed, if $0 < \rho_1 < \rho_2$ are two positive real numbers, by optimality it holds that

$$ J_{\rho_2}(U_{\rho_2}) \geq J_{\rho_1}(U_{\rho_2}) \geq J_{\rho_1}(U_{\rho_1}). $$

Let us select a sequence $(\rho_n)_{n \in \mathbb{N}} \subseteq (0, \infty)$ such that

$$ \left\{ \begin{array}{l} \rho_n \mathbb{E} \left[ \int_0^{T-h} U_n(t)^T U_n(t) dt \right] \underset{n \to \infty}{\to} 0 \\ \rho_n \underset{n \to \infty}{\to} 0. \end{array} \right. $$

Due to (16), we have that

$$ J_{\rho_n}(U_n) = \mathbb{E} \left[ \int_0^{T-h} Y_n(t)^T \overline{Q}(t) Y_n(t) dt \right] \\ + \mathbb{E} \left[ Y_n(T-h)^T \overline{G} Y_n(T-h) \right] \\ + \rho_n \mathbb{E} \left[ \int_0^{T-h} U_n(t)^T U_n(t) dt \right] \\ + \int_h^T V_{min,Q}(t)dt + V_{min,G}(T). $$

By definition of $\rho_n$ and using the assumptions made on $Y_n$, we obtain

$$ J_{\rho_n}(U_n) \underset{n \to \infty}{\to} \int_h^T V_{min,Q}(t)dt + V_{min,G}(T). $$

The optimality of $U_{\rho_n}$ yields

$$ \int_h^T V_{min,Q}(t)dt + V_{min,G}(T) \leq J_{\rho_n}(U_{\rho_n}) \leq J_{\rho_n}(U_n), $$

which implies that

$$ J_{\rho_n}(U_{\rho_n}) \underset{n \to \infty}{\to} \int_h^T V_{min,Q}(t)dt + V_{min,G}(T). $$

To conclude the proof, note that since $\rho \mapsto J_\rho(U_\rho)$ is an increasing function of $\rho$, then the above limit is unique for any sequence $(\rho_n)_{n \in \mathbb{N}} \subseteq (0, \infty)$ that converges to zero. $\square$

Establishing the convergence (23) consists of proving the existence of a sequence of controls that drives the state variance toward the minimum threshold. However, achieving this existence is not straightforward, as larger controllers may force the integral of the weighted variance to divergence. Below, we provide examples of SDEs for which finding such a sequence of controls is feasible.

### C. Sufficient conditions for convergence equivalence

1) Pole placement with a normal matrix: Let us consider the following assumption:

**Assumption 11.** The followings hold true:

1) The matrices $A$ and $B$ are time independent.

2) There exists a sequence of matrices $K_n \in \mathbb{R}^{m \times n}$ such that every $H_n \triangleq A - \overline{B}K_n$ is normal, and

$$ \max(\Re(Sp(H_n))) \leq -n. \quad (24) $$

**Lemma 12.** Let $S \in S_n^+$. Under Assumption 11, the sequence of controls $U_n(t) = K_n Y_n(t)$ satisfies

$$ \mathbb{E} [Y_n(t)^T S Y_n(t)] \leq \frac{\|S\| \|\sigma\|_{\infty}^2}{2n}. \quad (25) $$

*Proof.* The solution $Y_n$ to (6) with feedback control $U_n$ can be explicitly computed as

$$ Y_n(t) = \int_0^t e^{H_n(t-s)} \sigma(s) dW_s. $$

By applying Ito's quadratic variation formula to $Y_n$, we may therefore compute

$$ \begin{aligned} \mathbb{E} [Y_n(t)^T S Y_n(t)] &= \int_0^{t-h} \sigma(s)^T e^{H_n^T(t-h-s)} S e^{H_n(t-h-s)} \sigma(s) ds \\ &= \int_0^{t-h} \|\sqrt{S} e^{H_n(t-h-s)} \sigma(s)\|^2 ds \\ &\leq \|S\| \int_0^{t-h} \|e^{H_n(t-h-s)} \sigma(s)\|^2 ds. \end{aligned} $$

Since $H_n$ is normal and $(t-h-s) > 0$, it holds that

$$ \|e^{H_n(t-h-s)} \sigma(s)\|^2 \leq \max(Sp(e^{H_n(t-h-s)}))^2 \|\sigma(s)\|^2, $$

from which we finally infer that

$$ \begin{aligned} \mathbb{E} [Y_n(t)^T S Y_n(t)] &\leq \\ &\leq \|S\| \|\sigma\|_{\infty}^2 \int_0^{t-h} e^{-2n(t-h-s)} ds \leq \frac{\|S\| \|\sigma\|_{\infty}^2}{2n}. \end{aligned} \quad \square $$

Characterizing systems that satisfy Assumption 11 can be difficult. Yet, (25) is in particular satisfied by the fairly large class of fully actuated systems, see the next section. Note that assuming fully actuation is not particularly restricting, in that, if this assumption is not satisfied, one can prove the existence of random states that can not be reached by any open-loop control, see, e.g., [23].

2) Fully actuated systems:

**Assumption 13.** $B(t)$, $t \in [0, T-h]$, has rank $n$.

**Lemma 14.** Let $S \in S_n^+$. Under Assumption 13, there exists a matrix $K(t) \in \mathbb{R}^{m \times n}$ such that

$$ \bar{B}(t)K(t) = I. $$
In particular, the control sequence defined by

$$U_n(t) = -K(t)(A(t) + nI)Y_n(t)$$

yields the estimate (25).

*Proof.* Thanks to Assumptions 1 and 13, $\bar{B}(t)$ has rank $n$ for all $t \in [0, T-h]$, and therefore it admits a stable right inverse. Each process $Y_n$, stemming from each feedback control $U_n$ as defined above, satisfies

$$dY_n(t) = (A(t)Y_n(t) + \bar{B}(t)U_n(t))dt + \sigma(t)dW_t \\ = -nY_n(t) + \sigma(t)dW_t,$$

and we can therefore follow the same arguments as in the proof of Lemma 12 to obtain the estimate (25). □

Summing up, thanks to Theorem 9, we come up with a closed-loop-control-based method to steer the delayed stochastic system, while keeping the state variance close to the threshold variance at will. We demonstrate the efficiency of this method via numerical simulations next.

## VI. NUMERICAL RESULTS

To validate our theoretical findings, we implemented our optimal control-based method to regulate the temperature of a building in realistic settings, where the heat source deliver thermal energy up to some delay. We leveraged and enhanced the dynamical models outlined in [24]. In particular, we made these models more realistic by adding noise. This originates from various sources: 1) the stochastic nature of external temperature fluctuations (modeled using an Ornstein-Uhlenbeck process, see, e.g., [25]), and 2) the unpredictable usage of the building (modeled via the coefficient $\sigma_i$ below). Then, the dynamics are given by

$$dT(t) = \begin{bmatrix} \begin{pmatrix} -R_e & R_e \\ 0 & -\theta \end{pmatrix} T(t) + \begin{pmatrix} R_u \\ 0 \end{pmatrix} U(t-h) \\ -R_e T_{eq} \\ \theta T_p(t) + T_p(t) \end{bmatrix} dt + \begin{pmatrix} \sigma_i \\ \sigma_e \end{pmatrix} dW_t,$$

where $R_e = 5 \cdot 10^{-4}$, $R_u = 2 \cdot 10^{-4}$, $\theta = 3.5 \cdot 10^{-4}$, $T_{eq} = 20$, $\sigma_e = 0.1$, $\sigma_i = 0.05$, and with baseline (i.e., “predicted”) external temperature $T_p(t) = 5 + 5 \cos(0.004t)$. These dynamic are two dimensional. The first variable models the evolution of the temperature of the building (to control), while the second one models the evolution of the external temperature (that we can not control), which has mean $T_p(t)$ and quadratic variation $\int_0^t e^{-2\theta(t-s)} \sigma(s)^2 ds$. The simulation horizon is five days. This enables to stress test our control strategy in mitigating the fluctuations of the external temperature, that are induced by the day-night cycle. The threshold variance of the temperature of the building amounts to $V_{min} = 1.76$.

Figure 1 shows the trajectory of the system using our optimal control-based feedback. The temperature is efficiently stabilized even under poor knowledge of the external temperature. Figure 2 shows the evolution of the variance of the temperature of the building. Our method

enables to successfully track the threshold variance under controls with high gains, as granted by Theorem 6.

<figure><img src="image_1.png" /><figcaption>Fig. 1. A trajectory with an optimal control-based feedback with high gain.</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>Fig. 2. State variance through time compared to the theoretical lower bound.</figcaption></figure>

## VII. CONCLUSION

In this paper, we solved the (mean-covariance) steering problem for a general class of linear SDEs subject to an input delay. Our objective consisted of steering in finite time the state of the system to a final state with some given final probability distribution. In particular, we proposed a method to minimize the final state covariance. We did so by leveraging the Artstein transformation, thanks to which we derived a “predictor” of the state that tracks a non-delayed SDE. We then established a linear relationship between the predictor covariance and the original state covariance. In particular, this relationship revealed the existence of a structural minimal covariance below which the system can not be steered, and which is essentially due to the presence of the delay. Nevertheless, we proved the system can be steered to any covariance that is greater than this minimal covariance. Finally, we introduced an optimal control-based approach to minimize the system
covariance throughout the whole control horizon, comput-
ing upper bounds for this minimal variance. We assessed
the efficiency of this control strategy via numerical simu-
lations on realistic stochastic systems affected by delays.

Several exciting research directions are listed hereafter.
Extending our approach to linear SDEs with multiplicative
noise would enable the modeling of more sophisticated,
though relevant systems. However, due to the possible lack
of the Artstein transformation, it could be challenging to
derive an explicit expression for the variance threshold
as we did in the present work. Additionally, it would be
interesting to explore systems with multiple input delays.
One could build upon previous studies on optimal control
of ODEs with multiple input delays [26], and then try to
extend such results to more general SDEs.

REFERENCES

[1] Y. Chen, T. T. Georgiou, and M. Pavon, “Optimal Steering of a Linear Stochastic System to a Final Probability Distribution, Part I,” *IEEE Transactions on Automatic Control*, vol. 61, pp. 1158–1169, May 2016. Conference Name: IEEE Transactions on Automatic Control.

[2] Y. Chen, T. T. Georgiou, and M. Pavon, “Optimal Steering of a Linear Stochastic System to a Final Probability Distribution, Part II.,” *IEEE Transactions on Automatic Control*, vol. 61, pp. 1170–1180, May 2016. Conference Name: IEEE Transactions on Automatic Control.

[3] F. Liu and P. Tsiotras, “Optimal Covariance Steering for Continuous-Time Linear Stochastic Systems With Additive Noise,” Feb. 2023. arXiv:2206.11201 [cs, eess].

[4] S.-I. Niculescu, E. I. Verriest, L. Dugard, and J.-M. Dion, “Stability and robust stability of time-delay systems: A guided tour,” in *Stability and Control of Time-delay Systems* (L. Dugard and E. I. Verriest, eds.); Lecture Notes in Control and Information Sciences, pp. 1–71, Berlin, Heidelberg: Springer, 1998.

[5] Z. Artstein, “Linear systems with delayed controls: A re-
duction,” *IEEE Transactions on Automatic Control*, vol. 27,
pp. 869–879, Aug. 1982. Conference Name: IEEE Transactions
on Automatic Control.

[6] M. Krstic and A. Smyshlyaev, *Boundary Control of PDEs: A Course on Backstepping Designs*. SIAM, Sept. 2008. Google-Books-ID: xAjwYxQjU9UC.

[7] V. Léchappé, E. Moulay, F. Plestan, A. Glumineau, and A. Chri-
ette, “New predictive scheme for the control of LTI systems with
input delay and unknown disturbances,” *Automatica*, vol. 52,
pp. 179–184, Feb. 2015.

[8] T. Alves Lima, V. V. Viana, B. C. Torrico, F. G. Nogueira, and D. d. S. Madeira, “Newton-series-based observer–predictor control for disturbed input-delayed discrete-time systems,” *Automatica*, vol. 146, p. 110624, Dec. 2022.

[9] L. Shaikhet, *Lyapunov Functionals and Stability of Stochastic Functional Differential Equations*. Heidelberg: Springer International Publishing, 2013.

[10] F. Cacace, A. Germani, C. Manes, and M. Papi, “Predictor-Based Output-Feedback Control of Linear Stochastic Systems With Large I/O Delays,” *IEEE Transactions on Automatic Control*, vol. 66, pp. 307–313, Jan. 2021. Conference Name: IEEE Transactions on Automatic Control.

[11] B. Øksendal and A. Sulem, “A maximum principle for optimal control of stochastic systems with delay, with applications to finance,” pp. 64–79, 2001. ISSN: .,

[12] L. Chen and Z. Wu, “Maximum principle for the stochastic optimal control problem with delay and application,” *Automatica*, vol. 46, pp. 1074–1080, June 2010.

[13] W. Meng and J. Shi, “A global maximum principle for stochastic optimal control problems with delay and applications,” *Systems & Control Letters*, vol. 150, p. 104909, Apr. 2021.

[14] X. Liang, J. Xu, and H. Zhang, “Solution to stochastic LQ control problem for Itô systems with state delay or input delay,” *Systems & Control Letters*, vol. 113, pp. 86–92, Mar. 2018.

[15] N. Jin, S. Liu, and H. Zhang, “Tracking Problem for Itô Stochastic System with Input Delay,” in *2019 Chinese Control Conference (CCC)*, pp. 1370–1374, July 2019. ISSN: 1934-1768.

[16] W. Wang, L. Xu, J. Xu, X. Li, and H. Zhang, “Linear quadratic optimal control for time-delay stochastic system with partial information,” *International Journal of Systems Science*, vol. 0, pp. 1–12, June 2023. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/00207721.2023.2224801.

[17] Y. Wang and M. P. Chapman, “Risk-averse autonomous sys-
tems: A brief history and recent developments from the per-
spective of optimal control,” *Artificial Intelligence*, vol. 311,
p. 103743, Oct. 2022.

[18] A. Manitius and A. Olbrot, “Finite spectrum assignment problem for systems with delays,” *IEEE Transactions on Automatic Control*, vol. 24, pp. 541–552, Aug. 1979. Conference Name: IEEE Transactions on Automatic Control.

[19] J. Yong and X. Y. Zhou, *Stochastic controls: Hamiltonian systems and HJB equations*. No. 43 in Applications of mathematics, New York: Springer, 1999.

[20] J.-M. Coron, *Control and Nonlinearity*, vol. 136 of *Mathematical Surveys and Monographs*. Providence, Rhode Island: American Mathematical Society, Aug. 2009.

[21] N. Mahmudov, “Controllability of linear stochastic systems,” *IEEE Transactions on Automatic Control*, vol. 46, pp. 724–731, May 2001.

[22] N. Touzi, “Introduction,” in *Optimal Stochastic Control, Stochastic Target Problems, and Backward SDE* (N. Touzi, ed.), Fields Institute Monographs, pp. 1–4, New York, NY: Springer, 2013.

[23] Y. Wang, D. Yang, J. Yong, and Z. Yu, “Exact controllability of linear stochastic differential equations and related problems,” *Mathematical Control and Related Fields*, vol. 7, pp. 305–345, Mar. 2017. Publisher: Mathematical Control and Related Fields.

[24] S. van der Zwan and I. Pothof, “Operational optimization of district heating systems with temperature limited sources,” *Energy and Buildings*, vol. 226, p. 110347, Nov. 2020.

[25] S. Prabakaran, I. C. Garcia, and J. U. Mora, “A temperature stochastic model for option pricing and its impacts on the electricity market,” *Economic Analysis and Policy*, vol. 68, pp. 58–77, Dec. 2020.

[26] M. Basin and J. Rodriguez-Gonzalez, “Optimal control for linear systems with multiple time delays in control input,” *IEEE Transactions on Automatic Control*, vol. 51, pp. 91–97, Jan. 2006. Conference Name: IEEE Transactions on Automatic Control.