# Robust Recurrence of Discrete-Time Infinite-Horizon Stochastic Optimal Control with Discounted Cost

Robert Moldenhauer, Dragan Nesic, Mathieu Granzotto, Romain Postoyan, Andrew Teel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Robert Moldenhauer, Dragan Nesic, Mathieu Granzotto, Romain Postoyan, Andrew Teel. Robust Recurrence of Discrete-Time Infinite-Horizon Stochastic Optimal Control with Discounted Cost. 13th IFAC Symposium on Nonlinear Control Systems (NOLCOS, Jul 2025, Reykjavik, Iceland. hal-05054596

HAL Id: hal-05054596

https://hal.science/hal-05054596v1

Submitted on 2 May 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Robust Recurrence of Discrete-Time
Infinite-Horizon Stochastic Optimal Control
with Discounted Cost *

Robert H. Moldenhauer *,** Dragan Nešić *

Mathieu Granzotto * Romain Postoyan ** Andrew R. Teel ***

* Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3010, Australia (e-mail: moldenhauer.r@student.unimelb.edu.au, {dnesic,mathieu.granzotto}@unimelb.edu.au)

** Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France
(e-mail: {robert.moldenhauer, romain.postoyan}@univ-lorraine.fr)

*** Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA 93106 USA (e-mail: teel@ece.ucsb.edu)

Abstract We analyze the stability of general nonlinear discrete-time stochastic systems controlled by optimal inputs that minimize an infinite-horizon discounted cost. Under a novel stochastic formulation of cost-controllability and detectability assumptions inspired by the related literature on deterministic systems, we prove that uniform semi-global practical recurrence holds for the closed-loop system, where the adjustable parameter is the discount factor. Under additional continuity assumptions, we further prove that this property is robust.

Keywords: Optimal control, Stochastic systems, Dynamic programming, Recurrence, Robust stability

# 1. INTRODUCTION

Optimal control seeks to minimize a cost incurred during the system operation (Kirk, 2004), which is a naturally compelling way to ensure desirable performance properties for the controlled system. Optimal control is able to address decision-making problems not only in control engineering, but also in artificial intelligence, operations research, economics, medicine, and so on. We talk of stochastic optimal control when the dynamical system includes a random input and expected cost is minimized. This occurs in a range of applications involving substantial uncertainty, such as electric vehicles (Moura et al., 2011), wireless networked control systems (Wang et al., 2016), biological systems (Acikgoz and Diwekar, 2010; Todorov, 2005), and finance (Soner, 2004). Moreover, reinforcement learning is often formulated in terms of such stochastic optimal control problems, usually referred to as Markov decision processes in that context (Sutton and Barto, 2018).

In this paper, we focus on stochastic optimal control in discrete time over an infinite horizon with discounted costs. The optimal control problem is solved over stationary policies, and discounting ensures that policies with finite costs exist for a large class of stochastic systems. Our objective is to derive conditions under which the closed-loop feedback system resulting from near-optimal policies is stable. Allowing near-optimal, not just optimal

policies is of practical importance, as one may not be able to find exact optimal policies algorithmically, and approximate dynamic programming is employed instead Bertsekas (2012). The recent work by Granzotto et al. (2024a) achieved stability guarantees for the stochastic linear discounted quadratic problem. We are not aware of any equivalent results for general nonlinear systems and costs, which are therefore the subject of this paper.

We found that the notion of semi-global practical recurrence, which was first introduced by Grammatico et al. (2013), best describes the general behaviour of the considered closed loop. Roughly speaking, a set is said to be recurrent if almost all solutions revisit the set infinitely often, see (Subbaraman and Teel, 2013), which is inspired by Meyn and Tweedie (1993). Semi-global practical recurrence is a weaker property, that Grammatico et al. (2013) motivated through the following analogy: "A walking robot in a stochastically rough terrain is guaranteed to eventually fall down (with probability one as $t \to \infty$). However, one can still meaningfully quantify stochastic stability in terms of expected time to failure. As long as the robot does not fall down, it keeps returning back to certain walking postures, that we would hence call recurrent" (Grammatico et al., 2013, Section 7). We consider this property with respect to the discount factor $\gamma \in (0, 1)$. It then means that, if $\gamma$ is sufficiently close to 1, failure occurs late in expectation and, until that happens, states return near the recurrent set with high probability. For near-optimal instead of optimal policies, semi-global practical recurrence is preserved, albeit to a generally larger recurrent set depending on the cost gap.

<footnote>* This work was supported by the ARC under the Discovery Project DP210102600, by the France-Australia collaboration project IRP-ARS CNRS, and by the ANR grant OLYMPIA ANR-23-CE48-0006.</footnote>
Under additional conditions, we show robustness to model perturbations. This is important, as the optimal policies are typically designed on an ideal plant model, which may differ from the real-world plant. The semi-global practical recurrence is then with respect to the discount factor and a bound on the magnitude of disturbances.

The main tools used to obtain our results are Bellman equation and Lyapunov analysis. This follows in the footsteps of existing results on deterministic systems by Grimm et al. (2005); Postoyan et al. (2017); Granzotto et al. (2021, 2024a,b). There, under certain cost-controllability and detectability assumptions, a Lyapunov function is constructed by combining the optimal value function and Lyapunov like function characterizing detectability, such that its decrease is guaranteed by combining the Bellman equation with detectability. In this paper, we weaken the cost-controllability and detectability assumptions made by Postoyan et al. (2017) in order to address a large class of stochastic systems. We demonstrate that the new assumptions generalize the stochastic linear quadratic problem as a special case. Under these assumptions as well as some mild regularity and measurability conditions, we prove the desired semi-global practical recurrence. Robustness is then achieved under additional continuity conditions.

The remainder of this paper is organized as follows. Section 2 contains notation and basic definitions. In Section 3, we introduce the system and optimal control problem and define strictly causal random closed-loop solutions. We start Section 4 by introducing the stochastic versions of cost-controllability and detectability assumptions and illustrate them on an example, before stating the main result on semi-global practical recurrence. Section 5 states the robust version under additional continuity assumptions. Section 6 concludes the paper. Proofs are omitted for space reasons and are available in (Moldenhauer et al., 2025).

## 2. NOTATION AND BASIC DEFINITIONS

The symbol $\mathbb{R}$ denotes the set of real numbers, $\mathbb{R}_{\ge 0}$ the set of non-negative real numbers and $\mathbb{R}_{> 0}$ the set of positive real numbers. The notation $\mathbb{Z}_{\ge 0}$ ($\mathbb{Z}_{> 0}$, resp.) denotes the nonnegative (positive, resp.) integers. The Euclidean norm is denoted by $|\cdot|$. The spectral norm of a matrix is denoted by $||\cdot||_2$. Given sets $X, Y$, a function $f: X \to Y$ and $B \subseteq Y$, the pre-image of $B$ under $f$ is denoted by $f^{-1}(B)$. The notation $\mathbb{B}^n := \{x \in \mathbb{R}^n \mid |x| < 1\}$ denotes the $n$-dimensional open unit ball of $\mathbb{R}^n$ centered around the origin, where we just write $\mathbb{B}$ when $n$ is clear from context. For any set $S \subseteq \mathbb{R}^n$, the notation $\text{cl}(S)$ denotes the closure of $S$. A function $\alpha: \mathbb{R}_{\ge 0} \to \mathbb{R}_{\ge 0}$ is said to belong to class $K_{\infty}$ ($\alpha \in K_{\infty}$) if it is continuous, strictly increasing, zero at zero and unbounded. A function $f: X \to Y$, where $X \subseteq \mathbb{R}^n$ and $Y \subseteq \mathbb{R}^m$, is said to be locally bounded if images of compact sets are bounded. A function $f: X \to \mathbb{R}_{\ge 0}$, where $X \subseteq \mathbb{R}^n$, is said to be radially unbounded if pre-images of bounded sets are bounded; note that if $X$ is bounded, then $f$ is always radially unbounded. A set-valued mapping $\Gamma$ from a set $X$ to a set $Y$ is a mapping from $X$ to subsets of $Y$, and is notated as $\Gamma: X \rightleftharpoons Y$.

A measurable space is a pair $(\Omega, \mathcal{F})$ of a set $\Omega$ and a $\sigma$-algebra $\mathcal{F}$ over $\Omega$. Over subsets of $\mathbb{R}^n$, it is implied that the Borel $\sigma$-algebra is used as $\sigma$-algebra. A set $F \subseteq \Omega$ is said to be $\mathcal{F}$-measurable (or just measurable, if $\mathcal{F}$ is clear from context) if $F \in \mathcal{F}$. A probability space $(\Omega, \mathcal{F}, \mathbb{P})$ consists of a measurable space $(\Omega, \mathcal{F})$ and a probability measure $\mathbb{P}: \mathcal{F} \to [0, 1]$. Given measurable spaces $(\Omega, \mathcal{F})$ and $(\mathcal{X}, \mathcal{G})$, a function $f: \Omega \to \mathcal{X}$ is said to be measurable, if $f^{-1}(B) \in \mathcal{F}$ for any $B \in \mathcal{G}$. If, in addition, $(\Omega, \mathcal{F}, \mathbb{P})$ is a probability space, then such a measurable function is referred to as random variable, and we denote it by a boldface lowercase letter. The expected value of a random variable $\mathbf{x}$ is denoted by $\mathbb{E}[\mathbf{x}] = \int \mathbf{x} d\mathbb{P} = \int \mathbf{x}(\omega)d\mathbb{P}(\omega)$. For $F \in \mathcal{F}$ (such a set is referred to as an event) with $\mathbb{P}(F) > 0$, the conditional expectation of $\mathbf{x}$ is denoted by $\mathbb{E}[\mathbf{x}|F] = \mathbb{P}(F)^{-1} \int_F \mathbf{x} d\mathbb{P}$. We use round brackets, i.e., $\mathbb{P}(F)$, for the probability of an event $F \in \mathcal{F}$, and square brackets, i.e., $\mathbb{P}[\cdot]$, on logical predicates involving random variables, as short-hand notation for the set of all $\omega \in \Omega$ that satisfy the logical expression.

Whenever we apply a function or arithmetic operation on a set or a random variable, we understand it to be the set or random variable that results when the function or operation is applied element-wise or path-wise, respectively. For example, if $f$ is a function and $X$ is a set, $f(X) := \{f(x) \mid x \in X\}$. If $f$ is a function and $\mathbf{x}: \Omega \to \mathcal{X}$ is a random variable, then $f(\mathbf{x})$ is the random variable with $f(\mathbf{x})(\omega) = f(\mathbf{x}(\omega))$ for all $\omega \in \Omega$.

## 3. PROBLEM STATEMENT

In this section, we introduce the system and costs, followed by defining a set-valued near-optimal feedback policy and the strictly causal random closed-loop solutions it generates.

### 3.1 Class of systems and costs

Consider a function $f: \mathcal{X} \times \mathcal{U} \times \mathcal{V} \to \mathcal{X}$, where $\mathcal{X} \subseteq \mathbb{R}^n$ and $\mathcal{U} \subseteq \mathbb{R}^m$ are closed, $\mathcal{V} \subseteq \mathbb{R}^q$ is measurable and $n, m, q \in \mathbb{Z}_{> 0}$, and the discrete-time stochastic control system

$$x^+ = f(x, u, v) \quad (1)$$

with state $x \in \mathcal{X}$, control input $u \in \mathcal{U}$ and random input $v \in \mathcal{V}$. Let a probability space $(\Omega, \mathcal{F}, \mathbb{P})$, a random variable $\mathbf{v}: \Omega \to \mathcal{V}$ and an independently identically distributed (i.i.d.) sequence of random variables $(\mathbf{v}_0, \mathbf{v}_1, \dots)$ with the same distribution as $\mathbf{v}$ be given, where $\mathbf{v}_k$ acts as random input at time $k$.

In optimal control of stochastic systems, the optimization problem is often solved over feedback policies, rather than sequences of inputs (which is customary for deterministic optimal control, e.g., Grimm et al. (2005); Postoyan et al. (2017); Granzotto et al. (2021)). Policies allow the controller to respond to random inputs through their effect on the current state, see, e.g., Hernández-Lerma and Lasserre (1996); Bertsekas (2012). Let the set of admissible (feedback) policies be

$$\mathcal{H}_a := \{h: \mathcal{X} \to \mathcal{U} \mid h \text{ is measurable}\}. \quad (2)$$

We denote the random closed-loop solution to (1) at the $k^{\text{th}}$-step starting at state $x \in \mathcal{X}$ and under feedback policy $h \in \mathcal{H}_a$ by $\varphi(k, x, h): \Omega \to \mathcal{X}$, that is,
$$
\begin{equation}
\begin{aligned}
\varphi(k+1, x, h) &:= f(\varphi(k, x, h), h(\varphi(k, x, h)), \mathbf{v}_k), \\
\varphi(0, x, h) &:= x.
\end{aligned}
\tag{3}
\end{equation}
$$

Note that, as $\mathbf{v}_0, \dots$ are random variables, so is $\varphi(k, x, h)$.

We consider in this work the expected infinite-horizon dis-
counted cost over random closed-loop solutions $\varphi(\cdot, x, h)$
of (1). That is, for any $x \in \mathcal{X}$ and $h \in \mathcal{H}_a$, let

$$
J_{\gamma}(x, h) := \mathbb{E} \left[ \sum_{k=0}^{\infty} \gamma^k \ell(\varphi(k, x, h), h(\varphi(k, x, h))) \right] \quad (4)
$$

be the cost function, where $\ell: \mathcal{X} \times \mathcal{U} \to \mathbb{R}_{\ge 0}$ is the stage cost, which takes non-negative values, and $\gamma \in (0,1)$ is the discount factor. Note that (4) uses stationary policies, meaning that the same function $h$ is applied at every time step, rather than non-stationary, where the functions may differ between time steps. Under certain conditions it can be shown that stationary policies achieve the same minimum of the cost function as non-stationary policies (Hernández-Lerma and Lasserre, 1996, Theorem 4.2.3(d)).

The discount factor $\gamma \in (0, 1)$ generally ensures finiteness of $J_{\gamma}(x, h)$. In many stochastic systems, in particular with additive noise, no policy leads to convergence to 0 due to persistent random state variation. Then, the undiscounted ($\gamma = 1$) cost function is infinite for any policy, while the discounted ($\gamma \in (0, 1)$) cost function is finite for suitable policies keeping expected stage costs below a certain bound.

3.2 Optimal and near-optimal controls

For each $\gamma \in (0,1)$ we define the (optimal) value function
$V_{\gamma}: \text{dom}(V_{\gamma}) \to \mathbb{R}_{\ge 0}$ with

$$
\mathrm{dom}(V_{\gamma}) := \left\{ x \in \mathcal{X} : \inf_{h \in \mathcal{H}_a} J_{\gamma}(x, h) < \infty \right\} \quad (5)
$$

and

$$
V_{\gamma}(x) := \inf_{h \in \mathcal{H}_a} J_{\gamma}(x, h) \quad (6)
$$

for all $x \in \text{dom}(V_\gamma)$. The (near-)optimal feedback policy is
based on Bellman equation

$$
V_{\gamma}(x) = \inf_{u \in \mathcal{U}} \left( \ell(x, u) + \gamma \mathbb{E}[V_{\gamma}(f(x, u, \mathbf{v}))] \right) \quad \forall x \in \mathcal{X}, \quad (7)
$$

see (Hernández-Lerma and Lasserre, 1996, Theorem 4.2.3)
for sufficient conditions for satisfaction of (7). Generally,
the infimum may not be attained as a minimum, and in
practice only approximate solutions may be found. To
address this phenomenon, our controls can exceed the
optimal cost by at most a state-dependent and known error
bound $\eta: \mathcal{X} \to \mathbb{R}_{\ge 0}$ and are hence referred to as *near-
optimal*. With $\ell(x, u) + \gamma \mathbb{E}[V_{\gamma}(f(x, u, v))]$, which is the cost
incurred from $u$ followed by optimal control, being desired
to be no more than $\eta(x)$ above the minimal cost $V_{\gamma}(x)$,
the near-optimal set-valued feedback policy $\mathcal{U}_{\gamma}^{\eta}: \mathcal{X} \Rightarrow \mathcal{U}$
is defined by

$$
\begin{equation}
\begin{aligned}
\mathcal{U}_{\gamma}^{\eta}(x) := \{ & u \in \mathcal{U} \mid V_{\gamma}(x) + \eta(x) \\
& \ge \ell(x, u) + \gamma \mathbb{E}[V_{\gamma}(f(x, u, \mathbf{v}))] \}
\end{aligned}
\tag{8}
\end{equation}
$$

for any $x \in \mathcal{X}$. For any state $x$, we allow for any control $u \in \mathcal{U}_\gamma^\eta(x)$ to be applied to the system, provided a strict causality property is verified, which is introduced next. Note that (8) gives optimal inputs when $\eta(x) = 0$ for any $\forall x \in \mathcal{X}$, which our results allow.

3.3 Strictly causal random closed-loop solutions

We now define solutions to the closed loop of system (1)
and near-optimal feedback (8). We will study in particular
the strictly causal random solutions of such a system, that
is, when the control $u \in \mathcal{U}_{\gamma}^{\eta}(x)$ may depend only on past
random inputs, but not on the current and future random
inputs. Not only is this property required to establish
our results, but (Grammatico et al., 2013, Example 2)
demonstrates that without it robustness to even small
perturbations may not be guaranteed.

To formalize strict causality, let the natural filtration
$\mathcal{F}_{-1} \subseteq \mathcal{F}_0 \subseteq \dots$ of the random input sequence $(\mathbf{v}_0, \mathbf{v}_1, \dots)$
be given by

$$
\mathcal{F}_k := \{ \{\omega \in \Omega : (\mathbf{v}_0(\omega), \dots, \mathbf{v}_k(\omega)) \in F\} | F \in \mathcal{B}(\mathbb{R}^{q(k+1)}) \} \tag{9}
$$

for $k \in \mathbb{Z}_{\ge 0}$, where $\mathcal{B}(\mathbb{R}^{q(k+1)})$ denotes the Borel $\sigma$-algebra, and $\mathcal{F}_{-1} := \{\emptyset, \Omega\}$. Every $\mathcal{F}_k$ is a sub-$\sigma$-algebra of $\mathcal{F}$. The desired notion of solutions is now defined as follows.

**Definition 1.** A sequence **x** = (x₀, x₁, ...) of random variables **x**<sub>k</sub> : Ω → 𝒳 is a strictly causal random closed-loop solution of (1) and (8) for initial state x₀ ∈ 𝒳, denoted by **x** ∈ S<sub>γ</sub><sup>η</sup>(x₀), if x₀(ω) = x₀ for all ω ∈ Ω, and for any k ∈ Z<sub>≥0</sub> there exists an F<sub>k-1</sub>-measurable random variable **u**<sub>k</sub> : Ω → 𝒳 such that **u**<sub>k</sub>(ω) ∈ U<sub>γ</sub><sup>η</sup>(x<sub>k</sub>(ω)) and x<sub>k+1</sub>(ω) = f(x<sub>k</sub>(ω), u<sub>k</sub>(ω), v<sub>k</sub>(ω)) for all ω ∈ Ω. ◇

Note that Definition 1 implies that $\mathbf{x}_k$ is $\mathcal{F}_{k-1}$-measurable
for every $k \in \mathbb{Z}_{\ge 0}$. We use the concept of solutions in
Definition 1 rather than that of Section 3.1 for greater
generality, as it allows all control selections in $\mathcal{U}_{\gamma}^{\eta}$ (in par-
ticular, different controls for same states, but at different
times), as long as strict causality is maintained. Neverthe-
less, any solution $\varphi(\cdot, x, h)$ for $x \in \mathcal{X}$ and where $h \in \mathcal{H}_a$
is a selection of $\mathcal{U}_{\gamma}^{\eta}$ (that is, $h(x) \in \mathcal{U}_{\gamma}^{\eta}(x)$ for all $x \in \mathcal{X}$),
qualifies as a solution according to Definition 1. For further
reference, see strictly causal generalized random solutions
to stochastic difference inclusions in (Grammatico et al.,
2013, Section 5), of which Definition 1 is a special case.

4. SEMI-GLOBAL PRACTICAL RECURRENCE

In this section we present the main result about semi-
global practical recurrence, inspired by (Grammatico
et al., 2013, Definition 5), for strictly causal random
closed-loop solutions of system (1) and feedback policy
(8), as defined in Definition 1, under certain assumptions
that are stated in the following.

4.1 Assumptions

We start with mild measurability assumptions to ensure
that expectations exist.

Assumption 2. The functions $f, \ell$ and $\eta$ are measurable. Furthermore, $V_\gamma : \mathcal{X} \to \mathbb{R}$ is finite and measurable for any $\gamma \in (0, 1)$. $\diamond$

The main assumptions are similar to the ones made in
Grimm et al. (2005); Postoyan et al. (2017) on determin-
istic systems, but adapted for stochastic systems. They
relate the costs in the optimal control problem to a state
space metric $\sigma: \mathcal{X} \to \mathbb{R}_{>0}$ that recurrence revolves around. Function $\sigma$ can be defined as $|\cdot|$ or $|\cdot|^2$ when studying recurrence of the origin, or the distance to a subset of $\mathcal{X}$ that is desired to be recurrent. The first condition is cost-controllability, which in Postoyan et al. (2017) guarantees existence of a $\mathcal{K}_\infty$-function of $\sigma(x)$ that upper bounds the value function $V_\gamma(x)$ for all discount factors $\gamma \in (0,1)$. The second condition is detectability of the stage cost, and, roughly speaking, means, that small stage costs imply that $\sigma(x)$ becomes small eventually over time along solutions to (1). After stating the assumptions now, we explain them in more detail and illustrate them on an example below.

**Assumption 3.** Let $\sigma: \mathcal{X} \to \mathbb{R}_{>0}$ be measurable, locally bounded and radially unbounded.

(i) *Cost-controllability with lower bound:* There exist $\bar{\alpha}_V \in \mathcal{K}_\infty$ and $e_1, e_2 \ge 0$ such that for any $\gamma \in (0,1)$ there exists $c_\gamma \ge 0$ such that for any $x \in \mathcal{X}$,

$$-e_1 \le V_\gamma(x) - \frac{c_\gamma}{1-\gamma} \le e_2 + \bar{\alpha}_V(\sigma(x)). \quad (10)$$

Furthermore, there exists $c \ge 0$ such that $c_\gamma \le c$ for any $\gamma \in (0,1)$.

(ii) *Detectability:* There exist a measurable function $W: \mathcal{X} \to \mathbb{R}_{>0}$, functions $\bar{\alpha}_W, \alpha_W \in \mathcal{K}_\infty$ and $d > 0$, such that for any $x \in \mathcal{X}$ and $u \in \mathcal{U}$,

$$W(x) \le \bar{\alpha}_W(\sigma(x)), \quad \mathbb{E}[W(f(x, u, \mathbf{v}))] - W(x) \le -\alpha_W(\sigma(x)) + \ell(x, u) + d. \quad (11)$$

<figure><img src="image_3.png" /></figure>

Assumptions 3 generalizes (Postoyan et al., 2017, Assumption 1) from deterministic to stochastic systems through the parameters $c, d, e_1, e_2 \ge 0$. These parameters are designed to capture types of behaviour unique to stochastic systems, and cover a broad range of ways the stochastic input impacts the system, like additive and multiplicative noise. The parameters $c$ and $d$, as well as the near-optimality bound $\eta$, will appear in the statements of our recurrence results, in the way that lower values yield a smaller recurrent set. When $c, d, e_1, e_2$ and $\eta$ are all zero, (Postoyan et al., 2017, Assumption 1) is recovered.

Now let us explain Assumption 3 in detail. Regarding item (i), for $x \in \mathcal{X}$ with $\sigma(x) = 0$ the value function $V_\gamma(x)$ differs from $c_\gamma/(1-\gamma)$ only by the constants $e_1, e_2$, and for general $x \in \mathcal{X}$ the upper bound may increase through $\bar{\alpha}_V$ as $\sigma(x)$ increases. An equivalent formulation of part (3) is obtained if $c_\gamma/(1-\gamma)$ and $e_1$ are replaced by $\inf_{x \in \mathcal{X}} V_\gamma(x)$ and 0, respectively. However, $c_\gamma$ will appear in the main theorem and the presented version with $e_1$ allows for more choices for $c_\gamma$ and easier verification. The reason to include $c_\gamma$ is the same given in Section 3.1 for discounting. Random inputs, particularly additive noise, can lead to a certain minimum expected stage cost at every time instant, causing the value function $V_\gamma$ to blow up as $\gamma$ approaches 1. This is captured in the $c_\gamma/(1-\gamma)$ term, where $c_\gamma$ can be interpreted as a minimum asymptotic average stage cost and is required to be bounded for $\gamma \in (0,1)$. The division by $1-\gamma$ comes from the geometric series, when these costs $c_\gamma$ are added up with discount factor $\gamma$. Note that $e_1, e_2$ and $\bar{\alpha}_V$ need to be independent of $\gamma$. This means that, although $V_\gamma(x)$ may blow up as $\gamma \to 1$, the differences

$V_\gamma(x_1)-V_\gamma(x_2)$ for same $\gamma$ and different $x_1, x_2 \in \mathcal{X}$ cannot. The example below illustrates that this is a reasonable assumption.

Moving on to item (ii) of Assumption 3, $W$ can be chosen constantly zero if there exists $\alpha_W \in \mathcal{K}_\infty$ such that $\ell(x, u) \ge \alpha_W(\sigma(x))$ for all $(x, u) \in \mathcal{X} \times \mathcal{U}$. The formulation with $W$ is more general as it allows positive semi-definite $\ell$ that may depend on only some of the state components, for example. Inequality (11) resembles a dissipation-type inequality with storage function $W$ and supply rate $\ell$. While the special case $W=0$ provides that $\sigma(x)$ is small right away when $\ell(x, u)$ is small, in the general case small $\ell(x, u)$ and large $\sigma(x)$ mean that $W$ decreases in expectation over time due to (11), but since it cannot become negative, $\sigma(x)$ must become small eventually. Keeping to the analogy of dissipativity, one may expect the random input to supply some average amount of energy at every time instant, which is captured in $d$.

To see how Assumption 3 is consistent with and generalizes the linear stochastic case studied in Granzotto et al. (2024a), consider the next example.

**Example 4.** Consider the linear system

$$x^+ = f(x, u, v) := Ax + Bu + Lv \quad (12)$$

with matrices $A \in \mathbb{R}^{n \times n}, B \in \mathbb{R}^{n \times m}, L \in \mathbb{R}^{n \times q}$, and the stage cost

$$\ell(x, u) = x^\top Q x + u^\top R u, \quad (13)$$

where $Q \in \mathbb{R}^{n \times n}$ is positive semi-definite and $R \in \mathbb{R}^{m \times m}$ is positive definite. We are interested in recurrence of (open neighbourhoods of) the origin, and therefore choose $\sigma(x) := |x|$. If $(A, B)$ is stabilizable and $(Q^{1/2}, A)$ is detectable then, for any $\gamma \in (0,1)$, the algebraic Riccati equation

$$P_\gamma = Q + \gamma A^\top P_\gamma A - \gamma^2 A^\top P_\gamma B (R + \gamma B^\top P_\gamma B)^{-1} B^\top P_\gamma A \quad \text{has a positive definite solution } P_\gamma \in \mathbb{R}^{n \times n}, \text{ and the value function takes the form}$$

$$V_\gamma(x) = x^\top P_\gamma x + \frac{\gamma}{1-\gamma} \mathbb{E}[\mathbf{v}^\top L^\top P_\gamma L \mathbf{v}] \quad (14)$$

for any $x \in \mathbb{R}^n$ (Davis and Vinter, 1985, Chapter 6.2). If $P_\gamma$ is bounded over $\gamma \in (0,1)$, then item (i) of Assumption 3 is verified with $e_1 := e_2 := 0$,

$$c_\gamma := \gamma \mathbb{E}[\mathbf{v}^\top L^\top P_\gamma L \mathbf{v}], \quad \bar{\alpha}_V(s) := \sup_{\gamma \in (0,1)} \|P_\gamma\|_2 s^2 \quad (15)$$

(recall that $\| \cdot \|_2$ denotes the spectral norm, i.e., the largest singular value, of a matrix). Note that, despite $V_\gamma(x) \to \infty$ as $\gamma \to 1$, the difference $V_\gamma(x_1) - V_\gamma(x_2) = x_1^\top P_\gamma x_1 - x_2^\top P_\gamma x_2$ depends only on the quadratic term and therefore remains bounded as $\gamma \to 1$. Regarding detectability, if $(Q^{1/2}, A)$ is detectable, by (Postoyan et al., 2017, Appendix B) there exist a matrix $M \in \mathbb{R}^{n \times n}$ and a constant $a_W \in \mathbb{R}_{>0}$ such that item (ii) of Assumption 3 holds with $W(x) := x^\top M x, \alpha_W(s) := a_W s^2$, and $d := \mathbb{E}[\mathbf{v}^\top L^\top M L \mathbf{v}]$. $\diamond$

<figure><img src="image_4.png" /></figure>

Finally, we make the next mild assumption.

**Assumption 5.** The function $f$ and the set-valued mapping

$$\mathcal{U}_{(0,1)}^\eta : \mathcal{X} \Rightarrow \mathcal{U}, \quad x \mapsto \bigcup_{\gamma \in (0,1)} \mathcal{U}_\gamma^\eta(x) := \mathcal{U}_{(0,1)}^\eta(x) \quad (16)$$
are locally bounded, which for the set-valued mapping $U_{(0,1)}^{\eta}$ means that, for any compact set $D \subseteq \mathcal{X}$, the image $U_{(0,1)}^{\eta}(D) := \bigcup_{x \in D} U_{(0,1)}^{\eta}(x)$ is bounded. $\diamond$

To ease the test of Assumption 5, the next proposition gives sufficient conditions for local boundedness of $U_{(0,1)}^{\eta}$.

**Proposition 6.** Let Assumption 2 and 3 hold. Further assume that $\eta$ is locally bounded and that $\ell$ is level-bounded in $u$ locally uniformly in $x$, i.e., for each compact set $D \subseteq \mathcal{X}$ the function $\ell_D: \mathcal{U} \to \mathbb{R}_{\ge 0}$, $u \mapsto \inf_{x \in D} \ell(x, u) =: \ell_D(u)$ is radially unbounded. Then, the set-valued mapping $U_{(0,1)}^{\eta}$ defined in (16) is locally bounded. $\diamond$

The level-boundedness condition on $\ell$ (also see (Rockafellar and Wets, 1998, Definition 1.16)) penalizes large control effort and is fulfilled for the quadratic costs (13) due to $R$ being positive definite. The proof of Proposition 6 can be found in (Moldenhauer et al., 2025).

## 4.2 Main result

The main result of this section is presented in two parts: a lemma and the main recurrence theorem. These two parts are adaptions of items (i) and (ii) of the definition of semi-global practical recurrence in (Grammatico et al., 2013, Definition 5), respectively. The lemma states that for any $\Delta_0 > 0$, there exists a bounded set of the form $\sigma^{-1}(\Delta\mathbb{B})$ for some $\Delta > 0$, in which closed-loop solutions starting in $\sigma^{-1}(\Delta_0\mathbb{B})$ stay for a desired time $T$ with at least a desired probability $1-p$. Hereby, $\Delta$ is independent of $\gamma$. This is formalized below.

**Lemma 7.** Let Assumptions 2, 3 and 5 hold. Then,

$$ 
\begin{array}{l}
\forall \Delta_0 > 0 \quad \forall p > 0 \quad \forall T \in \mathbb{Z}_{\ge 0} \quad \exists \Delta > 0 \\
\forall \gamma \in (0, 1) \quad \forall x_0 \in \sigma^{-1}(\Delta_0 \mathbb{B}) \quad \forall \mathbf{x} \in S_{\gamma}^{\eta}(x_0) : \quad (17) \\
\mathbb{P}[\sigma(\mathbf{x}_k) < \Delta \quad \forall k \in \{0, \dots, T\}] > 1 - p. \quad \diamond
\end{array}
$$

The proof of Lemma 7 can be found in (Moldenhauer et al., 2025).

Leading up to the main theorem, for $\gamma \in (0, 1)$ and $\delta > 0$ we define the set

$$ A_{\gamma, \delta} := \{x \in \mathcal{X} \mid \alpha_W(\sigma(x)) \le c_{\gamma} + d + \eta(x) + \delta\}. \quad (18) $$

We will show semi-global practical recurrence of these sets, in the sense that for any $p > 0$, $\delta > 0$ and any maximum value $\Delta_0$ of $\sigma(x_0)$ for potential initial states $x_0$, it can be ensured with $\gamma$ close enough to 1 that the closed-loop solutions $\mathbf{x} \in S_{\gamma}^{\eta}(x_0)$ enter $A_{\gamma, \delta}$ with probability at least $1-p$ within some time $T$ independent of $\gamma, x_0$ and $\mathbf{x}$. “Semi-global” refers to the fact that, while for fixed $\gamma$ there can be $x_0$ (that have high $\sigma(x_0)$) for which recurrence does not hold, for any fixed $x_0$ recurrence does hold if $\gamma$ is close enough to 1. “Practical” is the analogous concept for approaching $A_{\gamma, 0}$. While for fixed $\gamma$ the recurrent set generally needs to be larger than $A_{\gamma, 0}$, namely $A_{\gamma, \delta}$, $\delta$ can be arbitrarily small if $\gamma$ is close enough to 1.

**Theorem 8.** Let Assumptions 2, 3 and 5 hold. Then,

$$ 
\begin{array}{l}
\forall \Delta_0, \delta > 0 \quad \forall p > 0 \quad \exists T \in \mathbb{Z}_{\ge 0} \quad \exists \gamma^* \in (0, 1) \\
\forall \gamma \in (\gamma^*, 1) \quad \forall x_0 \in \sigma^{-1}(\Delta_0 \mathbb{B}) \quad \forall \mathbf{x} \in S_{\gamma}^{\eta}(x_0) : \quad (19) \\
\mathbb{P}[\exists k \in \{0, \dots, T\} : \mathbf{x}_k \in A_{\gamma, \delta}] > 1 - p. \quad \diamond
\end{array}
$$

The proof of Theorem 8 can be found in (Moldenhauer et al., 2025).

Note that, since the system (1) is time-invariant, Lemma 7 and Theorem 8 can be combined to guarantee recurrence farther into the future as well, as for any time $T_0 \in \mathbb{Z}_{\ge 0}$, Lemma 7 can be used to construct a set in which solutions stay until $T_0$ with high probability, starting from which Theorem 8 can guarantee recurrence in overall time within $\{T_0, \dots, T_0 + T\}$ with high probability.

Next, Theorem 8 is generalized to the case with small strictly causal perturbations.

## 5. ROBUSTNESS

In this section, we state robust semi-global practical recurrence under additional continuity assumptions. Solutions are defined similarly to Definition 1, except that in addition to $\mathbf{u}_k$, perturbation random variables $\mathbf{p}_k^i, i \in \{1, 2, 3, 4\}$ are introduced.

**Definition 9.** For $\varepsilon > 0$, a sequence $\mathbf{x} = (\mathbf{x}_0, \mathbf{x}_1, \dots)$ of random variables $\mathbf{x}_k: \Omega \to \mathcal{X}$ is an $\varepsilon$-perturbed strictly causal random closed-loop solution of (1) and (8) for initial state $x_0 \in \mathcal{X}$, denoted by $\mathbf{x} \in S_{\gamma, \varepsilon}^{\eta}(x_0)$, if $\mathbf{x}_0 = x_0$, and for any $k \in \mathbb{Z}_{\ge 0}$ there exist $\mathcal{F}_{k-1}$-measurable $\mathbf{u}_k: \Omega \to \mathcal{U}$, $\mathbf{p}_k^2: \Omega \to \varepsilon\mathbb{B}^n$, $\mathbf{p}_k^3: \Omega \to \varepsilon\mathbb{B}^m$ such that

$$ \mathbf{x}_k + \mathbf{p}_k^2 \in \mathcal{X}, \quad \mathbf{u}_k \in U_{\gamma}^{\eta}(\mathbf{x}_k + \mathbf{p}_k^2) + \mathbf{p}_k^3, \quad (20) $$

and $\mathcal{F}_k$-measurable $\mathbf{p}_k^1, \mathbf{p}_k^4: \Omega \to \varepsilon\mathbb{B}^n$ such that

$$ \mathbf{x}_k + \mathbf{p}_k^1 \in \mathcal{X}, \quad \mathbf{x}_{k+1} = f(\mathbf{x}_k + \mathbf{p}_k^1, \mathbf{u}_k, \mathbf{v}_k) + \mathbf{p}_k^4. \quad (21) $$

Perturbations $\mathbf{p}_k^2$ and $\mathbf{p}_k^3$ apply to the input and output of the controller, and have to be $\mathcal{F}_{k-1}$-measurable like $\mathbf{u}_k$, meaning that they can depend only on $\mathbf{v}_0, \dots, \mathbf{v}_{k-1}$. On the other hand, $\mathbf{p}_k^1$ and $\mathbf{p}_k^4$ apply on the state before and after $f$ is applied, and need only to be $\mathcal{F}_k$-measurable, meaning they can depend on $\mathbf{v}_k$ as well.

We make the following assumption strengthening the measurability of Assumption 2 to continuity.

**Assumption 10.** The following holds.

(i) $\ell, \sigma$ and $\eta$ are continuous.

(ii) $(x, u) \mapsto f(x, u, v)$ is continuous for any $v \in \mathcal{V}$.

(iii) $\{V_\gamma | \gamma \in (0, 1)\}$ and $\{\mathbb{E}[V_\gamma(f(\cdot, \cdot, \mathbf{v}))] | \gamma \in (0, 1)\}$ are pointwise equi-continuous, i.e., $\forall \varepsilon > 0 \ \forall x \in \mathcal{X} \ \exists \delta > 0 \ \forall y \in \mathcal{X}$,

$$ |x - y| < \delta \Rightarrow \forall \gamma \in (0, 1) : |V_{\gamma}(x) - V_{\gamma}(y)| < \varepsilon, \quad (22) $$

similar for $\{\mathbb{E}[V_\gamma(f(\cdot, \cdot, \mathbf{v}))] | \gamma \in (0, 1)\}$. Furthermore, $W$ is continuous. $\diamond$

All conditions of Assumption 10 are satisfied in the LQR problem of Example 4.

We can now state the robust versions of Lemma 7 and Theorem 8. The difference in the next results is in the existence of a perturbation bound $\varepsilon$ (that generally depends on $\Delta_0, \delta, p$ and $T$) such that desired properties are fulfilled even for $\varepsilon$-perturbed, not just exact, strictly causal random closed-loop solutions. Parameter $\varepsilon$ is independent of $\gamma$ due to the pointwise equi-continuity in Assumption 10.
**Lemma 11.** Let Assumptions 2, 3, 5 and 10 hold. Then,
$$ \forall \Delta_0 > 0 \forall p > 0 \forall T \in \mathbb{Z}_{\ge 0} \exists \Delta > 0 \exists \varepsilon > 0 $$

$$ \forall \gamma \in (0, 1) \forall x_0 \in \sigma^{-1}(\Delta_0 \mathbb{B}) \forall \mathbf{x} \in S_{\gamma, \varepsilon}^{\eta}(x_0) : \quad (23) $$

$$ \mathbb{P}[\sigma(\mathbf{x}_k) < \Delta \quad \forall k \in \{0, \dots, T\}] > 1 - p. \quad \diamond $$

**Theorem 12.** Let Assumptions 2, 3, 5 and 10 hold. Then,
$$ \forall \Delta_0, \delta > 0 \forall p > 0 \exists T \in \mathbb{Z}_{\ge 0} \exists \varepsilon > 0 \exists \gamma^* \in (0, 1) $$

$$ \forall \gamma \in (\gamma^*, 1) \forall x_0 \in \sigma^{-1}(\Delta_0 \mathbb{B}) \forall \mathbf{x} \in S_{\gamma, \varepsilon}^{\eta}(x_0) : \quad (24) $$

$$ \mathbb{P}[\exists k \in \{0, \dots, T\} : \mathbf{x}_k \in \mathcal{A}_{\gamma, \delta}] > 1 - p. \quad \diamond $$

The proofs of Lemma 11 and Theorem 12 follow a similar structure to those of Lemma 7 and Theorem 8.

## 6. CONCLUSION

For discrete-time stochastic systems controlled by infinite-horizon discounted optimal control inputs, appropriate cost-controllability and detectability assumptions imply semi-global practical recurrence with respect to the discount factor. This property is uniform in the sense that the time to enter the recurrent set with high probability is independent of discount factor and initial state. Under additional continuity assumptions, robustness is guaranteed. Ongoing work is being done to obtain recurrence properties under relaxations of Assumption 3, as well as for the undiscounted case and stochastic model predictive control.

## REFERENCES

* Acikgoz, S.U. and Diwekar, U.M. (2010). Blood glucose regulation with stochastic optimal control for insulin-dependent diabetic patients. *Chemical Engineering Science*, 65(3), 1227-1236.

* Bertsekas, D. (2012). *Dynamic Programming and Optimal Control: Volume I*. Athena Scientific, Belmont, MA, USA, 4th edition.

* Davis, M.H.A. and Vinter, R.B. (1985). *Stochastic Modelling and Control*. Springer Netherlands, Dordrecht.

* Grammatico, S., Subbaraman, A., and Teel, A.R. (2013). Discrete-time stochastic control systems: A continuous Lyapunov function implies robustness to strictly causal perturbations. *Automatica*, 49(10), 2939-2952.

* Granzotto, M., Postoyan, R., Buşoniu, L., Nešić, D., and Daafouz, J. (2021). Finite-horizon discounted optimal control: Stability and performance. *IEEE Transactions on Automatic Control*, 66(2), 550-565.

* Granzotto, M., Postoyan, R., Nešić, D., and Teel, A.R. (2024a). Stability analysis of stochastic optimal control: The linear discounted quadratic case. *IEEE Transactions on Automatic Control*, 1-8.

* Granzotto, M., Silva, O.L.D., Postoyan, R., Nešić, D., and Jiang, Z.P. (2024b). Robust stability and near-optimality for policy iteration: For want of recursive feasibility, all is not lost. *IEEE Transactions on Automatic Control*, 69(12), 8247-8262.

* Grimm, G., Messina, M., Tuna, S., and Teel, A. (2005). Model predictive control: for want of a local control Lyapunov function, all is not lost. *IEEE Transactions on Automatic Control*, 50(5), 546-558.

* Hernández-Lerma, O. and Lasserre, J.B. (1996). *Discrete-Time Markov Control Processes*. Springer, New York, NY, USA.

* Kirk, D.E. (2004). *Optimal Control Theory: An Introduction*. Dover, New York, NY, USA.

* Meyn, S.P. and Tweedie, R.L. (1993). *Markov Chains and Stochastic Stability*. Springer-Verlag, London.

* Moldenhauer, R.H., Nešić, D., Granzotto, M., Postoyan, R., and Teel, A.R. (2025). Robust recurrence of discrete-time infinite-horizon stochastic optimal control with discounted cost. Technical Report: https://arxiv.org/abs/2504.20705.

* Moura, S.J., Fathy, H.K., Callaway, D.S., and Stein, J.L. (2011). A stochastic optimal control approach for power management in plug-in hybrid electric vehicles. *IEEE Transactions on Control Systems Technology*, 19(3), 545-555.

* Postoyan, R., Buşoniu, L., Nešić, D., and Daafouz, J. (2017). Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. *IEEE Transactions on Automatic Control*, 62(6), 2736-2749.

* Rockafellar, R. and Wets, R.J.B. (1998). *Variational Analysis*. Springer, Berlin, Heidelberg, Germany, 3rd edition.

* Soner, H.M. (2004). *Stochastic Optimal Control in Finance*. Scuola normale superiore, Istanbul, Turkey.

* Subbaraman, A. and Teel, A.R. (2013). A converse Lyapunov theorem for strong global recurrence. *Automatica*, 49(10), 2963-2974.

* Sutton, R.S. and Barto, A.G. (2018). *Reinforcement Learning: An Introduction*. MIT Press, Cambridge, MA, US, 2nd edition.

* Todorov, E. (2005). Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system. *Neural Computation*, 17(5), 1084-1108.

* Wang, Z., Wang, X., and Liu, L. (2016). Stochastic optimal linear control of wireless networked control systems with delays and packet losses. *IET Control Theory & Applications*, 10(7), 742-751.