# Discounted LQR: stabilizing (near-)optimal state-feedback laws

Jonathan de Brusse, Jamal Daafouz, Mathieu Granzotto, Romain Postoyan, Dragan Nesic

► To cite this version:

Jonathan de Brusse, Jamal Daafouz, Mathieu Granzotto, Romain Postoyan, Dragan Nesic. Discounted LQR: stabilizing (near-)optimal state-feedback laws. 64th IEEE Conference on Decision and Control, Dec 2025, Rio de Janeiro, Brazil. hal-05328846

HAL Id: hal-05328846

https://hal.science/hal-05328846v1

Submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Discounted LQR: stabilizing (near-)optimal state-feedback laws

Jonathan de Brusse<fnref n="1" />,<fnref n="2" />, Jamal Daafouz<fnref n="1" />,<fnref n="3" />, Mathieu Granzotto<fnref n="2" />, Romain Postoyan<fnref n="1" /> and Dragan Nešić<fnref n="2" />

*Abstract*— We study deterministic, discrete linear time-invariant systems with infinite-horizon discounted quadratic cost. It is well-known that standard stabilizability and detectability properties are not enough in general to conclude stability properties for the system in closed-loop with the optimal controller when the discount factor is small. In this context, we first review some of the stability conditions based on the optimal value function found in the learning and control literature and highlight their conservatism. We then propose novel (necessary and) sufficient conditions, still based on the optimal value function, under which stability of the origin for the optimal closed-loop system is guaranteed. Afterwards, we focus on the scenario where the optimal feedback law is not stabilizing because of the discount factor and the goal is to design an alternative stabilizing near-optimal static state-feedback law. We present both linear matrix inequality-based conditions and a variant of policy iteration to construct such stabilizing near-optimal controllers. The methods are illustrated via numerical examples.

## I. INTRODUCTION

While the stability properties of the undiscounted linear quadratic regulator (LQR) problem are well-understood, see e.g., [1]–[4], the case where the cost function involves a discount factor still presents challenges. Even when a linear time-invariant system is stabilizable and satisfies a standard detectability property with respect to the stage cost, the optimal closed-loop system, i.e., the system in closed-loop with the optimal feedback law, may be unstable because of the discount factor [5, Example 1], see also [6] as well as [7] for stochastic results. It is shown in [5, Corollary 3] that stability is always guaranteed in this context provided that the discount factor $\gamma$ is sufficiently close to 1. When this is not the case, namely when $\gamma$ is too small, a common technique consists in iteratively increasing the discount factor so that it is eventually close or equal to 1, see, e.g., [8]–[11]. Still, there are scenarios where the discount factor should be fixed and not necessarily close to 1 because of the problem at hand, or because having a small discount factor allows working with limited amount of data in a learning setting [12], and yet ensuring the stability of the closed-loop system is essential. Two questions then naturally arise:

1. When is stability guaranteed for the discounted LQR problem for a given value of $\gamma$, possibly small?

2. If the optimal feedback law is not stabilizing, how to construct a near-optimal stabilizing state-feedback law instead?

The objective of this work is to provide answers to both questions for deterministic, discrete linear time-invariant systems.

Question 1) admits a simple answer: it suffices to evaluate the eigenvalues of the optimal closed-loop system state matrix. However, it is important to also have at our disposal stability conditions involving the optimal value function. This is particularly relevant in a learning context where we seek to approximate or learn the optimal value function and use it to certify stability [11]–[13]. We show in this work that the existing conditions are subject to conservatism in the sense that they unnecessarily require $\gamma$ to be close to 1. To overcome these limitations, we present a novel relaxed sufficient condition that captures the one in [12,13] as a special case, thereby reducing conservatism in the range of $\gamma$ values that satisfy it. We then show that a necessary and sufficient condition can be obtained if we add a quadratic term to the optimal value function to obtain a Lyapunov function for the optimal closed-loop system. Similar techniques have been used in other optimal control problems, see, e.g., [5,7,14]. However, in all these references the extra term used to build a Lyapunov function is positive semi-definite, while the extra term that we propose may not be sign definite, which is key to making our condition both necessary and sufficient.

Afterwards, we address question 2). We derive for this purpose convex optimization problems with LMI constraints, which allow constructing a near-optimal stabilizing state-feedback law. In particular, we present sufficient conditions under which a stabilizing state-feedback can be constructed and whose cost is guaranteed to be less than a given constant for a given initial condition, like in e.g., [15, Chapter 10] and more recently in a data-driven context in [16] where similar near-optimal controller designs can be found for undiscounted costs. We further provide LMI conditions, which allow synthesizing a stabilizing state-feedback law that is independent of the considered initial condition this time and with a guaranteed margin on the mismatch with the optimal, but not stabilizing, gain. We finally provide a variant of policy iteration (PI) algorithm that enforces stability at each iteration while ensuring cost improvement. Interestingly, this new algorithm shares similarities with policy gradient methods [17] as we show. All the results are illustrated via several numerical examples for which closed-loop system stability is ensured without significantly degrading performance with respect to the optimal one.

<footnote>*Work funded by the ANR under grant OLYMPIA ANR-23-CE48-0006, the Discovery Project ARC DP250100300, and the IRP CNRS ARS.</footnote>

<footnote n="1">¹ J. de Brusse, J. Daafouz and R. Postoyan are with the Université de Lorraine, CNRS, CRAN, F-54000, Nancy, France. (emails: {jonathan.de-brusse, jamal.daafouz,romain.postoyan}@univ-lorraine.fr).</footnote>

<footnote n="2">² J. de Brusse, D. Nešić and M. Granzotto are with the Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3010, Australia (emails: jonathan.debrusse@student.unimelb.edu.au, {dnesic, mathieu.granzotto}@unimelb.edu.au).</footnote>

<footnote n="3">³ J. Daafouz is also with Institut Universitaire de France (IUF), F-75000, Paris, France.</footnote>
Proofs are omitted for space reasons and can be found in the extended version of this work [18].

## II. NOTATION

Let $\mathbb{C}$ be the set of complex numbers, $\mathbb{R}$ the set of real numbers, $\mathbb{Z}_{\ge 0}$ the set of non-negative integers including 0 and $\mathbb{Z}_{>0}$ the set of strictly positive integers. The symbol $\mathbb{R}^{n \times m}$ stands for the set of real matrices with $n \in \mathbb{Z}_{>0}$ rows and $m \in \mathbb{Z}_{>0}$ columns. For a symmetric matrix $Q \in \mathbb{R}^{n \times n}$, we write $Q \succ 0$ ($Q \succeq 0$) if $Q$ is positive (semi-)definite. The identity matrix is denoted by $\mathbf{1}$ and the zero matrix by $\mathbf{0}$, whose dimensions depend on the context. The symbol $(\bullet)$ in a matrix stands for the symmetric term, i.e., $\begin{bmatrix} A \\ B \\ C \end{bmatrix} = \begin{bmatrix} A & (\bullet)^{\top} \\ B^{\top} & C \end{bmatrix}$. For any $M \in \mathbb{R}^{n \times m}$ with $n, m \in \mathbb{Z}_{>0}$, the Frobenius norm of $M$ is denoted by $\|M\|$. For any square matrix $M \in \mathbb{R}^{n \times n}$ with $n \in \mathbb{Z}_{>0}$, the trace of $M$ is denoted by trace($M$) and the spectral radius of $M$ by $\rho(M)$. For an infinite sequence $\mathbf{u} := (u_0, u_1, \dots)$ where $u_0, u_1, \dots \in \mathbb{R}^m$ with $m \in \mathbb{Z}_{>0}$, $\mathbf{u}|_k$ stands for the truncation of $\mathbf{u}$ to the first $k \in \mathbb{Z}_{>0}$ elements, i.e., $\mathbf{u}|_k := (u_0, \dots, u_{k-1})$ and we use the convention $\mathbf{u}|_0 = \emptyset$ when $k = 0$.

## III. PROBLEM STATEMENT

### A. Discounted LQR

Consider the deterministic, discrete-time linear time-invariant system

$$x_{k+1} = Ax_k + Bu_k, \quad (1)$$

where $x_k \in \mathbb{R}^n$ and $u_k \in \mathbb{R}^m$ are the state and control input at time $k \in \mathbb{Z}_{\ge 0}$, respectively, with $n, m \in \mathbb{Z}_{>0}$. Matrices $A$ and $B$ are real, of appropriate dimensions and such that the next assumption holds.

**Standing Assumption 1 (SA1):** The pair $(A, B)$ is stabilizable.

Necessary and sufficient conditions under which SA1 holds are provided in [19, Lecture 14]. The discounted (LQR) problem consists of determining, for any given initial state $x_0 \in \mathbb{R}^n$, an infinite length sequence of inputs $\mathbf{u} = (u_k)_{k \in \mathbb{Z}_{\ge 0}} \in (\mathbb{R}^m)^{\mathbb{Z}_{\ge 0}}$ that minimizes the cost function

$$J_{\gamma}(x_0, \mathbf{u}) = \sum_{k=0}^{\infty} \gamma^k [\phi(k, x_0, \mathbf{u}|_k)^{\top} Q \phi(k, x_0, \mathbf{u}|_k) + u_k^{\top} R u_k] \quad (2)$$

where the discount factor $\gamma \in [0, 1]$ is fixed and $\phi(k, x_0, \mathbf{u}|_k)$ denotes the state obtained at time $k \in \mathbb{Z}_{\ge 0}$ by applying the sequence of inputs $\mathbf{u}|_k$ from the initial state $x_0$ to system (1). We use the convention $\phi(0, x_0, \mathbf{u}|_0) = x_0$. We assume that $Q$ and $R$ satisfy the next assumption.

**Standing Assumption 2 (SA2):** There exist real matrices $C$ and $D$ such that $Q = C^{\top}C \succ 0$ and $R = D^{\top}D \succ 0$. $\square$

SA2 implies that $(A, C)$ is observable. The optimal controller, also called optimal policy, which minimizes (2) is given by the state feedback law $h_{\gamma}^{*}(x) = K_{\gamma}x$ for any $x \in \mathbb{R}^n$ (see e.g., [2, Section 4.2]), where

$$K_{\gamma} := -\gamma(R + \gamma B^{\top} P_{\gamma} B)^{-1} B^{\top} P_{\gamma} A \quad (3)$$

and $P_{\gamma} = P_{\gamma}^{\top} \succ 0$ is the solution to the discrete-time algebraic Riccati equation

$$P_{\gamma} = \gamma A^{\top} P_{\gamma} A - \gamma^2 A^{\top} P_{\gamma} B (R + \gamma B^{\top} P_{\gamma} B)^{-1} B^{\top} P_{\gamma} A + Q. \quad (4)$$

The optimal cost at a given initial state $x_0 \in \mathbb{R}^n$ is then given by

$$V_{\gamma}^{*}(x_0) := \min_{\mathbf{u}} J_{\gamma}(x_0, \mathbf{u}) = J_{\gamma}(x_0, K_{\gamma}) = x_0^{\top} P_{\gamma} x_0, \quad (5)$$

where $J_{\gamma}(x_0, K_{\gamma})$ represents the cost induced by the repeated application of the optimal policy $h_{\gamma}^{*}(x) = K_{\gamma}x$ at each encountered state to (1). As shown in [3, Theorem 8], the stabilizability of the pair $(\sqrt{\gamma}A, \sqrt{\gamma}B)$, which follows from SA1, and the detectability of the pair $(\sqrt{\gamma}A, C)$, which follows from SA2, are necessary and sufficient conditions for the Riccati equation (4) to have a unique symmetric, positive definite solution. Moreover, as $Q \succ 0$ by SA2 and $V_{\gamma}^{*}(x) = J_{\gamma}(x, K_{\gamma}) \ge \ell(x, K_{\gamma}x) = x^{\top}(Q + K_{\gamma}^{\top} R K_{\gamma})x$ for any $x \in \mathbb{R}^n$, we derive that

$$P_{\gamma} \succeq Q \succ 0. \quad (6)$$

This property will be invoked in the sequel.

In the following, we recall that SA1 and SA2 alone do not always guarantee the stability of the optimal closed-loop system, i.e., system (1) with input $u = K_{\gamma}x$.

### B. Challenges with stability

The next example illustrates that SA1 and SA2 are not sufficient to ensure the stability of the optimal closed-loop system for all $\gamma \in [0, 1]$.

**Example 1:** We consider $A = \begin{bmatrix} -0.97 & 0 \\ 3.88 & -0.97 \end{bmatrix}$, $B = \begin{bmatrix} 2 \\ -1 \end{bmatrix}$, $Q = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix}$ and $R = 5$. We take $C = \begin{bmatrix} \sqrt{2} & 0 \\ 0 & \sqrt{3} \end{bmatrix}$ and $D = \sqrt{5}$ so that SA1 and SA2 hold. The evolution of the spectral radius of the matrix $A+BK_{\gamma}$ as a function of $\gamma$ is depicted in Fig. 1. As we can observe, stability is guaranteed for $\gamma \in [0, 0.02]$, before being lost by increasing $\gamma$ in the interval $[0.02, 0.12]$ after which it is recovered for $\gamma \in (0.12, 1]$. $\square$

<figure><img src="image_2.png" /><figcaption>Fig. 1. Spectral radius of $A+BK_{\gamma}$ over the interval $[0, 1]$, with a zoom on $\gamma \in [0, 0.2]$ for Example 1.</figcaption></figure>

To clarify this phenomenon, we first introduce some relevant notations. Given any $\gamma \in [0, 1]$, $K_{\gamma}$ denotes the set of all state feedback control gains $K \in \mathbb{R}^{m \times n}$ such that the matrix $\sqrt{\gamma}(A+BK)$ is Schur, i.e.,

$$K_{\gamma} := \{K \in \mathbb{R}^{m \times n} : \rho(\sqrt{\gamma}(A+BK)) < 1\}. \quad (7)$$

It is important to notice that $\sqrt{\gamma}(A+BK_{\gamma})$ is not the state matrix of the optimal closed-loop system, whenever $\gamma \in [0, 1]$.
We have that $\mathcal{K}_1 \subset \mathcal{K}_\gamma$ for any $\gamma \in [0, 1]$ since $\rho(\sqrt{\gamma}(A+ BK)) \le \rho(A+ BK)$ for any $K \in \mathbb{R}^{m \times n}$. However, $\mathcal{K}_\gamma \not\subset \mathcal{K}_1$, as illustrated in Example 1, where for $\gamma \in [0.02, 0.12]$ the gain $K_\gamma$ belongs to $\mathcal{K}_\gamma$ but not to $\mathcal{K}_1$. Indeed, the optimal control gain $K_\gamma \in \mathcal{K}_\gamma$ and $P_\gamma$ are solutions to the Lyapunov equation

$$ (\sqrt{\gamma}(A+ BK_\gamma))^\top P_\gamma (\sqrt{\gamma}(A+ BK_\gamma)) - P_\gamma + K_\gamma^\top R K_\gamma + Q = 0. \quad (8) $$

Thus, the optimal value function $V_\gamma^*$ in (5) is a Lyapunov function for the system

$$ x_{k+1} = \sqrt{\gamma}(A + BK_\gamma)x_k. \quad (9) $$

However, there is a priori no reason for $V_\gamma^*$ to also be a Lyapunov function for the optimal closed-loop system, i.e., system (1) with $u_k = K_\gamma x_k$, which corresponds to

$$ x_{k+1} = (A + BK_\gamma)x_k. \quad (10) $$

In other words, although $K_\gamma \in \mathcal{K}_\gamma$, there is no guarantee that $K_\gamma \in \mathcal{K}_1$, which means that the optimal feedback law may not stabilize system (1) as recalled in Example 1.

### C. Objectives

The first objective of this work is to develop conditions based on the optimal value function that allows assessing the stability of the optimal closed-loop system (10). This is the purpose of Section IV.

The second objective is to design near-optimal state-feedback laws that ensure the stability of system (1) when the optimal controller $u = K_\gamma x$ is not stabilizing for system (1). This is addressed in Section V.

## IV. STABILITY ANALYSIS BASED ON THE OPTIMAL VALUE FUNCTION

We begin by revisiting a condition previously proposed in the literature, see, e.g., [12,13], which is stated as follows

$$ Q + (\gamma - 1)P_{\gamma} \succ 0. \quad (11) $$

This condition guarantees that the optimal value function $V_\gamma^*$ is a common Lyapunov function, in the sense that it is a Lyapunov function for both systems (9) and (10), thereby ensuring that $K_\gamma \in \mathcal{K}_1$. Indeed, note that equation (8) can be written as

$$ (A+ BK_\gamma)^\top \gamma P_\gamma (A+ BK_\gamma) - \gamma P_\gamma + K_\gamma^\top R K_\gamma + Q + (\gamma-1)P_\gamma = 0. \quad (12) $$

Condition (11) ensures that $(A + BK_\gamma)^\top \gamma P_\gamma (A + BK_\gamma) - \gamma P_\gamma$ is negative definite, thereby making $V_\gamma^*$ a common Lyapunov function.

We argue that condition (12) is subject to some conservation. Indeed, we know that $P_\gamma \succeq Q$ by (6) and condition (11) is equivalent to $P_\gamma \prec \frac{1}{1-\gamma}Q$. Thus, $P_\gamma$ is constrained within bounds determined by $Q$, which makes (11) restrictive in general. The next theorem provides an alternative, relaxed condition, which does not rely on a Lyapunov construction.

**Theorem 1:** Given any fixed $\gamma \in [0, 1]$, consider the optimal gain $K_\gamma$ in (3) and the solution $P_\gamma$ to (4). Then, $K_\gamma \in \mathcal{K}_1$ if the following holds

$$ \gamma^2 P_\gamma B R^{-1} B^\top P_\gamma + Q + (\gamma - 1)P_\gamma \succ 0. \quad (13) $$

Theorem 1 provides a less conservative condition to ensure that $K_\gamma \in \mathcal{K}_1$ compared to (11) as it covers it as a special case. To see it, note that $\gamma^2 P_\gamma B R^{-1} B^\top P_\gamma \succeq 0$ for any $\gamma \in [0, 1]$, which implies that $\gamma^2 P_\gamma B R^{-1} B^\top P_\gamma + Q + (\gamma - 1)P_\gamma \succeq Q + (\gamma - 1)P_\gamma$. Therefore, the range of $\gamma$ values that satisfy (13) always includes the range of $\gamma$ values satisfying (11).

Another sufficient stability condition found in the literature [11, (7)] is given by

$$ K_{\gamma}^{\top} R K_{\gamma} + Q + (\gamma - 1) P_{\gamma} \succ 0. \quad (14) $$

Condition (14) also guarantees that $V_\gamma^*$ is a common Lyapunov function. However, it is unclear whether the satisfaction of (14) ensures that (13) holds. We show below that (14) imposes a much smaller range of $\gamma$ than (13) for Example 1. In any case, all the stability conditions presented so far are sufficient. To obtain necessary and sufficient conditions, the next theorem establishes that the optimal value function $V_\gamma^*$ in (5) needs to be modified to become a Lyapunov function by adding a suitable quadratic term.

**Theorem 2:** Given any fixed $\gamma \in [0, 1]$, consider the optimal gain $K_\gamma$ in (3) and the solution $P_\gamma$ to (4). Then, $K_\gamma \in \mathcal{K}_1$ if and only if there exists a symmetric matrix $X \in \mathbb{R}^{n \times n}$ such that $\gamma P_\gamma + X \succ 0$ and

$$ K_{\gamma}^{\top} R K_{\gamma} + Q + (\gamma - 1) P_{\gamma} + X - (A + BK_{\gamma})^{\top} X (A + BK_{\gamma}) \succ 0. \quad (15) $$

Theorem 2 provides a necessary and sufficient condition to ensure the stability of the optimal closed-loop system (10) for a given value of $\gamma$. Note that the requirement that $\gamma P_\gamma + X \succ 0$ does not necessarily imply that $X \succeq 0$.

To conclude this section, we revisit Example 1 in the light of the results presented so far.

**Example 1 revisited:** The condition (14) is satisfied for $\gamma \in [0.97, 1]$, while (13) is feasible for $\gamma \in [0.30, 1]$. On the other hand, the necessary and sufficient condition (15) yields that $\gamma$ needs to be in $[0, 0.02) \cup (0.12, 1]$ for system (10) to be stable, which is consistent with Fig. 1. $\square$

When $\gamma \in [0.02, 0.12]$, the optimal gain $K_\gamma$ is not stabilizing in Example 1. In this case, we may then be interested in designing an alternative, stabilizing gain whose cost is close to $V_\gamma^*$ in (5): this is the purpose of the next section.

## V. STABILIZING NEAR-OPTIMAL STATE-FEEDBACK LAWS

We are motivated in this section by the scenario where $K_\gamma \notin \mathcal{K}_1$. The objective is to design another gain, either denoted $\hat{K}_\gamma$ or $\bar{K}_\gamma$ as we will see, such that $A + B\hat{K}_\gamma$ (or $A + B\bar{K}_\gamma$) is Schur and that is near-optimal. By near-optimal, we either mean that:

(i) the induced cost $J_\gamma(x_0, \hat{K}_\gamma)$ is less than a given constant that is minimized for a given initial condition (Section V-A);

(ii) $\bar{K}_\gamma$ is as close as possible to the optimal gain $K_\gamma$ (Section V-B).
We finally introduce a variant of PI that generates a stabilizing gain while improving the induced cost of the previous controller at each iteration (Section V-C).

### A. LMI-based design with guaranteed cost

Without loss of generality<fnref n="1" />, we consider that SA2 holds with $C^\top D = 0$. The closed-loop form of (4), expressed as (8), can be written as

$$ (\sqrt{\gamma}(A+BK_{\gamma}))^{\top}P_{\gamma}(\sqrt{\gamma}(A+BK_{\gamma})) - P_{\gamma} + \mathcal{L}_{K_{\gamma}}^{\top}\mathcal{L}_{K_{\gamma}} = 0, \quad (16) $$

where $\mathcal{L}_{K_{\gamma}} := C + DK_{\gamma}$. The next theorem presents a method for synthesizing a near-optimal stabilizing gain $\hat{K}_{\gamma}$, whose associated cost is less than a known constant for a given initial condition, such as in [15, Chapter 10] and [16] where undiscounted costs are considered.

**Theorem 3:** Given any fixed $\gamma \in [0, 1]$, $\mu \in \mathbb{R}_{>0}$, initial condition $x_0 \in \mathbb{R}^n$, consider cost (2) with $C^\top D = 0$. Suppose there exist $X_\gamma \in \mathbb{R}^{n \times n}$, $Z_\gamma \in \mathbb{R}^{n \times n}$ symmetric and $G_\gamma \in \mathbb{R}^{n \times n}$, $Y_\gamma \in \mathbb{R}^{m \times n}$ such that

$$ \begin{bmatrix} \mu & (\bullet)^{\top} \\ x_0^{\top} & X_{\gamma} \end{bmatrix} \geq 0 \quad (17a) $$

$$ \begin{bmatrix} G_{\gamma} + G_{\gamma}^{\top} - X_{\gamma} & (\bullet)^{\top} & (\bullet)^{\top} \\ \sqrt{\gamma}AG_{\gamma} + \sqrt{\gamma}BY_{\gamma} & X_{\gamma} & (\bullet)^{\top} \\ CG_{\gamma} + DY_{\gamma} & 0 & 1 \end{bmatrix} \geq 0 \quad (17b) $$

$$ \begin{bmatrix} G_{\gamma} + G_{\gamma}^{\top} - Z_{\gamma} & (\bullet)^{\top} \\ AG_{\gamma} + BY_{\gamma} & Z_{\gamma} \end{bmatrix} \succ 0. \quad (17c) $$

Then $\hat{K}_{\gamma} = Y_{\gamma}G_{\gamma}^{-1} \in \mathcal{K}_1$ and $J_{\gamma}(x_0, \hat{K}_{\gamma}) \le x_0^{\top}X_{\gamma}^{-1}x_0 \le \mu$. Moreover, if (17) is feasible for some $\gamma$, then it is feasible for all $\gamma' \in [0, \gamma]$. □

Theorem 3 allows, when (17) is feasible, to construct a near-optimal, stabilizing gain $\hat{K}_{\gamma}$ for system (1) where, by near-optimal, we mean that, for a given initial condition $x_0 \in \mathbb{R}^n$,

$$ x_0^{\top} P_{\gamma} x_0 \le J_{\gamma}(x_0, \hat{K}_{\gamma}) \le x_0^{\top} X_{\gamma}^{-1} x_0 \le \mu. \quad (18) $$

To minimize the cost induced by $\hat{K}_{\gamma}$ with optimization solvers, one minimizes $\mu$ subject to constraint (17), which yields to the convex optimization problem

$$ X_{\gamma}, Y_{\gamma}, Z_{\gamma}, G_{\gamma}, \mu \stackrel{\min}{=} \mu \quad (19) $$

s.t. (17) holds.

We illustrate the outcomes of (19) on Example 1.

**Example 1 revisited:** Take the initial state $x_0 = [1 \ 1]^\top$ and the matrices $C = \begin{bmatrix} \sqrt{2} & 0 \\ 0 & \sqrt{3} \\ 0 & 0 \end{bmatrix}$, $D = \begin{bmatrix} 0 \\ \sqrt{5} \end{bmatrix}$ such that $C^\top D = 0$. As desired $\hat{K}_{\gamma} \in \mathcal{K}_1$ for all $\gamma \in [0, 1]$. The evolution of the cost induced by the gain $\hat{K}_{\gamma}$ solution of (19) as a function of $\gamma$ is shown in Fig. 2. As expected from (18), the cost $J_{\gamma}(x_0, \hat{K}_{\gamma})$ (green plot) is always upper-bounded by $x_0^{\top}X_{\gamma}^{-1}x_0$ (blue dashed line), and lower-bounded by the optimal cost $x_0^{\top}P_{\gamma}x_0$ (red dashed line). The upper-bound prevents $J_{\gamma}(x_0, \hat{K}_{\gamma})$ from becoming excessively large, ensuring a specified level of performance. □

<figure><img src="image_3.png" /><figcaption>Fig. 2. Cost $J_{\gamma}(x_0, \hat{K}_{\gamma})$ as a function of $\gamma$ with $x_0 = [1 \ 1]^\top$.</figcaption></figure>

### B. LMI-based design with bounds on controller gains mismatch

The design method presented in Section V-A ensures that the induced cost is less than a constant (that can be minimized using (19)) for a given initial condition. In this section, we instead aim at designing a stabilizing gain, which we denote $\bar{K}_{\gamma}$, that minimizes the mismatch with the optimal gain $K_{\gamma}$. This problem can be formulated as follow

$$ \min_{\bar{K}_{\gamma} \in \mathcal{K}_1} \nu_{\gamma} \quad (20) $$

$$ \text{s.t.} \quad \|\bar{K}_{\gamma} - K_{\gamma}\| \le \nu_{\gamma}. $$

This is equivalent to

$$ \begin{array}{ll} \min_{\bar{K}_{\gamma}, \bar{P}_{\gamma}, \bar{W}_{\gamma}} & \operatorname{trace}(\bar{W}_{\gamma}) \\ \text{s.t.} & (\bar{K}_{\gamma} - K_{\gamma})^{\top}(\bar{K}_{\gamma} - K_{\gamma}) \preceq \bar{W}_{\gamma}, \quad \bar{P}_{\gamma} \succ 0, \quad (21) \\ & (A + B\bar{K}_{\gamma})^{\top}\bar{P}_{\gamma}(A + B\bar{K}_{\gamma}) - \bar{P}_{\gamma} \prec 0. \end{array} $$

The constraints in (21) are nonlinear, rendering this optimization problem non-convex. The next theorem gives sufficient LMI conditions to ensure the satisfaction of the constraints in (21).

**Theorem 4:** Given any fixed $\gamma \in [0, 1]$ and consider the cost (2) with $C^\top D = 0$. Suppose there exist $\bar{L}_{\gamma} \in \mathbb{R}^{n \times n}$, $\bar{Z}_{\gamma} \in \mathbb{R}^{n \times n}$ symmetric and $\bar{S}_{\gamma} \in \mathbb{R}^{n \times n}$, $\bar{T}_{\gamma} \in \mathbb{R}^{m \times n}$ such that

$$ \begin{bmatrix} \bar{S}_{\gamma} + \bar{S}_{\gamma}^{\top} - \bar{L}_{\gamma} & (\bullet)^{\top} \\ \bar{T}_{\gamma} - K_{\gamma}\bar{S}_{\gamma} & 1 \end{bmatrix} \geq 0 \quad (22a) $$

$$ \begin{bmatrix} \bar{S}_{\gamma} + \bar{S}_{\gamma}^{\top} - \bar{Z}_{\gamma} & (\bullet)^{\top} \\ A\bar{S}_{\gamma} + B\bar{T}_{\gamma} & \bar{Z}_{\gamma} \end{bmatrix} \succ 0. \quad (22b) $$

Then $\bar{K}_{\gamma} = \bar{T}_{\gamma}\bar{S}_{\gamma}^{-1} \in \mathcal{K}_1$ and $(\bar{K}_{\gamma} - K_{\gamma})^{\top}(\bar{K}_{\gamma} - K_{\gamma}) \preceq \bar{L}_{\gamma}^{-1}$. □

Theorem 4 can be used to design a stabilizing gain $\bar{K}_{\gamma}$ with a guaranteed upper-bound on $(\bar{K}_{\gamma} - K_{\gamma})^{\top}(\bar{K}_{\gamma} - K_{\gamma})$ by maximizing the trace of $\bar{L}_{\gamma}$. This is achieved by solving the next convex optimization problem

$$ \max_{\bar{S}_{\gamma}, \bar{T}_{\gamma}, \bar{L}_{\gamma}, \bar{Z}_{\gamma}} \operatorname{trace}(\bar{L}_{\gamma}) \quad (23) $$

s.t. (22) holds.

**Example 1 revisited:** Fig. 3 compares the cost obtained with $\hat{K}_{\gamma}$ and $\bar{K}_{\gamma}$, solutions to (19) and (23) respectively, for $\gamma \in [0, 1]$ and $x_0 = [1 \ 1]^\top$. We introduce for this purpose

<footnote n="1">¹If an initial choice of C and D satisfying SA2 does not yield $C^\top D = 0$, we can always redefine them as $C_2 = [\frac{C}{D}]$ and $D_2 = [\frac{D}{C}]$ ensuring $C_2C_2^\top = Q$, $D_2D_2^\top = R$, and $C_2^\top D_2 = 0$.</footnote>
the relative error of the cost induced by a gain $K \in \mathbb{R}^{m \times n}$ at an initial state $x_0 \in \mathbb{R}^n \setminus \{0\}$, defined as

$$ \varepsilon_{\gamma}(x_0, K) := \frac{J_{\gamma}(x_0, K) - J_{\gamma}(x_0, K_{\gamma})}{J_{\gamma}(x_0, K_{\gamma})}, \quad (24) $$

which is used to compare the cost induced by gain K, namely $J_{\gamma}(x_0, K)$, with the optimal cost $J_{\gamma}(x_0, K_{\gamma})$. We exclude $x_0 = 0$ in the definition of $\varepsilon_{\gamma}$ as $J_{\gamma}(x_0, K_{\gamma}) = 0$ in this case, which makes it ill-defined. Hence, Fig. 3 presents the evolution of $\varepsilon_{\gamma}(x_0, \hat{K}_{\gamma})$ and $\varepsilon_{\gamma}(x_0, \bar{K}_{\gamma})$ as a function of $\gamma$. The results show that a stabilizing control gain can always be found by solving (23), with performance close to the optimal one. In this example, the gain $\bar{K}_{\gamma}$ outperforms $\hat{K}_{\gamma}$, achieving a relative error below 0.1% for any $\gamma \in [0, 1]$, whereas the cost obtained with $\hat{K}_{\gamma}$ exhibits a relative error of up to 30%. This illustrates that, for this example, Theorem 4 can be used to construct stabilizing state-feedback laws without significantly degrading the optimal cost. $\square$

<figure><img src="image_4.png" /><figcaption>Fig. 3. $\varepsilon_{\gamma}(x_0, \hat{K}_{\gamma})$ and $\varepsilon_{\gamma}(x_0, \bar{K}_{\gamma})$ as functions of $\gamma$ for $x_0 = [1 \ 1]^T$.</figcaption></figure>

To conclude this section, we finally present a novel variant of PI, which allows constructing a sequence of control gains in $K_1$, with iteratively improved costs. Interestingly, this algorithm can be combined with the results presented so far in this section as we explain.

### C. Near-optimal, recursively stabilizing PI

The new algorithm called sPI is presented in Algorithm 1. Given $\gamma \in [0, 1]$ and an initial control gain $K_0 \in K_1$, sPI starts with the same initial evaluation step as in standard PI to generate $P_0$ in (sPI.1). The differences with PI appear at the next iterations. At any iteration $j \in \mathbb{Z}_{>0}$, instead of solely aiming at improving the previous cost, sPI performs a linear combination between the controller gain that a standard PI improvement step would give and the previous controller gain, which is guaranteed to be in $K_1$. We use parameter $\alpha_j$ to define this linear combination, whose value is selected to ensure that the newly obtained controller belongs to $K_1$. After this update of the gain, we find the classical evaluation step in (sPI.3) of PI to generate matrix $P_j$. We further elaborate on the link between sPI and PI as well as other related algorithms in Remark 1 below.

The next theorem ensures the recursive feasibility<fnref n="2" /> of sPI in the sense that we can always find $\alpha_j$ and $K_j$ at

### Algorithm 1 sPI

Input: $K_0 \in K_1$, $\gamma \in [0, 1]$, A, B in (1) and Q, R in (2)

Output: $K_{\infty}, P_{\infty}$

1. **Initial evaluation step:**
$$ \sqrt{\gamma(A + BK_0)P_0}\sqrt{\gamma(A + BK_0)} - P_0 + K_0^\top R K_0 + Q = 0 \quad (\text{sPI.1}) $$
2. **for** $j \in \mathbb{Z}_{>0}$ **do**

3. **Improvement step:**
Select $\alpha_j \in (0, 1]$ such that $K_j \in K_1$ with
$$ K_j = \alpha_j(-\gamma(R + \gamma B^\top P_{j-1} B)^{-1} B^\top P_{j-1} A) + (1 - \alpha_j)K_{j-1} \quad (\text{sPI.2}) $$
4. **Evaluation step:**
$$ \sqrt{\gamma(A + BK_j)^\top P_j}\sqrt{\gamma(A + BK_j)} - P_j + K_j^\top R K_j + Q = 0 \quad (\text{sPI.3}) $$

5. **end for**

6. **return** $K_{\infty} \in K_1$ and $P_{\infty}$.

the improvement step at any iteration. The theorem also recovers one of the main properties of PI: the obtained cost is improved at any iteration. Obviously, we cannot guarantee that the sequence of $P_j$ converges to $P_{\gamma}$ as $K_{\gamma}$ does not a priori belong to $K_1$, while the generated sequence of $K_j$ do.

**Theorem 5:** Given $\gamma \in [0, 1]$ and $K_0 \in K_1$, Algorithm 1 verifies the following properties.

(i) (Recursive feasibility) For any $j \in \mathbb{Z}_{>0}$, there always exist $\alpha_j \in (0, 1]$ and $K_j \in K_1$ verifying (sPI.2) and (sPI.3).

(ii) (Recursive cost improvement) For any $j \in \mathbb{Z}_{>0}$, we have $P_j \le P_{j-1}$, implying that the cost difference satisfies $J_{\gamma}(x_0, K_j) - J_{\gamma}(x_0, K_{j-1}) \le 0$ for any $x_0 \in \mathbb{R}^n$. $\square$

Theorem 5 guarantees that the controller gain obtained at each iteration stabilizes (1), and that its cost cannot be bigger than the cost of the gain obtained at the previous iteration. A relevant way to initialize sPI is to take $K_0 \in \{\hat{K}_\gamma, \bar{K}_\gamma\}$ with $\hat{K}_\gamma, \bar{K}_\gamma$ solution of (19) and (23), respectively. In this way, sPI can only provide “better” controller gains belonging to $K_1$.

**Remark 1:** At any iteration $j \in \mathbb{Z}_{\ge 0}$ of sPI, when we can take $\alpha_j = 1$, we recover the standard iteration of the original PI [21]. Second, sPI shares similarities with policy gradient methods. Indeed, by adapting [17, (7)] to the discounted setting, (sPI.2) can be interpreted as an update of a Gauss-Newton method

$$ K_{j+1} = (1 - 2\eta)K_j + 2\eta(-\gamma(R + \gamma B^\top RB)^{-1}B^\top P_j A) $$

with $2\eta = \alpha_j$. However, in sPI: (i) the weights $\alpha_j$ are allowed to change at any iteration instead of being fixed; (ii) we impose $K_j$ to belong to $K_1$ at each iteration. Our approach also differs from [11], which relies on (14) and incorporates an updating mechanism for the discount factor to ensure that the gradient descent process yields a stabilizing policy, while sPI works for a fixed discount factor. $\square$

**Remark 2:** A relevant way to implement $\alpha_j$ is to maximize its value at each iteration. Then, a possible stopping criterion for the algorithm when $K_\gamma \notin K_1$ is, given a threshold $\epsilon > 0$, to stop iterating when there exists $j \in \mathbb{Z}_{>0}$

<footnote n="2">²See [20] for further discussions on the recursive feasibility of the original PI algorithm.</footnote>
such that $\alpha_j < \epsilon$. $\square$

**Example 1 revisited:** Let $\gamma = 0.1$, we have that $K_\gamma \notin K_1$, see Example 1. We begin with $K_0 := \hat{K}_\gamma \in K_1$, where $\hat{K}_\gamma$ is the solution to (19) with $x_0 = [1 \ 1]^\top$. At any iteration $j \in \mathbb{Z}_{>0}$, we discretize the interval $[0, 1]$ using a grid of length $10^{-5}$ and we denote by $\bar{\alpha}_j$ the maximal value $\alpha_j$ such that the improvement step in Algorithm 1 is feasible. To illustrate sPI over a large number of iterations, we select $\alpha_j = \frac{\bar{\alpha}_j}{10}$ at each iteration. Since $K_\gamma$ is not stabilizing for this value of $\gamma$, we set the stopping criterion to $\alpha_j < 10^{-5}$, see Remark 2. The evolution of the eigenvalues of $A+BK_j$ over the iterations $j$ is presented in Fig. 4 illustrating the fact that they lie inside the unit circle at any iteration as desired.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Unit disk (dashed red curve) and eigenvalues of $A+BK_j$ in the complex plane obtained with sPI (green crosses), with a zoomed-in view near the region where the real parts are close to 1, with $\gamma = 0.1$ and $K_0 = \hat{K}_\gamma = [-0.0081 \ -0.1409]$.</figcaption></figure>

For any $x_0 \in \mathbb{R}^n$ and any iteration $j \in \mathbb{Z}_{\ge 0}$, the cost induced by $K_j$ is given by $x_0^\top P_j x_0$. Hence, the closer $P_j$ is to $P_\gamma$ as in (5), the closer the cost induced by $K_j$ is to the one induced by $K_\gamma$. Fig. 5 shows $\|P_j - P_\gamma\|$ over the iterations $j$. We observe that $\|P_j - P_\gamma\|$ decreases with $j$ consistently with item (ii) of Theorem 5. Although $\|P_j - P_\gamma\|$ approaches zero after 40 iterations, the fact that $K_\gamma \notin K_1$, implies that a zero error cannot be asymptotically reached. To illustrate this, we provide a zoomed-in view for $j \ge 100$, revealing a residual error on the order of $10^{-3}$.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Norm of $P_j - P_\gamma$ as a function of the number of iterations $j$ for $\gamma = 0.1$ and $K_0 = [-0.0081 \ -0.1409]$, with a zoomed-in view for $j > 100$.</figcaption></figure>

## VI. CONCLUSION

We have presented novel stability (necessary and) sufficient conditions involving the optimal value function for the

discounted LQR. Moreover, when the optimal controller fails to stabilize the system, we have given methods to design near-optimal stabilizing state feedback laws both by solving convex optimization problems with LMI-based constraints and with a variant of policy iteration. The considered numerical examples show that the proposed approach may allow designing stabilizing controller without significantly degrading the optimal performance.

## REFERENCES

[1] B. Anderson and J. Moore, *Optimal Control: Linear Quadratic Methods*. Dover edition, Mineola, U.S.A., 2007.

[2] D. P. Bertsekas, *Dynamic Programming and Optimal Control*, 4th ed. Athena Scientific, Belmont, U.S.A., 2012, vol. 2.

[3] V. Kucera, "The discrete Riccati equation of optimal control," *Kybernetika*, vol. 8, no. 5, pp. 430-447, 1972.

[4] F. Lewis, D. Vrabie, and V. Syrmos, *Optimal Control*, 3rd ed. John Wiley & Sons, New-Jersey, U.S.A, 2012.

[5] R. Postoyan, L. Busoniu, D. Nešić, and J. Daafouz, "Stability analysis of discrete-time infinite-horizon optimal control with discounted cost," *IEEE Transactions on Automatic Control*, vol. 62, no. 6, pp. 2736-2749, 2017.

[6] V. Gaitsgory, L. Grüne, M. Höger, C. M. Kellett, and S. R. Weller, "Stabilization of strictly dissipative discrete time systems with discounted optimal control," *Automatica*, vol. 93, pp. 311-320, 2018.

[7] M. Granzotto, R. Postoyan, D. Nešić, and A. R. Teel, "Stability analysis of stochastic optimal control: The linear discounted quadratic case," *IEEE Transactions on Automatic Control*, 2024.

[8] J. C. Perdomo, J. Umenberger, and M. Simchowitz, "Stabilizing dynamical systems via policy gradient methods," in *Advances in Neural Information Processing Systems*, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021.

[9] A. Agazzi and C. X. Chen, "Policy gradient convergence in discounted LQR," 2020. [Online]. Available: https://apisemanticscholar.org/ CorpusID:227106893

[10] A. Lamperski, "Computing stabilizing linear controllers via policy iteration," in *IEEE Conference on Decision and Control, Jeju Island: South Korea*, 2020, pp. 1902-1907.

[11] F. Zhao, X. Fu, and K. You, "Learning stabilizing controllers of linear systems via discount policy gradient," 2021. [Online]. Available: https://arxiv.org/abs/2112.09294

[12] T. Westenbroek, F. Castaneda, A. Agrawal, S. Sastry, and K. Sreenath, "Lyapunov design for robust and efficient robotic reinforcement learning," *Conference on Robot Learning, Auckland, New Zealand*, 2022.

[13] J. Lai, J. Xiong, and Z. Shu, "Model-free optimal control of discrete-time systems with additive and multiplicative noises," *Automatica*, vol. 147, p. 110685, 2023.

[14] G. Grimm, M. Messina, S. Tuna, and A. Teel, "Model predictive control: for want of a local control Lyapunov function, all is not lost," *IEEE Trans. on Automatic Control*, vol. 50, no. 5, pp. 546-558, 2005.

[15] S. Boyd, L. E. Ghaoui, E. Feron, and V. Balakrishnan, "Linear matrix inequalities in system and control theory," *Volume 15 of Studies in Applied Mathematics*, SIAM, 1994.

[16] H. J. van Waarde and M. Mesbahi, "Data-driven parameterizations of suboptimal LQR and H2 controller," *IFAC-PapersOnLine*, vol. 53, no. 2, pp. 4234-4239, 2020.

[17] M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, "Global convergence of policy gradient methods for the linear quadratic regulator," in *International Conference on Machine Learning, Stockholm, Sweden*, 2018, pp. 1467-1476.

[18] J. de Brusse, J. Daafouz, M. Granzotto, R. Postoyan, and D. Nešić, "Discounted lqr: stabilizing (near-)optimal state-feedback laws," *arXiv preprint arXiv:2508.19599*, 2025.

[19] J. P. Hespanha, *Linear Systems Theory*. Princeton University Press, Princeton, U.S.A., 2009.

[20] M. Granzotto, O. L. De Silva, R. Postoyan, D. Nešić, and Z.-P. Jiang, "Robust stability and near-optimality for policy iteration: For want of recursive feasibility, all is not lost," *IEEE Transactions on Automatic Control*, vol. 69, no. 12, pp. 8247-8262, 2024.

[21] G. A. Hewer, "An iterative technique for the computation of steady state gains for the discrete optimal regulator," *IEEE Transactions on Automatic Control*, vol. 16, no. 4, pp. 382-384, 1971.