<figure><img src="image_1.png" /></figure>

HAL
open science

# Model-Free Optimal Static Output Feedback Control using Integral Reinforcement Learning

Eslam Mahmoud, Saïd Mammar, Mohand Smaïli

► To cite this version:

Eslam Mahmoud, Saïd Mammar, Mohand Smaïli. Model-Free Optimal Static Output Feedback Control using Integral Reinforcement Learning. 33rd Mediterranean Conference on Control and Automation (MED 2025), Jun 2025, Tangier, Morocco. 10.1109/MED64031.2025.11073290. hal-05097218

HAL Id: hal-05097218

https://univ-evry.hal.science/hal-05097218v1

Submitted on 4 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Model-Free Optimal Static Output Feedback Control using Integral Reinforcement Learning

Eslam Mahmoud

IBISC-EA4526

Université Évry Paris-Saclay

Évry, France

eslam.mahmoud@univ-evry.fr

Saïd Mammar

IBISC-EA4526

Université Évry Paris-Saclay

Évry, France

said.mammar@univ-evry.fr

Mohand Smaïli

IBISC-EA4526

Université Évry Paris-Saclay

Évry, France

mohand.smaili@univ-evry.fr

**Abstract**— This paper presents a novel model-free static output feedback (SOF) control strategy for continuous-time systems, leveraging integral reinforcement learning (IRL) within an off-policy framework. In many practical scenarios, full-state measurements are unavailable, requiring control based solely on sensor outputs. The proposed method addresses this challenge by utilizing only input-output data, avoiding the need for state estimators or observers. The approach offers a practical balance of simplicity, stability, and performance, making it a more accessible alternative than dynamic output feedback methods. The proposed algorithm is adaptive and suits real-world applications involving uncertain or unknown system dynamics. Simulation results demonstrate the method's effectiveness, with comparative analysis showing its better performance over existing model-free approaches.

**Index Terms**— Static Output Feedback, Reinforcement Learning, Adaptive Control, Optimal Control, Integral Reinforcement Learning, Model-Free Control.

## I. INTRODUCTION

In many practical applications, the full state information is not available; therefore, researchers have shown interest in output feedback control, which relies solely on the measured data from sensors. Similar to state feedback, output feedback can also be classified based on model knowledge: it is considered model-based when the system dynamics are known and used for designing controller gains, or model-free when no prior knowledge of the system dynamics is assumed.

Optimal feedback controllers are generally computed offline, assuming full knowledge of the system. On the other hand, adaptive controllers operate online and learn to compensate for unknown system dynamics, but they typically do not converge to the optimal solution. The advantage of Reinforcement Learning (RL) is that it can function in a model-free environment, making it suitable for learning both unknown systems and optimal control policies.

Reinforcement Learning was originally developed for discrete-time dynamic systems. However, dealing with continuous-time systems poses additional challenges due to the involvement of derivatives of the cost function in the Bellman equation, which requires knowledge of the system dynamics. To address this difficulty, Vrabie et al. [1]

introduced the concept of Integral Reinforcement Learning (IRL). The IRL version of the Bellman equation is given by:

$$V(x(t)) = \int_{t}^{t+\delta t} r(x(\tau), Kx(\tau))d\tau + V(x(t+\delta t))$$

This approach successfully applies reinforcement learning to address continuous-time feedback control problems. The equation above is used in an on-policy setting. On-policy algorithms use a behavior policy during the learning phase that matches the policy being learned. However, these methods often struggle to sufficiently explore the state and action spaces. Additionally, they frequently rely on episodic learning, which requires resetting the system states—an impractical condition in many real-world applications. In contrast, off-policy learning methods incorporate a control input that is not constrained by the feedback policy being learned. This provides greater flexibility in selecting exploration strategies [2]. Zhang et al. [3] demonstrated that off-policy methods are more effective, as they reduce the bias introduced by noise added to the behavior policy to satisfy the persistence of excitation (PE) condition. Despite Vrabie et al. addressing the challenge of unknown system dynamics, their method still required the input matrix to update the controller gains, meaning the approach was only partially model-free. Jiang [4] resolved the dependency on the input matrix B by substituting the Riccati equation with the terms of model matrices in the key IRL equation. This modification eliminated the dependence on system dynamics, resulting in a fully model-free algorithm for state feedback control. To extend IRL to output feedback control, Zhu et al. [5] proposed an algorithm; however, it has two main limitations. First, it remains only partially model-free. Second, it relies on a state observer to estimate the full system state, even though this estimation is required only during the learning phase. To avoid the dependency on state estimation, Modares et al. [6] replaced the states with N samples of delayed output. Additionally, they introduced a discount factor to noise-induced bias. However, this modification does not guarantee system stability. Later, Rizvi et al. [7] addressed the issues related to the discount factor and noise bias by introducing a state parameterization technique. They replaced the state with filtered inputs and outputs, defining a dynamic system

This research is supported by the French ARTISMO (Intelligent Estimation Algorithms for Mobility) financed by ANR (National Research Agency).
that embeds an observer within the algorithm. As a result, this approach utilizes only input and output data. In their state parameterization, they neglected the exponential term by assuming that, over a long period, the estimated state converges to the true state. However, this assumption may not hold when the available time is limited. To overcome this limitation, Chen et al. [8] introduced a pre-learning phase, allowing the algorithm sufficient time to approximate the true state, thereby achieving optimal control. Despite these advances, dynamic output feedback algorithms still require some knowledge of the system states, such as their dimensions. Furthermore, obtaining an appropriate initial controller for these algorithms remains challenging and is not straightforward, as will be discussed in the following sections. These limitations motivate the development of a novel static output feedback algorithm that is fully model-free, operates in an off-policy setting, and does not rely on observers or state parameterization techniques. This makes the proposed method easier to implement. A comparative study with existing model-free continuous-time approaches is conducted to validate the efficiency and demonstrate the performance of the proposed algorithm.

The paper is structured as follows: Section II presents the problem formulation, Section III introduces the proposed algorithm, Section IV discusses the simulation results, and Section V concludes with future work.

**Notation:** $\mathbb{R}$ denotes the set of real numbers, and $\mathbb{R}^{n_1}$ and $\mathbb{R}^{n_1 \times n_2}$ represent the $n_1$-dimensional real vector space and the space of real $n_1 \times n_2$ matrices, respectively. $\mathbb{Z}_+$ is the set of positive integers. The transpose of a matrix/vector is denoted by $T$, and $I_{n_1}$ is the identity matrix of dimension $n_1$. The Kronecker product is denoted by $\otimes$, $\text{vec}(\cdot)$ represents the vectorization operator, and the pseudoinverse of $z$ is $z^\dagger$.

## II. FROM STATE TO OUTPUT FEEDBACK

Consider the following linear time-invariant continuous-time system:

$$ \begin{cases} \dot{x}(t) = Ax(t) + Bu(t) \\ y(t) = Cx(t) \end{cases} \quad (1) $$

where $x \in \mathbb{R}^n$ is the state vector, $y \in \mathbb{R}^p$ is the output vector, and $u \in \mathbb{R}^m$ is the input vector. The matrices $A, B$, and $C$ are of appropriate and compatible dimensions.

In the following, it is assumed that the pair $(A, B)$ is controllable and the pair $(A, C)$ is observable.

By adding and subtracting the term $BK^i x$, where $K^i$ is the state feedback gain, and defining the closed-loop dynamic matrix as $A_{cl}^i = A - BK^i$, the system dynamics can be rewritten as:

$$ \dot{x} = (A - BK^i)x + B(u + K^i x) = A_{cl}^i x + B(u + K^i x) \quad (2) $$

Considering the cost function $v(x) = x^T P^i x$, where $P^i$ is a positive definite matrix, its time derivative is given by:

$$ \begin{aligned} \frac{d(x^T P^i x)}{dt} &= \dot{x}^T P^i x + x^T P^i \dot{x} \\ &= (A_{cl}^i x + B(u + K^i x))^T P^i x \\ &\quad + x^T P^i (A_{cl}^i x + B(u + K^i x)) \end{aligned} \quad (3) $$

After rearranging the terms, we obtain:

$$ \frac{d(x^T P^i x)}{dt} = x^T (A_{cl}^i)^T P^i + P^i A_{cl}^i) x + 2(u + K^i x)^T B^T P^i x \quad (4) $$

To eliminate the dependency on the system dynamics in (4), [4] suggested using the iterative Riccati equation developed by Kleinman [9]. This approach starts with an initial gain $K^0$ and iteratively solves the two equations in (5) until convergence, i.e. until there is no significant change in $P^i$.

$$ \begin{aligned} A_{cl}^i T P^i + P^i A_{cl}^i &= -Q - C^T K^i T R K^i \\ K^{i+1} &= R^{-1} B^T P^i \end{aligned} \quad (5) $$

where $Q > 0$ and $R > 0$ are symmetric positive definite weight matrices. Substituting (5) into (4), the equation becomes:

$$ \frac{d(x^T P^i x)}{dt} = -x^T (Q + K^{i+1} R K^i) x + 2(u + K^i x)^T R K^{i+1} x \quad (6) $$

Integrating both sides of (6) over the interval $[t, t + \delta t]$, we obtain:

$$ \begin{aligned} \int_{t}^{t+\delta t} \frac{d(x^T P^i x)}{d\tau} d\tau &= x^T (t+\delta t) P^i x (t+\delta t) - x^T(t) P^i x(t) \\ &= 2 \int_{t}^{t+\delta t} (u + K^i x)^T R K^{i+1} x d\tau \\ &\quad - \int_{t}^{t+\delta t} x^T Q^i x d\tau \end{aligned} \quad (7) $$

where $Q^i = Q + K^{i+1} R K^i$.

The model-free state feedback problem can thus be solved without requiring knowledge of the system matrices. In practice, the iteration primarily uses the second equation in (5) to compute $K^{i+1}$ from $P^i$, along with (7). However, it is important to note that the input matrix $B$ is still required. The previous key learning equation (7), originally developed in [4], was formulated within a policy iteration framework for model-free state feedback. Several approaches have been proposed to address the case of model-free output feedback. As suggested in [6] and [10], the state vector $x$ can be replaced with an $N$-sample delayed output vector $\bar{y}_t$ such that

$$ \bar{y}_t = Gx(t), \quad (8) $$

where

$$ \begin{aligned} \bar{y}_t &= [y(t)^T \quad y(t - \delta t)^T \quad \dots \quad y(t - (N-1)\delta t)^T]^T, \\ G &= [C^T \quad e^{-\delta t(A - BK)^T} C^T \quad \dots \quad e^{-N\delta t(A - BK)^T} C^T]^T, \end{aligned} \quad (9) $$

and $\delta t$ is the sampling interval, where $\bar{y} \in \mathbb{R}^{pN \times 1}$ and $G \in \mathbb{R}^{pN \times n}$.

Assuming that the matrix $G$ is invertible, the state $x(t)$ can be expressed as

$$ x(t) = G_N \bar{y}_t = [L_1, \dots, L_N] \bar{y}_t \quad (10) $$

where $G_N$ denotes the inverse of $G$. Consequently, the control input can be rewritten as

$$ u = -Kx = -KG_N \bar{y}_t = -\bar{K}_o \bar{y}_t \quad (11) $$
where $\bar{K}_o$ represents the dynamic output feedback gain. The matrix $G$, however, depends on the system matrices $A, B$, and $C$. To obtain the initial $\bar{y}$ from the initial states $x(0)$ and also to compute the initial $\bar{K}_o$, we must first determine $G$ to initiate the algorithm. The number of samples $N$ significantly impacts the learning results. Typically, $N$ is set to the dimension of the state, although it is often beneficial to set $N$ higher. However, the optimal value of $N$ remains undetermined, and one must ensure that the matrix $G$ is full column rank. Simulation tests showed that varying the value of $N$ affects performance, although the system remains $N$-sample stabilizable.

Another approach to transform state feedback into output feedback is to consider a delayed input $u(t-\eta)$, where $\eta$ represents an unknown small input delay [10]. This method involves state parameterization in terms of filtered input and output [7]. Specifically, consider the state as obtained from

$$x(t) = M_u \zeta_u(t) + M_y \zeta_y(t) + e^{(A-LC)t} x(0), \quad (12)$$

where $\zeta_u$ and $\zeta_y$ are the filtered input and output, respectively. These can be derived from the following dynamic systems:

$$\begin{aligned} \dot{\zeta}_u &= A_\zeta \zeta_u + bu, \\ \dot{\zeta}_y &= A_\zeta \zeta_y + by, \end{aligned} \quad (13)$$

Where $A_\zeta$ is a canonical form matrix that shares the same eigenvalues as $(A-LC)$, defined by the user. These two systems are considered embedded observers within the algorithm. Additionally, $M_u$ and $M_y$ are matrices derived from the coefficients of the numerator of $(sI - A + LC)^{-1}B$ and $(sI - A + LC)^{-1}L$, respectively. The matrix $L$ is a user-defined observer matrix chosen to achieve the desired eigenvalues, and $I$ represents the identity matrix.

Since $\lim_{t \to \infty} e^{(A-LC)t}\dot{x}(0) = 0$, the last term of (12) is ignored, yielding

$$\begin{aligned} x(t) &= M_u \zeta_u(t) + M_y \zeta_y(t) \\ &= [M_u \quad M_y] z, \end{aligned} \quad (14)$$

where $z = \begin{bmatrix} \zeta_u \\ \zeta_y \end{bmatrix}$. The control input $u = -Kx$ can thus be rewritten as

$$\begin{aligned} u &= -K \begin{bmatrix} M_u & M_y \end{bmatrix} z \\ &= -\bar{K}_o z, \end{aligned} \quad (15)$$

where $\bar{K}_o = K \begin{bmatrix} M_u & M_y \end{bmatrix}$. The gain $\bar{K}_o$ relates only to the system input $u$ and output $y$. Note that, in this case, it is necessary to collect data over some time to ensure that the last term of (12) is close to zero, which enhances the results. Additionally, the problem of constructing $M_u, M_y, \zeta_u, \zeta_y$, and $A_\zeta$, which depend on the system dynamics, still remains. In the next section, an algorithm is presented that overcomes the need for system matrices. It directly works with input and output without using an observer or state parameterization, making it more attractive and easier to implement.

### III. COMPUTATIONAL ADAPTIVE OPTIMAL STATIC OUTPUT CONTROL DESIGN WITH COMPLETELY UNKNOWN DYNAMICS

The purpose of this section is to develop a new algorithm derived from the iterative Riccati equation. We first present

the algorithm for model-based output feedback without using any output filtering.

#### A. Model-Based Algorithm

The notations are similar to those used in the previous section. For the iteration index $i = 0, 1, \dots$, the gain matrices $K_o^i$ and the matrix $P^i$ can be obtained using a modified Riccati equation developed in [11], which is formulated here within a policy iteration framework described in Algorithm 1. Here, $\epsilon > 0$ is a user-defined threshold value.

##### Algorithm 1 Model-Based Policy Iteration for Static Output Feedback

**Initialization:** Choose any stabilizing static output feedback gain matrix $K_o^0$.

**Execution:**

Let the iteration index $i = 0$. Solve for $P^0$ the following equation:

$$\begin{aligned} (A - BK_o^0 C)^T P^0 + P^0 (A - BK_o^0 C) \\ + Q + C^T (K_o^0)^T R K_o^0 C = 0 \end{aligned} \quad (16)$$

**repeat**

Increment the iteration index $i = i + 1$.

Calculate the state feedback gain matrix:

$$K_o^i = R^{-1} B^T P^{i-1} C^T \quad (17)$$

Solve the following equation:

$$\begin{aligned} (A - BK_o^i C)^T P^i + P^i (A - BK_o^i C) \\ + Q + C^T (K_o^i)^T R K_o^i C = 0 \end{aligned} \quad (18)$$

**until** $\|P^i - P^{i-1}\| \le \epsilon$ for $i \ge 1$

**Return:** $P^i, K_o^i$

The proof of the algorithm is omitted due to the page limit.

#### B. Model-free algorithm

We now move to the model-free version of the algorithm. Consider again the system of equation (1), the linear time-invariant continuous-time system

$$\dot{x} = Ax + Bu \quad (19)$$

Add and subtract $BK_o^i Cx$, where $K_o^i$ is the static output feedback gain, so the equation becomes

$$\dot{x} = (A - BK_o^i C)x + B(u + K_o^i Cx) = A_{cl}^i x + B(u + K_o^i Cx) \quad (20)$$

Taking the derivative of the cost function $v(x(t)) = x^T P^i x$,

$$\begin{aligned} \frac{d(x^T P^i x)}{dt} &= \dot{x}^T P^i x + x^T P^i \dot{x} \\ &= (A_{cl}^i x + B(u + K_o^i Cx))^T P^i x \\ &\quad + x^T P^i (A_{cl}^i x + B(u + K_o^i Cx)) \end{aligned} \quad (21)$$

After rearranging, we get

$$\frac{d(x^T P^i x)}{dt} = x^T (A_{cl}^i)^T P^i + P^i A_{cl}^i)x + 2(u + K_o^i Cx)^T B^T P^i x \quad (22)$$

From the Riccati equation, we have

$$\begin{aligned} A_{cl}^i P^i + P^i A_{cl}^i &= -C^T Q_o C - C^T K_o^i R K_o^i C \\ R K_o^{i+1} C &= B^T P^i \end{aligned} \quad (23)$$
Substituting (23) into (22), we get

$$
\begin{align}
\frac{d(x^T P^i x)}{dt} &= -x^T (C^T Q_o C + C^T K_o^i R K_o^i C)x \nonumber \\
&\quad + 2(u + K_o^i C x)^T R K_o^{i+1} C x \tag{24} \\
&= -x^T C^T (Q_o + K_o^i R K_o^i) C x \nonumber \\
&\quad + 2(u + K_o^i C x)^T R K_o^{i+1} C x \nonumber
\end{align}
$$

From the relation $y = Cx$, we obtain

$$
\frac{d(x^T P^i x)}{dt} = -y^T (Q_o + K_o^{iT} R K_o^i) y + 2(u + K_o^i y)^T R K_o^{i+1} y \quad (25)
$$

Up to this point, the procedure is almost identical to the
previous algorithms, where we need to replace the states
multiplied with the kernel matrix $P$ in the cost function.

Now, we replace $x$ with $C^\dagger y$ (see Lemma 1):

$$
\begin{align}
x^T P x &= y^T (C^T)^T P C^T y, \quad C^\dagger = C^T (C C^T)^{-1} \equiv C^T \tag{26} \\
x^T P x &= y^T (C P C^T) y \nonumber
\end{align}
$$

This becomes

$$
x^T P x = y^T \bar{P} y \quad (27)
$$

**Lemma 1 (Stability Under Partial State Reconstruction).**
Assume the system (19) with output $y = Cx$ satisfies:

1) C has full row rank, ensuring that $C^\dagger = C^T (C C^T)^{-1}$ exists. Since $C C^T$ is the identity matrix $I$, it follows that $C^\dagger = C^T$.

2) The system is static output feedback stabilizable [6].

Then, substituting $x = C^\dagger y$ in the cost function $x^T P x = y^T \bar{P} y$ preserves closed-loop stability under the static output feedback gain $K_o$. Unmeasured states remain bounded, and the system achieves stability under the static output feedback gain $K_o$.

Proof: The term $y = Cx$ omits unmeasured states, and substituting $x = C^\dagger y$ only reconstructs the measurable part of $x$, leading to partial state reconstruction. As a result, the cost function $x^T P x$ becomes invalid unless the unmeasured subspace is stable. Since $K_o$ acts solely on the outputs, it cannot directly affect the unmeasured states.

Integrating both sides of (24) gives us:

$$
\begin{equation}
\begin{aligned}
\int_t^{t+\delta t} \frac{d(x^T P^i x)}{dt} d\tau &= \int_t^{t+\delta t} \frac{d(y^T \bar{P}^i y)}{dt} d\tau \\
&= y^T (t+\delta t) \bar{P}^i y (t+\delta t) - y^T (t) \bar{P}^i y (t) \\
&= 2 \int_t^{t+\delta t} (u + K_o^i y)^T R K_o^{i+1} y \, d\tau \\
&\quad - \int_t^{t+\delta t} y^T Q^i y \, d\tau
\end{aligned}
\tag{28}
\end{equation}
$$

where $Q^i = Q_o + K_o^{iT} R K_o^i$.

Using the Kronecker product to separate the kernel matrix:

$$
\begin{align*}
& y^T (t + \delta t) \bar{P}^i y (t + \delta t) - y^T (t) \bar{P}^i y (t) \\
&= [y^T (t + \delta t) \otimes y^T (t + \delta t) - y^T (t) \otimes y^T (t)] \mathrm{vec}(\bar{P}^i) \\
&= (\bar{y}_{t+\delta t} - \bar{y}_t)^T \hat{P}^i
\end{align*}
$$

where $\hat{P} \in \mathbb{R}^{\frac{1}{2}p(p+1)}$ and $\bar{y} \in \mathbb{R}^{\frac{1}{2}p(p+1)}$ are defined as:

$$
\begin{align*}
\hat{P} &= [p_{11}, 2p_{12}, \dots, 2p_{1n}, p_{22}, 2p_{23}, \dots, 2p_{(n-1)n}, p_{nn}]^T, \\
\bar{y}_t &= [y_1^2, y_1y_2, \dots, y_1y_n, y_2^2, y_2y_3, \dots, y_{n-1}y_n, y_n^2]^T
\end{align*}
$$

Also,

$$
y^T Q^i y = (y^T \otimes y^T) \mathrm{vec}(Q^i),
$$

$$
(u + K_o^i y)^T R K_o^{i+1} y = ((y^T \otimes y^T)(I_p \otimes K_o^{iT} R) + (y^T \otimes u^T)(I_p \otimes R)) \mathrm{vec}(K_o^{i+1})
$$

For a positive integer $l$, we define the regressors $\delta_{yy} \in \mathbb{R}^{l \times \frac{1}{2}p(p+1)}$, $I_{yy} \in \mathbb{R}^{l \times p^2}$, and $I_{yu} \in \mathbb{R}^{l \times mp}$ such that:

$$
\begin{align}
\delta_{yy} &= [\bar{y}(t_1) - \bar{y}(t_0) \quad \bar{y}(t_2) - \bar{y}(t_1) \quad \dots \quad \bar{y}(t_l) - \bar{y}(t_{l-1})]^T, \notag \\
I_{yy} &= \left[ \int_{t_0}^{t_1} y \otimes y \, d\tau \quad \int_{t_1}^{t_2} y \otimes y \, d\tau \quad \dots \quad \int_{t_{l-1}}^{t_l} y \otimes y \, d\tau \right]^T, \notag \\
I_{yu} &= \left[ \int_{t_0}^{t_1} y \otimes u \, d\tau \quad \int_{t_1}^{t_2} y \otimes u \, d\tau \quad \dots \quad \int_{t_{l-1}}^{t_l} y \otimes u \, d\tau \right]^T \tag{29}
\end{align}
$$

where $0 \le t_0 < t_1 < \dots < t_l$.

Then, for any given stabilizing gain matrix $K_o^i$, we can
rewrite (28) in compact form:

$$
\Theta^i \left[ \hat{P}^i \middle| \mathrm{vec}(K_o^{i+1}) \right] = \Psi^i \quad (30)
$$

where $\Theta^i \in \mathbb{R}^{l \times [\frac{1}{2}p(p+1)+mp]}$ and $\Psi^i \in \mathbb{R}^l$ are defined as:

$$
\begin{align}
\Theta^i &= [\delta_{yy}, -2I_{yy}(I_p \otimes K_o^{iT} R) - 2I_{yu}(I_p \otimes R)], \tag{31} \\
\Psi^i &= -I_{yy} \operatorname{vec}(Q^i) \nonumber
\end{align}
$$

If $\Theta^i$ has full column rank, (30) can be directly solved as:

$$
\left[ \hat{P}^i \middle| \operatorname{vec}(K_o^{i+1}) \right] = (\Theta^i)^{-1} \Theta^i \Psi^i. \quad (32)
$$

Since in (32) we have $\hat{P} \in \mathbb{R}^{\frac{1}{2}p(p+1)}$ and $\mathrm{vec}(K_o^{i+1}) \in \mathbb{R}^{mp}$, there are $\frac{1}{2}p(p+1) + mp$ unknowns. Therefore, we need at least $l \ge \frac{1}{2}p(p+1) + mp$ samples, such that:

$$
\mathrm{rank}([I_{yy}, I_{yu}]) = \frac{p(p+1)}{2} + mp \quad (33)
$$

Then $\Theta^i$ has full column rank for all $i \in \mathbb{Z}_+$. This can be achieved by injecting a noise signal into the control input during data collection. The noise can take many forms (e.g., random noise, exponentially decaying noise, or a sum of sinusoidal signals with a wide range of frequencies) [4]. Algorithm 2 summarizes the developed model-free algorithm, where $e$ is the probing noise signal, and $\epsilon$ is a user-defined threshold value.

C. *Proof of Convergence*

From the analysis of model-based output feedback, it can
be derived that (28) is equivalent to (18). Therefore, equation
(32) is equivalent to (18) and (17). According to the proof of
convergence for the model-based approach, we have $K_o^i \to$
$K_o^*$ and $P^i \to P^*$ as $i \to \infty$. The proof is complete.

The proposed method is not affected by the noise problem.
The proof is the same as the one presented in [7] and is
omitted due to the page limit.
**Algorithm 2** Model-Free Policy Iteration for Static Output Feedback

**Initialization:** Choose any stabilizing static output feedback gain matrix $K_o^0$ and set $i = 0$.

**Execution:**

Let $u = -K_o^0 y + e$, where $t \in [t_0, t_1]$, and compute $\delta_{yy}$, $I_{yy}$, and $I_{yu}$ until the condition in equation (33) is satisfied.

**repeat**

Increment the iteration index: $i = i + 1$.

Solve for $P^i$ and $K_o^{i+1}$ from the following equation:

$$ \begin{bmatrix} \hat{P}^i \\ \operatorname{vec}(K_o^{i+1}) \end{bmatrix} = (\Theta^{iT} \Theta^i)^{-1} \Theta^{iT} \Psi^i. $$

**until** $\|P^i - P^{i-1}\| \le \epsilon$ for $i \ge 1$

**Return:** $P^i, K_o^i$

## IV. SIMULATION STUDY

This section presents two examples: a second-order system for comparison with existing methods, and a vehicle lateral dynamics model to validate the proposed approach.

### A. Comparison of Different Algorithms on a Simple System

Consider the dynamic system as

$$ A = \begin{bmatrix} -1 & 1 \\ -1 & 0 \end{bmatrix}, \quad B = \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \quad C = \begin{bmatrix} 1 & 0 \end{bmatrix} \quad (34) $$

The initial states for the simulation are taken as $x(0) = \begin{bmatrix} 2 \\ 2 \end{bmatrix}$, and the initial static output feedback controller gains are obtained using the Linear Matrix Inequalities (LMI) method presented in [12], with the value of $K_o^0 = 2.5192$. Additionally, $Q_o$ and $R$ are chosen to be 3 and 1, respectively. The threshold value $\epsilon$ is selected as $1 \times 10^{-3}$, and the sampling time $\delta_t$ is 0.02s. The final value for the controller gain is -0.5302. The simulations were conducted on a computer with an Intel(R) Core(TM) i7-10700 CPU 2.90GHz, 16GB of RAM, and MATLAB R2023b. The probing noise $e$ is set as a sum of sinusoidal signals:

$$ e = \sum_{i=1}^{8 \times f_N} \sin(\omega_i t + \text{random} \times 2 \times \pi) \quad (35) $$

The frequency range $\omega_i$ [rad/s] is defined as $[2\pi f_N, -2\pi f_N]$, where $f_N$ is the Nyquist frequency, satisfying the Shannon criterion [13]. Data collection and controller gain computation each takes 0.01s. Fig. 1 shows that our algorithm performs similarly to LQR, despite relying only on input and output data. In Fig. 1, y-initial and y-final represent the system responses using the initial and final optimal controller gains, respectively. Fig. 2, included for presentation, demonstrates convergence, even though state access is not assumed during training.

Applying the N-samples method from [6] requires parameters such as the delayed output horizon $N$ to construct $\bar{y}(0)$, the state dimension $n$, and the initial dynamic controller $\bar{K}_o^0$, which depends on $G_N$ and the system dynamics. In contrast, our approach only requires an initial stable static output feedback controller, which can be easily obtained. We set $N = 6$,

<figure><img src="image_3.png" /><figcaption>Fig. 1: Comparison between SOF controller and LQR</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: Tracking states after applying the SOF controller</figcaption></figure>

yielding $\bar{y}(0) = [2 \ 1.9994 \ 1.99976 \ 1.9946 \ 1.9903 \ 1.9847]^T$ and an initial controller $\bar{K}_o^0 = [2.4547 \ 1.5364 \ 0.5923 \ -0.3778 \ -1.3741 \ -2.3966]$ from the state feedback gains $K_x = [0.1199 \ 0.3398]$. Data collection and controller gain computation take 0.09s and 0.015s, respectively, with final controller gains of $[2.0429 \ 0.9651 \ 0.0541 \ -0.6317 \ -1.1325 \ -1.3308]$.

For the dynamic output feedback controller in [7], the state dimension $n$ is required to determine the number of desired eigenvalues needed to construct $A_\zeta$. These eigenvalues influence how quickly the constructed states converge to the true value. Additionally, $M_u$ and $M_y$ must be constructed to obtain the initial dynamic controller gains $\bar{K}_o^0$, both of which depend on the system dynamics. To improve the results, we added 5s pre-learning phase before data collection, as done in [8]. The eigenvalues are all set to -3. The constructed matrices $A_\zeta$, $M_u$, and $M_y$ are:

$$ A_\zeta = \begin{bmatrix} 0 & 1 \\ -9 & -6 \end{bmatrix}, \quad M_u = \begin{bmatrix} 1 & 0 \\ 6 & 1 \end{bmatrix}, \quad M_y = \begin{bmatrix} 8 & 5 \\ 3 & 8 \end{bmatrix} $$

The initial dynamic gain is chosen as $\bar{K}_o^0 = [2.1587 \ 0.3398 \ 1.9786 \ 3.3179]$. It takes 1.7s for data collection and 0.02s for solving the controller gains. The final gains are $[-3.6073 \ 1.1962 \ -1.9178 \ -0.9361]$. Fig. 3 shows the comparison between our proposed method and the other two methods, with LQR ($K_x^* = [0.26779 \ 0.7321]$) as a reference for comparison.
<figure><img src="image_5.png" /><figcaption>Fig. 3: Comparison of output feedback algorithms with LQR</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4: Validation of the proposed method on vehicle lateral dynamics</figcaption></figure>

## B. Validation on a Vehicle Lateral Dynamics System

Consider the vehicle lateral dynamics system given by:

$$ \dot{x} = \begin{bmatrix} -4.402 & -12.460 & 0 & 0 \\ 20.87 & -5.187 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 10 & 15 & 0 \end{bmatrix} x + \begin{bmatrix} 24.127 \\ 15.8609 \\ 0 \\ 0 \end{bmatrix} u \quad (36) $$

The system vector is defined as $x = [\dot{y} \dot{\Psi} \Psi_L y_L]^T$, where $\dot{y}$ represents the lateral velocity, $\dot{\Psi}$ is the vehicle's yaw angle, $\Psi_L$ is the relative yaw angle, and $y_L$ is the lateral deviation. The control input is the front steering angle, denoted as $u = \delta_f$. It is taken from [14]. The Lane Keeping Assistance (LKA) problem aims to keep the vehicle centered within its lane by applying corrective steering inputs. Formulated as a regulation problem, the goal is to reduce the lateral deviation to zero. The output vector consists of the last three states, which are easily measured by conventional sensors like an inertial unit and a camera: $y = [\dot{\Psi} \Psi_L y_L]^T$. The lateral velocity $\dot{y}$ is typically estimated using an observer. The objective is to design a stabilizing static output feedback controller without relying on $\dot{y}$. The initial SOF controller gains are chosen as $K_o^0 = [0.0833 \ 0.6106 \ 0.1125]$, with $Q_o = 15 \times I_3$, and $R=6$. The threshold value $\epsilon$ is chosen as $1 \times 10^{-3}$. The initial states for simulation are $x(0) = [0 \ 0 \ 0 \ 1]^T$ with a sampling time $\delta t = 0.05$s. It takes 0.1s to collect the data and 0.02s to solve for the controller. Fig. 4 shows the proposed controller's performance, closely matching the LQR ($K_x^* = [0.8565 \ 1.6079 \ 2.8481 \ 1.5811]$). In this figure, y-initial, and y-final represent the system responses using the initial and final optimal controller gains, respectively.

The final SOF controller gains are [1.5957 2.8123 1.7294].

**Remark 1.** The amplitude of the exploration noise must be carefully selected, as an inappropriate value can destabilize the system. In the first problem, it is set to 1, while in the LKA problem, it is chosen to be 1e-5.

## V. CONCLUSIONS

We developed a static output feedback, model-free policy iteration algorithm that relies solely on input and output data, eliminating the need for state parameterization or observer design during learning. Despite this simplification,

the algorithm delivers satisfactory results, making it a strong alternative to dynamic output feedback methods, due to its ease of implementation. Future work will focus on extending the method to nonlinear systems.

## REFERENCES

[1] D. Vrabie, O. Pastravanu, M. Abu-Khalaf, and F. L. Lewis, "Adaptive optimal control for continuous-time linear systems based on policy iteration," Automatica, vol. 45, no. 2, pp. 477-484, 2009.

[2] S. A. A. Rizvi and Z. Lin, Output Feedback Reinforcement Learning Control for Linear Systems. Springer, 2023.

[3] L. Zhang, J. Fan, W. Xue, V. G. Lopez, J. Li, T. Chai, and F. L. Lewis, "Data-driven h optimal output feedback control for linear discrete-time systems based on off-policy q-learning," IEEE Trans. on Neural Networks and Learning Systems, vol. 34, no. 7, pp. 3553-3567, 2021.

[4] Y. Jiang and Z.-P. Jiang, "Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics," Automatica, vol. 48, no. 10, pp. 2699-2704, 2012.

[5] L. M. Zhu, H. Modares, G. O. Peen, F. L. Lewis, and B. Yue, "Adaptive suboptimal output-feedback control for linear systems using integral reinforcement learning," IEEE Transactions on Control Systems Technology, vol. 23, no. 1, pp. 264-273, 2014.

[6] H. Modares, F. L. Lewis, and Z.-P. Jiang, "Optimal output-feedback control of unknown continuous-time linear systems using off-policy reinforcement learning," IEEE Transactions on Cybernetics, vol. 46, no. 11, pp. 2401-2410, 2016.

[7] S. A. A. Rizvi and Z. Lin, "Output feedback reinforcement learning control for the continuous-time linear quadratic regulator problem," in 2018 Annual American Control Conference (ACC). IEEE, 2018, pp. 3417-3422.

[8] C. Chen, L. Xie, K. Xie, F. L. Lewis, and S. Xie, "Adaptive optimal output tracking of continuous-time systems via output-feedback-based reinforcement learning," Automatica, vol. 146, p. 110581, 2022.

[9] D. Kleinman, "On an iterative technique for riccati equation computations," IEEE Transactions on Automatic Control, vol. 13, no. 1, pp. 114-115, 1968.

[10] G. Wang, B. Luo, and S. Xue, "Integral reinforcement learning-based optimal output feedback control for linear continuous-time systems with input delay," Neurocomputing, vol. 460, pp. 31-38, 2021.

[11] J. Zhao, J. Na, G. Gao, S. Han, Q. Chen, and S. Wang, "Online output-feedback optimal control of linear systems based on data-driven adaptive learning," IFAC-PapersOnLine, vol. 53, no. 2, pp. 1596-1601, 2020.

[12] M. Chadli and P. Borne, Multiple models approach in automation: Takagi-Sugeno fuzzy systems. John Wiley & Sons, 2012.

[13] C. Li, G. G. Yin, L. Guo, C.-Z. Xu et al., "State observability and observers of linear-time-invariant systems under irregular sampling and sensor limitations," IEEE Transactions on Automatic Control, vol. 56, no. 11, pp. 2639-2654, 2011.

[14] E. Mahmoud, D. Ichalal, and S. Mammar, "Reinforcement learning based autonomous vehicles lateral control," in 2023 IEEE International Conference on Networking, Sensing and Control (ICNSC), vol. 1. IEEE, 2023, pp. 1-6.