# Data-Driven Robust Control Using Reinforcement Learning
## 2 Materials and Methods
### Robust Control Using Reinforcement Learning
#### 2.1.3 Stability Analysis  
With the control policy as described in Equation (17), the equation for the closed-loop system can be derived as follows:  
\[\dot{x}(t)=Ax(t)-BKx(t)=(A-BK)x(t). \tag{26}\]  
**Lemma 1**. : _Assuming that the closed-loop system described by Equation (26) is stable, solving for \(P\) in Equation (8) is equivalent to finding the solution of the underlying Lyapunov equation [8]:_  
\[P(A-BK)+(A-BK)^{T}P=-Q. \tag{27}\]Proof of Lemma 1.: We start with Equation (27) and try to prove that matrix \(P\) is also the solution of Equation (8). Consider \(V(x(t))=x^{T}(t)Px(t)\), where \(P\) is the solution of Equation (27):  
\[ \begin{split}\dot{V}(x(t))&=\frac{d(x^{T}(t)Px(t))}{ dt}\\ &=\dot{x}^{T}(t)Px(t)+x^{T}(t)Px(t)\\ &=x^{T}(t)\Big{[}(A-BK)^{T}P+P(A-BK)\Big{]}x(t)\\ &=-x^{T}(t)Qx(t)\quad\text{(using Equation (\ref{eq:P1})).}\end{split}  \tag{28}\]  
Since the closed-loop system is stable, the Lyapunov Equation (27) has a unique solution, \(P_{i}>0\). From (28), this solution will satisfy  
\[\frac{d(x^{T}(t)P_{i}x(t))}{dt}=-x^{T}(t)Qx(t), \tag{29}\]  
which is equivalent to  
\[x^{T}(t+T)Px(t+T)-x^{T}(t)Px(t)=\int_{t}^{t+T}-x^{T}(\tau)Qx(\tau)d\tau. \tag{30}\]  
Therefore, \(P\) is also the solution of Equation (8). Lemma 2.: _Given matrices \(E\) and \(F\) with appropriate dimensions, the following LMI can be obtained:_  
\[EF^{T}+FE^{T}\leq EE^{T}+FF^{T}. \tag{31}\]  
Proof of Lemma 2.: From the properties of matrix norm, we have  
\[(E-F)(E-F)^{T}\geq 0, \tag{32}\]  
which is equivalent to  
\[EE^{T}+FF^{T}-EF^{T}-FE^{T}\geq 0 \tag{33}\]  
or  
\[EF^{T}+FE^{T}\leq EE^{T}+FF^{T}. \tag{34}\]  
Lemma 3.: _Given \(A\) as a square matrix with dimension \(n\times n\) and \(x\) as a vector with dimension \(n\times 1\), the following LMI can be obtained:_  
\[x^{T}Ax\leq x^{T}Cx, \tag{35}\]  
_where \(C=\operatorname{maximize}(A,x)\) as in Definition 1._  
Proof of Lemma 3.: We have  
\[ \begin{split} x^{T}Ax&=\sum_{i,j=1,2\dots n}a_{ij} x_{i}x_{j}\leq\sum_{i,j=1,2\dots n}|a_{ij}x_{i}x_{j}|\\ &=\sum_{i,j=1,2\dots n}c_{ij}x_{i}x_{j}=x^{T}Cx,\end{split}  \tag{36}\]  
where \(c_{ij}=\left\{ \begin{array}{l}\max(a_{ij})\text{ if }x_{i}x_{j}\geq 0\\ \min(a_{ij})\text{ if }x_{i}x_{j}<0\end{array} \right.\) with \(i,j=1\dots n\). Theorem 1.: _Consider a dynamic system that can be represented by Equation (1) with the state matrix \(A\) unknown. Assume that the sign of all state variables cannot be changed between each policy update interval and the estimated value function at iteration \(i\) is \(V_{i}(x(t))=x^{T}(t)P_{i}x(t)\) with \(P_{i}=\hat{P}_{i}+\Delta P_{i}\). If_  
* _The current control policy_ \(u(t)=\pi_{i}(x(t))=-K_{i}x(t)\) _is stabilizing;_
* _The LMI given in (_22_) is satisfied with some positive constant_ \(\gamma\)_;_  
_then the closed-loop system with the control policy \(u(t)=-K_{i+1}x(t)\) is quadratic stable with convergence rate \(\alpha\)._  
Proof of Theorem 1.: Since the current control policy is stable, the estimated parameter matrix \(P_{i}\) is positive definite. Hence, \(V_{i}(x(t))=x_{t}^{T}P_{i}x_{t}>0\). Here, \(V_{i}(x(t))\) is used as the Lyapunov function for the updated control policy \(u(t)=\pi_{i+1}(x(t))=-K_{i+1}x(t)\). For notation convenience, the state vector \(x(t)\) and input vector \(u(t)\) are denoted as \(x_{t}\) and \(u_{t}\), respectively.