# Robust Control of an Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning
## 3 Robust Control of Uncertain Linear System  
This section mainly presents the robust optimal control methods for the uncertain IPS modeled in the previous section by selecting the appropriate performance index function and solving an ARE to construct the robust control law. When the uncertain parameters of the system change within a certain range, this robust control law can cause the system to become asymptotically stable.  
The following lemmas are proposed to prove the main results of this paper.  
Lemma 1: _The nominal system (12) corresponding to system (11) is stabilizeable._  
**Proof.** For the four-dimensional continuous time-invariant system presented in system (12), the controllability matrix is constructed as  
\[G=[B\quad A(z_{0})B\quad A(z_{0})^{2}B\quad A(z_{0})^{3}B]\]  
Therefore, we have  
\[rank(G)=rank\begin{bmatrix}0&0.8832&-0.0780&1.4899\\ 0.8832&-0.0780&1.4899&-0.2626\\ 0&2.3566&-0.2082&65.5990\\ 2.3566&-0.2082&65.5990&-6.1442\end{bmatrix}=4\]  
Therefore, system (12) is completely controllable, which means that the system can be stabilized. This completes the proof.  
**Lemma 2**.: _There is an \(m\times n\) matrix \(\delta(z)\), such that the system matrices \(A(z)\) and \(A(z_{0})\) satisfy the following matched condition._  
\[A(z)-A(z_{0})=B\delta(z) \tag{13}\]  
**Proof.**  
\[A(z)-A(z_{0})=\begin{bmatrix}0&0&0&0\\ 0&0.8832(0.1-z)&0&0\\ 0&0&0&0\\ 0&2.3566(0.1-z)&0&0\end{bmatrix}=\begin{bmatrix}0\\ 0.8832\\ 0\\ 2.3566\end{bmatrix}\delta(z)=B\delta(z) \tag{14}\]  
where  
\[\delta(z)=\begin{bmatrix}0&0.1-z&0&0\end{bmatrix} \tag{15}\]  
This completes the proof.  
**Lemma 3**.: _For any \(z\in[0,1]\), there exists a positive semidefinite matrix \(F\), such that \(\delta(z)\) satisfies_  
\[\delta(z)^{T}\delta(z)\leq F \tag{16}\]  
_where \(F\geq 0\)._  
**Proof.** According to Lemma 2, we can obtain  
\[\delta(z)^{T}\delta(z)=\begin{bmatrix}0&0&0&0\\ 0&(0.1-z)^{2}&0&0\\ 0&0&0&0\\ 0&0&0&0\end{bmatrix}\leq\begin{bmatrix}0&0&0&0\\ 0&0.81&0&0\\ 0&0&0&0\\ 0&0&0&0\end{bmatrix}=F \tag{17}\]  
This completes the proof.  
For nominal system (12), we construct the following ARE.  
\[SA(z_{0})+A(z_{0})^{T}S+M-SBB^{T}S=0 \tag{18}\]  
where \(M=F+I\). According to the above three lemmas and ARE (18), we propose the following theorem.  
**Theorem 1**.: _Let us suppose that \(S\) is a symmetric positive definite solution to ARE (18). Then, for all uncertainties \(z\in[0,1]\), the feedback control \(u=Kx\), \(K=-B^{T}S\) can make system (11) asymptotically stable._Proof.: We define the following Lyapunov function.  
\[V(x)=x^{T}Sx \tag{19}\]  
We set \(u=Kx\) and take the time derivative of Lyapunov Function (19) along system (11). We can then obtain  
\[\dot{V}(x)=x^{T}\Big{[}(A(z))^{T}+K^{T}B^{T}\Big{]}Sx+x^{T}S[(A(z))+BK]x\]  
According to Lemma 2, we can obtain  
\[\dot{V}(x)= x^{T}\Big{[}(A(z_{0})+B\delta(z))^{T}+K^{T}B^{T}\Big{]}Sx+x^{T}S[(A (z_{0})+B\delta(z))+BK]x\] \[= x^{T}[A(z_{0})^{T}S+SA(z_{0})+\delta(z)^{T}B^{T}S]x+x^{T}SB\delta( z)x+2x^{T}SBKx\]  
On substituting ARE (18) into the above equation, we obtain  
\[\dot{V}(x)= -x^{T}Mx+x^{T}SBB^{T}Sx+x^{T}\delta(z)^{T}B^{T}Sx+x^{T}SB\delta(z) x+2x^{T}SBKx\]  
because \(K=-B^{T}S\),  
\[\dot{V}(x)= -x^{T}Mx+x^{T}K^{T}Kx-x^{T}\delta(z)^{T}Kx+x^{T}SB\delta(z)x+2x^{T}SBKx\]  
As  
\[-x^{T}K^{T}Kx-2x^{T}K^{T}\delta(z)x=-x^{T}\big{(}K+\delta(z)\big{)}^{T}\big{(} K+\delta(z)\big{)}x+x^{T}\delta(z)^{T}\delta(z)x\leq x^{T}\delta(z)^{T} \delta(z)x\]  
we can obtain  
\[\dot{V}(x)= -x^{T}Mx+x^{T}K^{T}Kx-x^{T}\delta(z)^{T}Kx+x^{T}SB\delta(z)x+2x^{ T}SBKx\] \[= -x^{T}Mx-x^{T}K^{T}Kx-2x^{T}\delta(z)^{T}Kx\] \[= -x^{T}Mx-x^{T}\big{(}K+\delta(z)\big{)}^{T}\big{(}K+\delta(z) \big{)}x+x^{T}\delta(z)^{T}\delta(z)x\] \[\leq -x^{T}Mx+x^{T}\delta(z)^{T}\delta(z)x\] \[\leq -x^{T}(M-F)x\] \[\leq -x^{T}x\]  
Therefore,  
\[\dot{V}(x)=0 x=0\] \[\dot{V}(x)\leq 0 x\
eq 0\]  
According to the Lyapunov stability theorem [35], the uncertain system (11) is asymptotically stable. Theorem 1 has thus been proved.