# Robust Control of an Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning
## 5 Robust Control of Nonlinear IPS
### Robust Control of Nonlinear IPS  
To obtain the robust control law for an uncertain nonlinear IPS, we propose the following two lemmas. **Lemma 5**. : _There exists an uncertain function \(G(z,x)\) such that \(\bar{F}(z,x)\) can be decomposed into the following form._  
\[\bar{F}(z,x)=BG(z,x)\]  
Proof. : \[\bar{F}(z,x)= \begin{bmatrix}0\\ \frac{-(1+m_{1}L^{2})z}{W}x_{2}\\ \frac{-m_{1}Lz}{W}x_{2}\end{bmatrix} [cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+ 2x_{3}-4x_{4}}{x_{2}}]\] \[= \begin{bmatrix}0\\ \frac{l+m_{1}L^{2}}{W}\\ 0\\ \frac{m_{1}L}{N}\end{bmatrix} (-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1} +2x_{3}-4x_{4}}{x_{2}}])=BG(z,x)\]  
where \(G(z,x)=-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+2x_{3}-4x_{4}}{x_{2}}]\). This completes the proof. **Lemma 6**. : _There exists an upper bound function \(f_{max}(x)\) such that \(G(z,x)\) satisfies_  
\[\mid G(z,x\mid\leq f_{max}(x) \tag{29}\]  
Proof. : \[\mid G(z,x)\mid =\mid-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+2x_{3}-4x _{4}}{x_{2}}]\mid\] \[=\mid zx_{2}cos(x_{1}x_{2}+x_{3}x_{4})+z(0.5x_{1}+2x_{3}-4x_{4})\mid\] \[\leq\mid x_{2}cos(x_{1}x_{2}+x_{3}x_{4})+z(0.5x_{1}+2x_{3}-4x_{4})\mid\] \[\leq\mid x_{2}+0.5x_{1}+2x_{3}-4x_{4}\mid\] \[=f_{max}(x)\]  
This completes the proof. We construct the optimal control problems for a nominal system. \[\dot{x}=\bar{A}x(t)+Bu(t) \tag{30}\]We determine a controller \(u\), which minimizes the following performance index  
\[J(x_{0},u)=\int_{0}^{\infty}[f_{max}^{2}(x)+x^{T}x+u^{T}u]dt \tag{31}\]  
Based on the performance index Function (31), the cost function to the admissible control policy \(u(x)\) is  
\[V(x)=\int_{t}^{\infty}[f_{max}^{2}(x)+x^{T}x+u^{T}u]dt \tag{32}\]  
We define \(\
abla V\) as the gradient of \(V(x)\) with respect to \(x\). Finding differentiation on both sides of (32) with respect to \(t\) yields the following Bellman equation. \[f_{max}^{2}(x)+x^{T}x+u^{T}u+\
abla V^{T}[\bar{A}x+Bu]=0 \tag{33}\]  
Then, we define the following Hamiltonian function. \[\mid G(z,x)\mid =\mid-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+2x_{3}-4x _{4}}{x_{2}}]\mid\]0  
On assuming that the minimum exists and is unique, the optimal control function for the given problem is then obtained as  
\[\mid G(z,x)\mid =\mid-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+2x_{3}-4x _{4}}{x_{2}}]\mid\]1  
On substituting Equation (35) into Equation (33), the HJB equation that satisfies the optimal function \(V_{opt}(x)\) can be obtained as  
\[\mid G(z,x)\mid =\mid-zx_{2}[cos(x_{1}x_{2}+x_{3}x_{4})+\frac{0.5x_{1}+2x_{3}-4x _{4}}{x_{2}}]\mid\]2  
with the initial condition \(V_{opt}(0)=0\). On solving the optimal function \(V_{opt}(x)\) from Equation (36), the solution of the optimal control problem can be obtained. The solution of the robust control problem can then be obtained. The following theorem shows that the optimal control \(u_{opt}=-\frac{1}{2}B^{T}\
abla V_{opt}\) is a robust controller for a nonlinear IPS. Theorem 3.2: _On considering the nominal system (30) with the performance index (31) and assuming that solution \(V_{opt}(x)\) of the HJB Equation (36) exists, the optimal control law (35) can then globally stabilize the IPS (28)._  
Proof: We select \(V_{opt}(x)\) as the Lyapunov function. On considering the performance index function (31), \(V_{opt}(x)\geq 0\) is in evidence, and \(V_{opt}(0)=0\).