# Robust Control of an Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning
## 4 RL Algorithm for Robust Optimal Control  
In this section, we propose an RL algorithm for solving the robust control problem of an IPS through online PI. According to ARE (18), the following optimal control problem is constructed. For the nominal system,  
\[\dot{x}=A(z_{0})x+Bu\]  
we find a control \(u\), such that the following performance index reaches a minimum. \[J=\int_{t}^{\infty}[x^{T}Mx+u^{T}u]dt \tag{20}\]where \(M=F+I>0\). For any initial time \(t\), the optimal cost can be written as  
\[V[x(t)] =\int_{t}^{\infty}\Bigl{(}x^{T}Mx+u^{T}u\Bigr{)}dt\] \[=\int_{t}^{t+\delta t}\Bigl{(}x^{T}Mx+u^{T}u\Bigr{)}dt+\int_{t+ \delta t}^{\infty}\Bigl{(}x^{T}Mx+u^{T}u\Bigr{)}dt\] \[=\int_{t}^{t+\delta t}\Bigl{(}x^{T}Mx+u^{T}u\Bigr{)}dt+V[x(t+ \delta t)]\]  
From Lyapunov Function (19), we obtain  
\[x(t)^{T}Sx(t)=\int_{t}^{t+\delta t}\Bigl{(}x^{T}Mx+u^{T}u\Bigr{)}dt+x(t+\delta t )^{T}Sx(t+\delta t) \tag{21}\]  
where \(S\) is the solution to ARE (18). We propose the following RL algorithm for solving a robust controller. In Algorithm 1, by providing an initial stabilizing control law, repeated iterations are performed between steps 3 and 4 until convergence. We can then obtain the robust control gain \(K\) of system (11). 1. \(M=F+I\) is computed. 2. An initial stabilization control gain \(K_{0}\) is selected. 3. Policy evaluation: \(S_{i}\) is solved using the equation \(x^{T}(t)S_{i}x(t)=\int_{t}^{t+\delta t}x^{T}(M+K_{i}^{T}K_{i})xdt+x^{T}(t+ \delta t)\). 4. Policy improvement: \(K_{i+1}=-B^{T}S_{i}\). 5. We set \(i=i+1\), and steps 3 and 4 are repeated until \(\|S_{i+1}-S_{i}\|\leq\epsilon\), where \(\epsilon>0\) is a small constant. **Algorithm 1** RL Algorithm for Uncertain Linear IPS  
**Remark 1**. : _Step 3 in Algorithm 1 is the policy evaluation, and step 4 is the policy improvement. Equivalently, the solving of the equation in step 3 is actually solving a least squares problem. In the integral interval, if sufficient data are obtained in the system, the least square method can be used to solve \(S_{i}\)._  
Next, we prove the convergence of Algorithm 1. However, it is necessary to prove the following Lemma first. **Lemma 4**. : _On assuming that the matrix \(A(z_{0})+BK_{i}\) is stable, solving the matrix \(S_{i}\) from step 3 of Algorithm 1 becomes equivalent to solving the following equation._  
\[S(A(z_{0})+BK_{i})+(A(z_{0})+BK_{i})^{T}S+M+K_{i}^{T}K_{i}=0 \tag{22}\]  
Proof. : We rewrite the equation of step 3 in Algorithm 1 as follows  
\[\lim_{\delta t\to 0}\frac{x^{T}(t+\delta t)S_{i}x(t+\delta t)-x^{T}(t)S_{i}x(t)}{ \delta t}+\lim_{\delta t\to 0}\frac{\int_{t}^{t+\delta t}x^{T}(M+K_{i}^{T}K_{i})xdt}{ \delta t}=0 \tag{23}\]  
According to the definition of the derivative, it can be observed that the first term of Equation (23) is the derivative of \(x^{T}(t)S_{i}x(t)\) with respect to time \(t\). We thus obtain  
\[\frac{d(x^{T}(t)S_{i}x(t))}{dt}+\lim_{\delta t\to 0}\frac{d}{\delta t }\int_{t}^{t+\delta t}x^{T}(M+K_{i}^{T}K_{i})xdt=0 \tag{24}\]Further re-arranging Equation (24) yields  
\[x^{T}[S(A(z_{0})+BK_{i})+(A(z_{0})+BK_{i})^{T}S+M+K_{i}^{T}K_{i}]x=0 \tag{25}\]  
which means that (22) is established. Next, we reverse the process. Along the stable system \(\dot{x}=(A+BK_{i})x\), the time derivative of the Lyapunov function \(V_{i}(x)=x^{T}S_{i}x\) is calculated. We can then obtain  
\[J=\int_{t}^{\infty}[x^{T}Mx+u^{T}u]dt \tag{20}\]0  
On integrating both sides of the equation (26) in the interval \((t,\delta t)\), we obtain  
\[J=\int_{t}^{\infty}[x^{T}Mx+u^{T}u]dt \tag{20}\]1  
This completes the proof. According to the existing conclusions [36], iterative relations (22) and step 3 of Algorithm 1 converge to form the solution of ARE (18). **Remark 2**. : _The behavior of the control is evaluated using a cost function, which is similar to the reward in RL. The agent corresponds to the controller in optimal control, and the control process corresponds to the environmental model in RL.