# Robust Control of an Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning
## 6 Numerical Simulation Results
### Example 1  
Considering system (11), whose state-space model can be referenced in [34], our objective is to obtain a robust control \(u\) such that it is stable. Based on Lemmas 1-3, the weighting matrix \(M\) is selected as  
\[M=F+I=\begin{bmatrix}1&0&0&0\\ 0&1.81&0&0\\ 0&0&1&0\\ 0&0&0&1\end{bmatrix}\]  
We present the initial stability control law  
\[u_{0}=[1.0900\quad 4.1230\quad-24.8908\quad-6.7726]x\]  
The initial state of the nominal system is selected as \(x_{0}=[0\quad 1\quad 1\quad 1]^{T}\). The time-step size for the collecting system status and input information is set as 0.01 s. Algorithm 1 converges after six iterations, and the \(S_{d}\) matrix and control gain \(K_{d}\) converge to the following optimal solutions:  
\[S_{d}=\begin{bmatrix}2.4465&2.0822&-6.2489&-1.2066\\ 2.0822&4.3346&-14.0702&-2.7082\\ -6.2489&-14.0702&100.8262&18.9646\\ -1.2066&-2.7082&18.9646&3.6646\end{bmatrix} \tag{43}\]  
and  
\[K_{d}=[1.0044\quad 2.5538\quad-32.2652\quad-6.2440] \tag{44}\]  
There are 10 independent numerical samples in the matrix \(S_{d}\). These 10 numerical samples are collected in each iteration to address the least squares problem. The evolution of the control signal \(u\) is presented in Figure 5. Figure 6 illustrates the iterative convergence process of the \(S\) matrix, where \(S(i,j)\) represents the element lying at the intersection of the \(i\)-th row and the \(j\)-th column in the symmetric matrix \(S\), where \(i=1,2,3,4\), \(j=1,2,3,4\).  
Figure 5: Control signal \(u\) of the linearized system.  
The ARE (18) is solved directly by Matlab, the \(S\) matrix and optimal feedback \(K\) are obtained as follows.  
\[S=\begin{bmatrix}2.4455&2.0802&-6.2326&-1.2039\\ 2.0802&4.3307&-14.0381&-2.7032\\ -6.2326&-14.0381&100.5677&18.9228\\ -1.2039&-2.7032&18.9228&3.6579\end{bmatrix} \tag{45}\]  
\[K=[1.000\quad 2.5455\quad-32.1952\quad-6.2327] \tag{46}\]  
As is apparent, the results from the two methods are very similar. Figure 7 presents the closed-loop trajectory of system (11). Figure 7a-d represent the closed-loop system trajectories for uncertain parameters \(z=0.1,0.4,0.7,1.0\), respectively. It is easy to observe that the system is stable, which means that the controller is valid  
Table 2 displays the respective partial eigenvalues of the system (11) with \(u=Kx\) under varying values of \(z\). From Table 2, we can observe that the eigenvalues of the closed-loop system all have negative real parts. Thus, the uncertain linear system (11) with robust control \(u=Kx\) is asymptotically stable for all \(0\leq z\leq 1\).  
\begin{table}
\begin{tabular}{c c c c c} \hline \(z\) & \(\lambda\)**1** & \(\lambda\)**2** & \(\lambda\)**3** & \(\lambda\)**4** \\ \hline
0.1 & \(-6.60\) & \(-4.23\) & \(-0.85+0.32\)i & \(-0.85-0.32\)i \\
0.2 & \(-6.73\) & \(-4.33\) & \(-0.78+0.43\)i & \(-0.78-0.43\)i \\
0.3 & \(-6.86\) & \(-4.41\) & \(-0.71+0.50\)i & \(-0.71-0.50\)i \\
0.4 & \(-7.00\) & \(-4.48\) & \(-0.65+0.55\)i & \(-0.65-0.55\)i \\
0.5 & \(-7.14\) & \(-4.54\) & \(-0.60+0.59\)i & \(-0.60-0.59\)i \\
0.6 & \(-7.28\) & \(-4.59\) & \(-0.55+0.62\)i & \(-0.55-0.62\)i \\
0.7 & \(-7.42\) & \(-4.63\) & \(-0.50+0.65\)i & \(-0.50-0.65\)i \\
0.8 & \(-7.56\) & \(-4.67\) & \(-0.46+0.67\)i & \(-0.46-0.67\)i \\
0.9 & \(-7.70\) & \(-4.70\) & \(-0.42+0.68\)i & \(-0.42-0.68\)i \\
1.0 & \(-7.84\) & \(-4.73\) & \(-0.38+0.69\)i & \(-0.38-0.69\)i \\ \hline \end{tabular}
\end{table}
Table 2: Characteristic root of system (11) when z takes different values.  
Figure 6: S-matrix iterative process of the linearized system.