# Robust Control of an Inverted Pendulum System Based on Policy Iteration in Reinforcement Learning
## 2 Model Formulation
### Modeling of Inverted Pendulum System  
The inverted pendulum experimental device comprises a pendulum and a trolley [33]. Its structure is presented in Figure 1. The encoder is a photoelectric rotary one, and the motor is a direct-current servo motor. For a detailed information on the experimental platform, see [34]. Moreover, its simplified physical model is presented in Figure 2, which mainly includes the pendulum and trolley. In Figure 2, owing to the interaction between the trolley and pendulum, the trolley is subjected to a force \(F_{3}\) from the pendulum, which acts in the lower left direction. Furthermore, the pendulum is subjected to a force \(F_{4}\) from the trolley, which acts in the upper right direction. In addition, the pendulum and trolley are also subjected to other forces, as shown in Figures 3 and 4, respectively. The trolley is driven by a motor to perform horizontal movements on the guide rail. In Figure 3, the trolley is subjected to the force \(F_{1}\) from the motor and gravity. \(F_{2}\) represents the resistance between the trolley and guide rail. Furthermore, \(N_{1}\) and \(P_{1}\) are the two components of force \(F_{3}\). In Figure 4, the pendulum is subjected to gravity \(G=m_{1}g\), and \(N_{2}\) and \(P_{2}\) are the two components of force \(F_{4}\). Figure 1: Inverted pendulum system diagram. To facilitate subsequent calculations, we define the parameter of the first-order IPS, as shown in Table 1. The time parameter symbol \((t)\) is omitted, which indicates that \(x\) represents \(x(t)\). In Figure 3, according to [PERSON]'s second Law, the trolley satisfies the following equation in the horizontal direction:  
\[F_{1}-N_{1}-F_{2}=m_{2}\ddot{x} \tag{1}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Unit** & **Significance** \\ \hline \(m_{1}\) & kg & Mass of the trolley \\ \(m_{2}\) & kg & Mass of the pendulum \\ \(L\) & m & Half the length of the pendulum \\ \(z\) & N/m/s & Friction coefficient between the trolley and guide rail \\ \(x\) & m & Displacement of the trolley \\ \(\theta\) & rad & Angle from the upright position \\ \(l\) & kg\(\cdot\)m\({}^{2}\) & Moment of inertia of pendulum \\ \hline \hline \end{tabular}
\end{table} 
Table 1: IPS parameter symbols. Figure 4: Force analysis of the pendulum. Figure 3: Force analysis of the trolley. Figure 2: First-order inverted pendulum physical model. We assume that the resistance is proportional to the speed of the trolley. Therefore, \(F_{2}=z\dot{x}\), \(z\) is the proportional coefficient. Moreover, in Figure 4, the pendulum satisfies the following equation in the horizontal direction:  
\[N_{2}=m_{1}\frac{d^{2}}{dt^{2}}(x-L\text{sin}\theta)=m_{1}\dot{x}+m_{1}L\dot{ \theta}^{2}\text{sin}\theta-m_{1}L\ddot{\theta}\text{cos}\theta \tag{2}\]  
Considering that \(N_{1}=N_{2}\) in Figure 2, and on substituting (2) into (1), we obtain  
\[F_{1}=(m_{1}+m_{2})\dot{x}+z\dot{x}+m_{1}L\dot{\theta}^{2}\text{sin}\theta-m_{1 }L\ddot{\theta}\text{cos}\theta \tag{3}\]  
Next, in Figure 4, we analyze the resultant force in the vertical direction of the pendulum, and the following equation can be obtained. \[P_{2}-m_{1}g=m_{1}\frac{d^{2}}{dt^{2}}(\text{cos}\theta)=-m_{1}L\dot{\theta}^{ 2}\text{cos}\theta-m_{1}L\ddot{\theta}\text{sin}\theta \tag{4}\]  
The component force of \(N_{2}\) in the direction perpendicular to the pendulum is  
\[N_{2}\text{cos}\theta=m_{1}\frac{d^{2}}{dt^{2}}(x-L\text{sin}\theta)\text{cos }\theta=m_{1}\dot{x}\text{cos}\theta+m_{1}L\dot{\theta}^{2}\text{sin}\theta \cos\theta-m_{1}L\ddot{\theta}\text{cos}^{2}\theta \tag{5}\]  
Based on the torque balance, we can obtain the following equation  
\[I\ddot{\theta}=P_{2}L\text{sin}\theta+N_{2}L\text{cos}\theta \tag{6}\]  
where \(I\) is the moment of inertia of the pendulum. On substituting Equations (4) and (5) into Equation (6),  
\[(I+m_{1}L^{2})\ddot{\theta}-m_{1}gL\text{sin}\theta=m_{1}L\ddot{x}\text{cos}\theta \tag{7}\]  
Thus far, Equations (3) and (7) constitute the dynamic model of the IPS.