# Research on the Deep Deterministic Policy Algorithm Based on the First-Order Inverted Pendulum
## 4 Experimental Results and Analysis
### Simulation Environment  
The simulation environment was mainly conducted in the Matlab/Simulink environment, using the Simscape Multibody Toolbox and Reinforcement Learning Toolbox provided in the development environment for experimental simulation. The Simscape Multibody Toolbox provides a multi-body simulation environment for mechanical systems, which can use modules such as solids, joints, constraints, and force elements to model multi-body systems and solve the motion equations of complete mechanical systems. At the same time, 3D animations are generated to visualize the motion effects. In this paper, the slider cart of the inverted pendulum is connected to the pendulum rod through joint structure and coordinate transformation to form a complete first-order inverted pendulum model. The key parameters are shown in Table 1.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Pendulum mass (kg) & 0.2 \\ Cart mass (kg) & 0.46 \\ Length from centroid to hinge point (m) & 0.25 \\ frictional coefficient & 0.08 \\ Gravitational acceleration (m \(\times\) s\({}^{2}\)) & 9.81 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Inverted Pendulum Parameters.  
Figure 5: Improved DDPG flowchart.  
The Reinforcement Learning Toolbox is considered as a proxy agent for the DDPG algorithm, connecting the entire network. This module is connected to the corresponding environment and set reward functions. The reward \(r_{t}\) is defined by Equation (20):  
\[r_{t}=-0.1(5\theta_{t}^{2}+x_{t}^{2}+0.05u_{t-1}^{2})-100B \tag{20}\]  
where \(\theta_{t}\) is the angle displacement from the upright position of the pole. \(x_{t}\) is the displacement from the center position of the cart. \(u_{t-1}\) is the control effort from the previous time step. \(B\) is a flag (1 or 0) that indicates whether the cart is out of bounds. The coefficient parameter is applied for standardization, and the stop value is \(-400\). After training, the Agent module can act as a controller to input the system and receive feedback adjustment inputs. The experimental environment is shown in Figure 6.  
Combining mature engineering test routines, the PID controller selects a displacement controller with PID parameters of \(\mathrm{K_{p}}=16\), \(\mathrm{K_{i}}=5\), and \(\mathrm{K_{d}}=3\) and angular displacement controller with PID parameters of \(\mathrm{K_{p}}=125\), \(\mathrm{K_{i}}=20\), and \(\mathrm{K_{d}}=6\). The hyperparameter settings selected in DDPG training are shown in Table 2.  
Before the simulation, the initial state of the entire inverted pendulum has a greater impact on the control results of the nonlinear system. Considering the actual physical significance of an inverted pendulum, the input quantity needs to be limited. The range of motion is set to [\(-3.5\),\(3.5\)] m. The starting position of the DDPG controller is set at the center of the guide rail, and the swing rod is in a natural drooping state. The entire simulation process achieves the transformation of the swing rod from a natural drooping state to a vertical upward state and maintains balance within the limited range of motion. The PID controller is initially in a vertical upward state and maintains balance during the simulation process.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Critic network learning rate & 0.001 \\ Actor network learning rate & 0.0005 \\ Critic network structure & (\(5\times 128\times 200+1\times 200)\times 1\) \\ Actor network structure & \(5\times 128\times 200\times 1\) \\ Discount factor & 0.995 \\ Target update frequency & 10 \\ Update interval & 100 \\ Soft update parameters & 0.001 \\ Activation function & ReLU \\ \hline \hline \end{tabular}
\end{table}
Table 2: DDPG Training Hyperparameters.  
Figure 6: Experimental environment.