## 4 Numerical Results and Discussion
### Physical Properties' Parameter Study  
The hyperparameters for the neural networks learning and DDPG algorithm employed for the training of the agent are mentioned in Table 1, and the corresponding learning curves are shown in (a) and (b) of Figure 5.  
The optimal policy is found at Episode 435 considering the criteria \(r_{avg}>-80\), and \(r_{avg}\) is the average cumulative reward in a window of five episodes. Table 2 reports the properties of the environments considered, where no dry friction is present.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Meaning** & **Value** \\ \hline  & Actor learning rate & \(5\times 10^{-4}\) \\  & Critic learning rate & \(10^{-3}\) \\  & Gradient threshold & \(1.00\) \\ \(D\) & Experience buffer length & \(10^{6}\) \\ \(d\) & Replay buffer size & \(128.00\) \\ \(\tau\) & Polyak averaging factor & \(10^{-3}\) \\ \(\gamma\) & Value function discount factor & \(0.99\) \\ \(t_{s}\) & Sampling time & \(0.02\) s \\ \hline \hline \end{tabular}
\end{table}
Table 1: Neural networks and Deep Deterministic Policy Gradient (DDPG) hyperparameters for the training.  
Figure 5: Learning curves of the agent. (**a**) Cumulative reward per episode of training. (**b**) Average cumulative reward in a moving window including five episodes.  
For the purposes of the present work, the swing-up time is defined according to the following formula:  
\[\begin{array}{ll}\text{if}\quad\left|\theta\right|<A_{l}\quad\text{and}\quad \left|\theta\right|\geq B_{l}\quad\text{and}\quad\left|\dot{\theta}\right|\leq C _{l}\quad\text{and}\quad\left|\dot{x}\right|\leq D_{l}\\ T_{su}=t\\ \text{else}\\ T_{su}=0\\ \text{end} \tag{16}\]  
where \(t\) is the time variable, \(T_{su}\) is the swing-up time, \(x\) is the cart linear displacement, \(\theta\) is the pendulum angular displacement, \(\dot{x}\) is the cart linear velocity, \(\dot{\theta}\) is the pendulum angular velocity, while \(A_{l}=5\) (deg), \(B_{l}=0\) (deg), \(C_{l}=0.1\) (rad/s), and \(D_{l}=0.5\) (m/s) are limit constants.  
The training is carried out in the environment of Case 1, producing the agent \(A\). The remaining cases report the performance of agent \(A\) in each defined environment. The magnitudes for the physical properties are the extreme values for which agent \(A\) manages to perform the swing-up under the conditions mentioned in Equation (15). In particular, Cases 2 to 7 report individual variations of the three physical parameters \(m_{c}\), \(m_{p}\), and \(L\) where, based on the cumulative reward obtained during the task, it is found that the increase in \(m_{c}\) has the greatest impact on the agent performance. An interesting effect is observed in Cases 4 and 5, where the respective 90% increase and decrease in \(m_{p}\) yield close cumulative rewards. Figures 6 and 7 show the state response of agent \(A\) in Cases 1, 4, and 5.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Case** & \(m_{c}\)**(kg)** & \(m_{p}\)**(kg)** & \(L\) **(m)** & \(T_{su}\)**(s)** & \(\boldsymbol{\sum r}\) \\ \hline
1 & 1.00 & 1.00 & 0.10 & 0.98 & \(-71.55\) \\
2 & 0.06 & 1.00 & 0.10 & 1.54 & \(-233.93\) \\
3 & 2.00 & 1.00 & 0.10 & 5.1 & \(-269.89\) \\
4 & 1.00 & 0.10 & 0.10 & 1.70 & \(-204.93\) \\
5 & 1.00 & 1.90 & 0.1 & 3.30 & \(-205.93\) \\
6 & 1.00 & 1.00 & 0.07 & 1.70 & \(-262.46\) \\
7 & 1.00 & 1.00 & 0.19 & 3.42 & \(-197.49\) \\
8 & 0.65 & 0.65 & 0.06 & 1.44 & \(-226.57\) \\
9 & 1.25 & 1.25 & 0.12 & 1.28 & \(-201.23\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Cases of the sensitivity analysis of agent \(A\) without the presence of dry friction (\(\mu=0\)).  
Figure 6: Cart position and velocity during the task for three different values of \(m_{p}\). (**a**) Cart displacement during the swing-up task. (**b**) Cart velocity during the swing-up tasks.  
This is due to the oscillating pattern of \(F_{c}\) in Case 8 shown in (a) of Figure 10. Therefore, the agent performance is more susceptible to joint equal inertial decreases.