## 3 Dynamic Model Description and Numerical Experiments' Setup
This is because at zero velocity, the friction force is null. Therefore, it cannot reproduce stiction, the pre-sliding effect, or the Stribeck effect. Although there is a partial loss of information, nonlinear viscous friction laws such as the smooth Coulomb model are widespread in engineering applications devoted to implementing a quick but realistic analysis of articulated mechanical systems. More complex and accurate friction models can exhibit these effects, but require multiple additional physical parameters. In this work, given that the main interest is to perform a sensitivity analysis of the proposed control system, a smooth Coulomb friction model will be considered. Therefore, the dry friction force is given by:  
\[F_{f}=-\mu\left|N\right|\tanh\left(\hat{x}\right) \tag{10}\]  
where \(\mu\) is the dry friction coefficient, \(N\) is the instant normal force produced by the contact between the cart and the road, whereas the hyperbolic tangent is employed as an artificial function of the cart velocity necessary during the numerical simulations for smoothing the dry friction force described by the Coulomb law of friction. Figure 3 shows the Simulink implementation of the RL workflow, where the vector of observations \(\mathbf{s}\) is defined as:  
\[\mathbf{s}=\left[ \begin{array}{cccccc}\mathbf{s}_{\theta}&\mathbf{c}_{\theta }&\hat{\theta}&x&\hat{x}\end{array} \right]^{T} \tag{11}\]  
where the abbreviations \(\sin(\theta)=\mathbf{s}_{\theta}\) and \(\cos(\theta)=\mathbf{c}_{\theta}\) are used. Figure 3: Simulink implementation of the RL workflow. The instant reward function for the environment is defined as follows:  
\[r= \begin{cases}&r_{1},\quad|x|\leq x_{\text{lim}}\\ &\\ &r_{2},\quad|x|>x_{\text{lim}}\end{cases}  \tag{12}\]  
where \(x_{lim}\) is the maximum displacement allowed for the cart during the swing-up task, while \(r_{1}\) and \(r_{2}\) are defined as:  
\[r_{1}=\left(A_{r}|x|^{n}+B_{r}|\theta|^{n}+C_{r}|F_{c}|^{n}\right)D_{r} \tag{13}\]  
and:  
\[r_{2}=r_{1}+E_{r} \tag{14}\]  
where the constant parameters are respectively set as \(A_{r}=10^{-2}\), \(B_{r}=10^{-1}\), \(C_{r}=5\), \(D_{r}=-10^{-2}\), \(E_{r}=-10^{2}\), and \(n=2\). The shape of the reward function allows for penalizing the magnitude of \(x\), \(\theta\), and \(F_{c}\). Therefore, the optimal policy will be that in which the system manages to perform the swing-up with the lowest force possible, and the cart is in the initial position at the end of the task. On the other hand, the actor \(\mathbf{\pi}_{\varphi}\) and critic \(\mathbf{Q}_{\theta}\) neural networks are multilayer perceptrons with architectures as respectively shown in (a) and (b) of Figure 4.  
The training process of the agent is done employing the DDPG algorithm under the following restrictions:  
\[F_{c}\leq F_{\text{max}},\quad|x|\ \leq x_{\text{lim}},\quad t\leq t_{f} \tag{15}\]  
where \(F_{\text{max}}\) is the maximum force allowed to the controller, set to 20 (N), \(x_{\text{lim}}\) is the maximum lateral displacement of the cart, equal to 1 (m), and \(t_{f}\) is the maximum duration of the task, set to 25 (s). Then, the sensitivity analysis of the controller consists of evaluating its performance under modified environments featuring variations of the physical properties of the system. Finally, to expand the optimal behavior space for the agent, a post-training of its structural parameters is performed in a modified environment.