# Velocity Control of a Multi-Motion Mode Spherical Probe Robot Based on Reinforcement Learning
## 3 Controller Design
### Adaptive PID Controller Based on Reinforcement Learning  
The structure of the adaptive PID controller based on reinforcement learning in this paper is shown in Figure 4. It is possible to divide the entire controller into two parts. One part is the PID controller, which directly controls the motor torques of the MMSPR and realizes velocity adjustments. The other part of the controller is reinforcement learning, which outputs PID parameters for adaptive parameter adjustment. The PID controller used in this paper is the position PID controller, whose mathematical expression can be formulated as:  
\[u(t)=K_{p}e(t)+K_{i}\sum_{n=0}^{t}e(n)+K_{d}(e(t)-e(t-1)) \tag{7}\]  
For reinforcement learning, choosing appropriate state space, action space, and reward function is essential. To solve the velocity control problem, we select the dynamic characteristics of the MMSPR as observations. Specifically, the rotation angles and angular velocities of the MMSPR and the pendulum around the \(X\) and \(Y\) axes are involved. In addition, to facilitate the SAC algorithm establishing a mapping between the MMSPR's velocity errors and the controller parameters more effectively, velocity errors of the MMSPR in \(X\) and \(Y\) directions are added into the state space, namely \(e_{x}\), and \(e_{y}\). As a result, the state space of reinforcement learning is ten dimensions. According to the algorithm architecture proposed in this paper, the actions of reinforcement learning are the parameters of the PID controller, as shown below:  
\[a_{t}=\pi(s_{t})=\left(K_{px},\ K_{py},\ K_{ix},\ K_{iy},\ K_{dx},\ K_{dy}\right) \tag{8}\]  
To ensure the stability of the control system, the action ranges are set to \(K_{ps}=[0,3]\), \(K_{is}=[0,1]\), \(K_{ds}=[0,0.5]\), \(s=\{x,y\}\). Furthermore, since the motor torques are limited in real-world applications, the output ranges of the PID controller are set to be within 10 Nm, \(\tau_{x},\tau_{y}\in(-10\ \text{Nm},10\ \text{Nm})\). An important part of reinforcement learning is designing an appropriate reward function, which determines whether the agent can converge to the optimal solution. In this paper, our purpose is to control the MMSPR's rolling velocities, that is, to reduce the errors between the actual and desired velocity of the MMSPR in \(X\) and \(Y\) directions. The velocity errors \(e_{x}\) and \(e_{y}\) are formulated as:  
\[e_{x}=\dot{x}_{1}\times R-v_{xd},e_{y}=\dot{x}_{3}\times R-v_{yd} \tag{9}\]  
where \(\dot{x}_{1}\) and \(\dot{x}_{3}\) are the rolling angular velocities of the MMSPR in the \(X\) and \(Y\) directions, respectively. Therefore, the smaller \(e_{x}\) and \(e_{y}\) are, the larger the reward value of the agent will be. In order to obtain smaller static errors when the system is stable, \(e_{x}\) and \(e_{y}\) in the reward function are multiplied by a certain ratio. When the velocity errors remain large, the reward should change sharply to expedite the convergence. However, it is expected to be stable when the errors are small to ensure the agent outputs actions that maintain small errors. Therefore, _tanh_ is used to make the 2-norm of the velocity error nonlinear, and the reward function is defined as:  
\[ \begin{array}{l}reward1=1-tanh(10\times|e_{1}|)\\ reward2=1-tanh(10\times|e_{2}|)\\ reward=0.5\times reward1+0.5\times reward2\end{array}  \tag{10}\]  
Due to the utilization of the _tanh_ function, the velocity errors are difficult to reduce after it is below a threshold. Switching control is introduced into our controller to obtain smaller errors further. After the errors fall below the threshold value \(\alpha\), the controller will switch from the variable-parameter PID controller adjusted by reinforcement learning to the PID controller with fixed parameters. Figure 4: The structure of the designed controller. The red part represents the reinforcement learning module, which adjusts the parameters of the PID controller. The blue part represents the controller module, which outputs the motor torques.