# Velocity Control of a Multi-Motion Mode Spherical Probe Robot Based on Reinforcement Learning
## 1 Introduction
This controller combines hierarchical sliding mode control (HSMC) with adaptive laws to address the spherical robot's balance control and velocity control during motion. The PID algorithm is extensively used in industrial applications, and its parameter tuning is critical to achieving effective control. However, it becomes challenging to obtain the desired control effect when dealing with nonlinear and strongly coupled systems by merely setting fixed controller parameters. Furthermore, the controller's performance may deteriorate if the system's parameters exceed the controller's range [27], indicating poor adaptability and generalization. Reinforcement learning is a method to find the optimal solution from the interaction between the agent and the environment [28]. Combining it with the traditional PID algorithm can improve the generalization and adaptability of a PID controller. [PERSON] proposed a PID controller based on reinforcement learning to enhance the trajectory tracking performance in a quadrotor [29]. Combining the proximal policy optimization (PPO) algorithm with the traditional PID controller, this method has improved response time, reduced overshoot, minimized control errors, enhanced stability, and strong anti-interference capabilities compared with the traditional PID controller. [PERSON] developed a Q-learning PID controller for trajectory tracking in mobile robots [30]. This controller adjusts the output of the PID controller with the output from Q-learning, achieving precise trajectory tracking. [PERSON] introduced an adaptive PID controller based on radial basis function (RBF) networks for complex nonlinear systems [31]. The controller utilizes the RBF network to adaptively adjust the PID controller's parameters based on the system's current state errors, which achieves stable control without system modeling in advance. Park adjusted proportion-integration (PI) parameters online through reinforcement learning so that the vehicle could follow the target more accurately [32]. According to the above research, the combination of reinforcement learning and the traditional PID controller improves the performance of control systems, which is useful for practical engineering applications. Due to the nonlinear coupling and parameter uncertainty in the dynamic model of the MMSPR, it is complex to adjust the controller parameters manually. Furthermore, controllers with fixed parameters have constrained adaptability. Considering the requirements of deep space exploration tasks and the integrated optical functional modules, this article researches the control method for the MMSPR. The aim is to address the challenges concerning the robot's motion stability during the rolling process and rapid braking after crossing obstacles. Moreover, the feasibility and effectiveness of the controller proposed in this paper are substantiated through simulation. The structure of this paper is organized as follows. Section 2 introduces the mechanical design and dynamic model of the spherical robot. Section 3 describes the design of the adaptive PID controller based on reinforcement learning. Section 4 shows and discusses the simulation results in motion stability and rapid braking situations. Finally, Section 5 provides conclusions and future research prospects.