## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Path-Following and Obstacle Avoidance Controller Based on DDPG  
The algorithm performance and convergence speed of reinforcement learning are highly dependent on the correctness of the state space, action space, and reward. In the process of path-following and obstacle avoidance control, the solution of using two agents to achieve control tasks is obviously complicated and inconvenient [19]. In order to solve these issues, this paper unifies the two types of control by designing the state space and reward based on the specific requirements of the two types of control. This can ensure that the wheeled mobile robot achieves effective obstacle avoidance in the process of path-following. Figure 2: Actor–critic architecture of path-following and obstacle avoidance control. In this paper, the primary goal is to minimize the errors expressed in the above goal of path-following, and the state space \(S\) is expressed as:  
\[S_{pf}{=\ \{\ \tilde{x}_{e}\ \ \ \tilde{y}_{e}\ \ \ \tilde{\theta}_{e}\ \ \ v\ \ \ w\ \ \}} \tag{12}\]  
Considering obstacle avoidance control, the state space \(S\) can be redesigned as:  
\[S{=\ \{\ \tilde{x}_{e}\ \ \tilde{y}_{e}\ \ \tilde{\theta}_{e}\ \ \ \tilde{d}_{obs}\ \ \ \theta_{obs\_e}\ \}} \tag{13}\]  
where \(d_{obs}\) and \(\theta_{obs\_e}\) are the parameters of state for avoiding collisions, \(d_{obs}\) represents the distance between the obstacle center \(d_{obs}\) and the center \(O_{t}\) of the robot, and \(t\) is the current time step, as shown in Figure 3. When the robot is far from the obstacle during path-following, the current control of the robot is considered to be relatively safe. The parameter is only considered when the robot goes to the safe region setting for avoiding obstacles. Therefore, this paper also divides the obstacle avoidance region for the obstacle, as shown in Figures 3 and 4, and the parameters \(d_{obs}\) and \(\theta_{obs\_e}\) are defined as follows:  
\[d_{obs}=\left\{ \begin{array}{cc}d_{obs}\ \ \ ifd_{obs}<r_{e}\\ 0\ \ \ ifd_{obs}\geq r_{e}\end{array} \right. \tag{14}\]  
\[\theta_{obs\_e}=\left\{ \begin{array}{cc}|\theta_{obs\_t}-\tilde{\theta}| \ \ \ ifd_{obs}<\theta_{s}\\ 0\ \ \ ifd_{obs}\geq\theta_{s}\end{array} \right. \tag{15}\]  
where \(r_{e}\) is the radius of the minimum obstacle avoidance control area. Figure 4: Schematic diagram of the components of the obstacle avoidance region. Figure 3: Obstacle avoidance illustration for a NWMR for path-following. The above is considered for the case of a single obstacle; it is not applicable to an environment containing multiple obstacles. In the case of multiple obstacles, the minimum representative value technique is proposed, considering several \(d_{obs}\) and \(\theta_{obs\_e}\), where the minimum values for both parameters are chosen as the elements of state for avoiding more collisions. The minimum representative value can be expressed as:  
\[d_{obs}=\min(d_{obs}),i\in[1,k] \tag{16}\]  
\[\theta_{obs\_e}=\theta_{obs\_e_i},i\in[1,k] \tag{17}\]  
where \(k\) is the number of obstacles. Since there are multiple obstacles corresponding to different states, for reward setting considering all obstacles, penalty rewards are considered for them all. The minimum representative approach is then able to ensure that the agent maintains the behavior of obstacle avoidance during training and that it is feasible to achieve path-following and obstacle avoidance control for NWMRs. In RL, the agent can learn to adjust its strategy according to the reward so that it can avoid multiple obstacles in path-following. Compared with the environmental state set according to the number of obstacles [28], this method can reduce the dimensionality of the state, thus achieving the effect of obstacle avoidance while reducing the computational burden and saving computing resources.