## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Path-Following and Obstacle Avoidance Controller Based on DDPG
\right]\);
3:for all\(t\in[1,T]\)do
4: Initialize a random noise \(\mathcal{N}_{t}\) for the deterministic strategy;
5: Randomly select an action as a control input based on the current environment strategy and exploration noise \(u_{t}=\mu(s_{t}|\theta^{u})+\mathcal{N}_{t}\);
6: Execute \(u\), then obtain reward \(re_{t}\) and new state \(s_{t+1}\);
7: Put \(transition(s_{t},u_{t},re_{t+1},s_{t+1})\) into experience replay buffer \(D\);
8:if number of \(transition>\) Memory then
9: Extract randomly a batch of transitions from \(D\);
10: Update actor network and critic network, (9) (11);
11: Update target network for stable training as:
12:\(\theta^{Q^{\prime}}=\tau\theta^{Q}+(1-\tau)\tau\theta^{Q^{\prime}}\)
13:\(\theta^{\prime^{\prime}}=\tau\theta^{\prime n}+(1-\tau)\tau\theta^{\prime n^{ \prime}}\)
14:endif
15:endfor
16:endfor
```

**Algorithm 1** Path-Following and Obstacle Avoidance Control Strategy for NWMRs

## 4 Results and Discussions

In order to verify the path-following and obstacle avoidance control strategy proposed in this paper, several sets of simulation experiments were conducted. Firstly, only the path-following was investigated and compared with the model predictive control (MPC) method. Secondly, path-following and obstacle avoidance simulation experiments were conducted, and results were given for validating the effectiveness of the proposed controller performance for multiple obstacle avoidance in training environments. ### Training Setting

In the environment of the simulation, the initial position \([\tilde{x}_{init}\;\;\tilde{y}_{init}\;\;\tilde{\theta}_{init}]\) of the NWMR was randomly selected around the end point of the path, where the initial position can be expressed as:

\[\begin{array}{l}\tilde{x}_{init}=\theta_{p\_end}\pm\Delta e_{1}\\ \tilde{y}_{init}=\rho(\theta_{p\_end})\pm\Delta e_{2}\\ \tilde{\theta}_{init}=\theta_{path\_r\_end}\pm\Delta e_{3} \tag{21}\]

where \(\Delta e_{1},\Delta e_{2},\Delta e_{3}\in[-1,1]\) were used to generate different initial values in each start episode in the environment, and the maximum linear and angular velocities were set as 3 m/s and \(\pi/2\) rad/s, respectively, in the training. In the training simulation, the time step \(\Delta t\) was set to 0.5 s (2 Hz) and the size of the mini-batch to 64. To establish the training networks, the Adam optimizer was used to train both the actor and critic networks. The hyperparameters are shown in Table 1, and the networks were built using the machine learning library **Pytorch**. The learning rate \(\alpha\) was set to 0.001 for the actor network, and \(\beta\) was set to 0.01 for the critic network. The target network transition gain \(\tau\) was selected as 0.01, and the discount factor \(\gamma\) was selected as 0.9. For exploration of the training, the Ornstein-Uhlenbeck exploration method discussed in [34] was used. The sinusoidal path can be parametrized, as investigated in [22], as:

\[y=30\sin(\frac{\pi}{20}x-\frac{\pi}{2}) \tag{22}\]

### Comparison of Path-Following between the Proposed Method and MPC

In this experiment, the agent was trained for 400 episodes, with a total of 240,000 training steps in the simulations. To validate the path-following capabilities of the proposed method, the MPC algorithm [35] was introduced to present its performance for path-following, and a comparison was made between the two methods. Moreover, the effectiveness of the proposed algorithm is further illustrated by showing the following effects at different training stages. Figure 4(a) shows the path-following effect of the proposed algorithm and also adds the MPC algorithm for a comparison of the results. As shown in Figure 4(b), errors for each waypoint of the path are presented, It is shown that the convergence performance of the proposed algorithm is better than that of the MPC algorithm at the turn. It can also be shown that the longitudinal error performance using the proposed algorithm is better than that using the MPC and that the other errors show the same or better performance.