## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Path-Following and Obstacle Avoidance Controller Based on DDPG
The comparison of inputs between the two algorithms is given in Figure 4(c). Figure 4(d) presents the reward changes in every step of the final episode in the training, and Figure 4(e) shows the average score per 100 episodes in the whole training process. It can be shown that the trend of the reward score is consistent with the path-following effect, by combining the results in Table 2, which represents the four stages of the training process, chosen as 100, 200, 300, and 400 episodes. It can be shown that the agent tended to show worse performance results for following the path at the initial stage of the training, which is irrelevant to the initial purpose of path-following. As the training continued, the path-following performance tended to develop, resulting in a decrease in the cross-track error and the angular error, with scores climbing steadily. Finally, the proposed algorithm results in better performance than MPC considering the comparison of the longitudinal and cross errors. According to the results of the path-following simulation experiments, the proposed control strategy performs more robustly and has more accurate characteristics. It is capable of moving close to the waypoints of the path with different starting points, whereas the comparison algorithm needs to adjust its parameters to meet the requirements of the initial point. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer** & **Actor/Target Actor** & **Critic/Target Critic** & **Activation Function** \\ \hline Input layer & 5 & 5 & ReLU \\
1 st hidden layer & 400 & 400 & ReLU \\
2 nd hidden layer & 300 & 300 & ReLU \\
3 rd hidden layer & 300 & 300 & ReLU \\ Output layer & 2 & 1 & Tanh \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters of networks. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Stage** & **MSE for Longitudinal Errors** & **MSE for Cross Errors** \\ \hline Stage 1 (proposed) & 16.5702 & 2.111 \\ Stage 2 (proposed) & 0.6008 & 0.3512 \\ Stage 3 (proposed) & 0.0005 & 0.0705 \\ Stage 4 (proposed) & 0.0069 & 0.0031 \\ MPC & 1.5378 & 0.1930 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Path-following controller performance at different training stages. ### The Performance of Path-Following with Collision Avoidance

In this experiment, the agent was trained for 1000 episodes, with a total of 600,000 training steps in the simulations. Two obstacles were chosen in order to perform a validation study of the proposed algorithm for path-following and obstacle avoidance. The learning rate \(\beta\) was reduced to 0.002 for the critic network, and the discount factor \(\gamma\) was changed to 0.98. The number of obstacles \(k\) was set at two, with centers at [10, 0] and [47, \(-10\)] around the waypoints of the reference path, and the common parameters \(r_{obs}\), \(r_{c}\), \(r_{e}\) were set to 3.5 m, 5.5 m, and 7.5 m, respectively. The parameters \(\phi_{7z}\) and \(\phi_{pz}\) were set to 100 and 10, as scores for the reward and penalty, respectively. For each obstacle, the severe penalty \(\psi\) was 100 in every episode, and the parameter \(\theta_{s}\) was set at \(\pi/4\). Figure 5: Comparisons of proposed method and MPC for performance of path-following control. (**a**) Comparison of paths for following. (**b**) Errors in path-following. (**c**) Inputs for path-following. (**d**) Scoring of each step with proposed method. (**e**) Average scoring for each episode with proposed method. ## 5 Conclusions

In this paper, a deep-reinforcement-learning-based controller was proposed for path-following for nonholonomic wheeled mobile robots(NWMRs). The deep deterministic policy gradient (DDPG) algorithm was utilized to establish a control law for linear and steering velocities, and the learning-based control policy was trained using repeated path-following simulations.