## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Path-Following and Obstacle Avoidance Controller Based on DDPG
The path-following results demonstrated the effectiveness of the proposed method, and the comparisons showed that our method had better efficiency and more robust performance than the MPC method for path-following control without collisions. For research on path-following and obstacle avoidance control, a new approach was proposed to deal with redesigning the state and reward in RL. Moreover, minimum value techniques for the state were given for the path-following and obstacle avoidance controller, and the results showed the feasibility of solving the multiple obstacles environment problem during path-following for the control of NWMRs. Conceptualization, X.C. and S.Z. ; methodology, S.Z. ; software, S.Z. ; validation, X.C., S.C. and J.Z. ; formal analysis, S.C. and J.Z. ; investigation, X.C. and S.Z. ; resources, X.C. ; data curation, S.Z. ; writing--original draft preparation, S.Z. ; writing--review and editing, X.C. ; visualization, S.Z. ; supervision, Q.X. ; project administration, Q.X. ; funding acquisition, X.C. All authors have read and agreed to the published version of the manuscript. This research was funded by the Key-Area Research and Development Program of Guangdong Province, grant number 2019B090918004. Not applicable. The authors declare no conflict of interest. The following abbreviations are used in this manuscript: \begin{tabular}{l l} NWMR & Nonholonomic Wheeled Mobile Robot \\ RL & Reinforcement Learning \\ DDPG & Deep Deterministic Policy Gradient \\ DPG & Deterministic Policy Gradient \\ \end{tabular} 

Figure 6: Performances of the proposed method for path-following and obstacle avoidance control. (**a**) Comparison of paths for following. (**b**) Errors in path-following. (**c**) Inputs for path-following. (**d**) Scoring for each step with proposed method. (**e**) Average scoring for each episode with proposed method. ## References

* (1) [PERSON]; [PERSON]; [PERSON] Model predictive path-following for constrained nonlinear systems. In Proceedings of the 48 th IEEE Conference on Decision and Control (CDC) Held Jointly with 2009 28 th Chinese Control Conference, Shanghai, China, 15-18 December 2009; pp. 8642-8647. * (2) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Path-following control of Mecanum-wheels omnidirectional mobile robots using nonsingular terminal sliding mode. _Mech. Syst. Signal Process._**2021**, _147_, 107128. [CrossRef]
* (3) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Path Following of Wheeled Mobile Robots Using Online-Optimization-Based Guidance Vector Field. _IEEE/ASME Trans. Mechatron._**2021**, _26_, 1737-1744. [CrossRef]
* (4) [PERSON]; [PERSON]; [PERSON] Neural adaptive command filtered control for cooperative path following of multiple underactuated autonomous underwater vehicles along one path. _IEEE Trans. Syst. Man Cybern. Syst._**2021**, _52_, 2966-2978. [CrossRef]
* (5) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Finite-time unknown observer based coordinated path-following control of unmanned underwater vehicles. _J. Frankl. Inst._**2021**, _358_, 2703-2721. [CrossRef]
* (6) [PERSON]; [PERSON]; [PERSON] Deep reinforcement learning for quadrotor path following with adaptive velocity. _Auton. Robot._**2021**, _45_, 119-134. [CrossRef]
* (7) [PERSON]; [PERSON] [PERSON] A constrained error-based MPC for path following of quadrotor with stability analysis. _Nonlinear Dyn._**2020**, _99_, 899-918. [CrossRef]
* (8) [PERSON]; [PERSON]; [PERSON] [PERSON] A guiding vector-field algorithm for path-following control of nonholonomic mobile robots. _IEEE Trans. Control. Syst. Technol._**2017**, _26_, 1372-1385. [CrossRef]
* (9) [PERSON]; [PERSON]; [PERSON] [PERSON]; [PERSON] Information-Aware Lyapunov-Based MPC in a Feedback-Feedforward Control Strategy for Autonomous Robots. _IEEE Robot. Autom. Lett._**2022**, \(7\), 4765-4772. [CrossRef]
* (10) [PERSON]; [PERSON]; [PERSON]; [PERSON]; [PERSON] Path following control of tracked vehicle using modified sup controller optimized with particle swarm optimization (PSO). _Int. J. Dyn. Control._**2022**, 1-10.