## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Path-Following and Obstacle Avoidance Controller Based on DDPG
Considering the path-following issue only, based on the current evaluation of the robot status, the basic reward function is designed as:  
\[re_{b}=-(|\bar{x}_{e}|+|\bar{y}_{e}|+|\bar{\theta}_{e}|) \tag{18}\]  
When the robot moves into obstacle avoidance regions, the reward function is redesigned by using extra punishments or rewards on the basis of the original reward for tracking control, making the robot capable of bypassing the obstacle without collisions. Based on the division into different regions, the reward function for obstacle avoidance regions is expressed as:  
\[re_{e}=\left\{ \begin{array}{cc}re_{b}+\max(\phi_{rz},1/\tanh(\frac{d_{obs}- r_{obs}}{r_{e}-r_{e}}))&\text{RewardsZone}\\ re_{b}-\max(\phi_{pz},1/\tanh(\frac{d_{obs}-r_{obs}}{r_{e}-r_{obs}}))&\text{ PunishmentZone}\\ re_{b}-\psi&\text{Obstacle}\end{array} \right. \tag{19}\]  
where \(\phi_{rz}\) and \(\phi_{pz}\) are parameters that limit the reward and penalty, respectively. Both are able to prevent large, abrupt changes in the single-step reward, which can cause instability during training. More specifically, not only is the distance to the circle of the obstacle for the reward function considered but the errors between the robot navigation angle and the obstacle direction are also taken into account in this work when the NWMR moves into the Punishment Zone. Based on this concept, the reward function is represented as:  
\[re_{e}=\left\{ \begin{array}{cc}re_{b}-\max(\phi_{pz},1/\tanh(\frac{d_{obs}- r_{obs}}{r_{e}-r_{obs}}))&if\theta_{obs}\leq\theta_{s}\\ re_{b}-\max(\phi_{pz},\xi/\tanh(\frac{d_{obs}-r_{obs}}{r_{e}-r_{obs}}))&if \theta_{obs}>\theta_{s}\end{array} \right. \tag{20}\]  
where \(\xi\) is the parameter that moderates the penalty according to the errors \(\theta_{obs\_e}\). If the NWMR collides with an obstacle during movement, then the task is considered a failure, a severe penalty \(\psi\) is imposed directly as the reward in this step, and the training environment will convert to a new episode. Due to this severe negative reward, the robot is able to follow the basic path-tracking control strategy in the learning of obstacle avoidance, and eventually it is able to complete the motion control for the whole set path while avoiding the obstacles. According to the above path-following and obstacle avoidance control strategy, the control process based on deep reinforcement learning is shown in Algorithm 1:```
0: robot random initial pose \(q_{o}\), path \(p\), training \(Max\_episode\), time step \(\Delta t\), learning rate \(\alpha\) for actor network and \(\beta\) for critic network, parameter \(\tau\in(0,1)\) for stability of training, discount factor \(\gamma\), experience replay buffer size \(N\), the number \(k\) of obstacles, obstacle avoidance position \(o_{obs}\), \(i\in[1,k]\), parameters \(r_{obs}\), \(r_{e}\), \(r_{c}\), \(\theta_{s}\), and parameters related to obstacle avoidance \(\psi\), \(\xi\), \(\theta_{s}\), \(\phi_{\tau z}\), \(\phi_{\tau z}\); Initialize: critic network \(Q(s,a|\theta^{Q})\) and actor network \(\mu(s|\theta^{u})\) randomly, target network \(Q^{\prime}(s,a|\theta^{Q^{\prime}})\) and \(\mu^{\prime}(s|\theta^{u})\);
1:for each\(episode\in[1,Max\_episode]\)do
2: Obtain an observation of random initial pose to NWMR in environment, then output position error \(q_{e}(t)\) through path parameters, and finally obtain initial state \(s_{1}=\left[ \begin{array}{cc}\tilde{x}_{e}&\tilde{y}_{e}&\tilde{\theta}_{e} &d_{obs}&\theta_{obs\_e}\end{array}