## 3 Path-Following and Obstacle Avoidance Control Strategy Incorporating Reinforcement Learning
### Reinforcement Learning Control Method  
RL can directly interact with the environment without having any information in advance [19]. Classical reinforcement learning approaches are based on the Markov decision process (MDP), consisting of the set of states \(S\), the set of actions \(A\), the rewards \(R\), and the transition probabilities \(T\) that capture the dynamics of a system [31]. According to the Markov property, the next state \(s_{t+1}\) is obtained by the model from the state \(s_{t}\) and action \(a_{t}\). This is called transition probability model \(p(s_{t+1}|s_{t},a_{t})\), and a reward \(r(s_{t},a_{t})\) is obtained after state transition evaluation. The whole process from \(s_{t}\) to \(s_{t+1}\) can be considered as one training step of the reinforcement learning, where the aim is to find the optimal strategy \(\pi^{*}\), i.e., the stochastic policy \(\pi(a|\mathrm{s})\) or the deterministic policy \(\pi(\mathrm{s})\) that can be evaluated by using the value function \(V^{\pi}(s)\) or the state value function \(Q^{\pi}(s_{t},a_{t})\), which is expressed as:  
\[ \begin{split}&\pi^{*}=\underset{\pi}{\arg\max}\;V^{\pi}(s_{t})\\ &=\underset{\pi}{\arg\max}\;Q^{\pi}(s_{t},a_{t})\end{split}  \tag{4}\]  
where both value functions are shown in Equations (5) and (6) separately, and the accumulated discounted reward \(R_{t}\) is given in Equation (7). \[V^{\pi}(s_{t})=E_{\pi}[R_{t}|s_{t}] \tag{5}\]  
\[Q^{\pi}(s_{t},a_{t})=E_{\pi}[R_{t}|s_{t},a_{t}] \tag{6}\]  
\[R_{t}=r_{t}+r_{t+1}+r_{t+2}+\cdots=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k+1} \tag{7}\]  
Recently, researchers have used the techniques of experience replay and a separate target network to eliminate instability by establishing the large-scale neural network called DQN in the RL problem. This has already shown excellent performance [32]. However, the DQN is limited by the discrete nature of its action space, and is not capable of dealing with continuous control problems [19]. To overcome the difficulty of accurate expressions for actions, deterministic policy gradient (DPG) is proposed for handling the continuous action space [33]. The deterministic policy \(a{=}\mu_{\theta}(\mathrm{s}):\mathcal{S}\mapsto\mathcal{A}\) is considered instead of the policy \(\pi_{\theta}(a|s)\) selected stochastically in state \(S\), and the vector \(\theta\in\mathbb{R}^{n}\) is its parameter. If the target policy is deterministic, then the value function \(Q^{\pi}(s_{t},a_{t})\) can be expressed as \(Q^{\mu}(s_{t},a_{t})\), and the expectation can be avoided [33] in Equation (8):  
\[Q^{\mu}(s_{t},a_{t})=\tau_{s_{t},s_{t+1}\sim E}[r(s_{t},a_{t})+\gamma Q^{\mu}( s_{t+1},\mu(s_{t+1}))] \tag{8}\]  
If there is an approximator parametrized by \(\theta^{Q}\), then it can be optimized by minimizing the following loss:  
\[L(\theta^{Q})=\mathbb{E}_{s_{t}\sim\rho^{\theta},a_{t}\sim\beta_{t}\gamma^{ \rho}E}[(Q(s_{t},a_{t}|\theta^{Q})-y_{t})^{2}] \tag{9}\]  
where \(y_{t}\) is dependent on \(\theta^{Q}\):  
\[y_{t}=r(s_{t},a_{t})+\gamma Q(s_{t+1},\mu(s_{t+1})|\theta^{Q}) \tag{10}\]\(Q(s,a)\) can be considered as the critic, which is learned by Q-learning using the Bellman equation and is updated by the expected return from the actor network using a DPG [33]:  
\[ \begin{split}\
abla_{\theta^{p}}J&\approx\mathbb{E}_{s _{t}\sim\rho^{\mathcal{B}}}[\
abla_{\theta^{p}}Q(s,a|\theta^{Q})|_{s=s,t=\mu(s _{t}|\theta^{p})}]\\ &=\mathbb{E}_{s_{t}\sim\rho^{\mathcal{B}}}[\
abla_{a}Q(s,a|\theta ^{Q})|_{s=s_{t},\mu=(s_{t})}\
abla_{\theta^{p}}\mu(s|\theta^{p})|_{s=s_{t}}] \end{split}  \tag{11}\]  
Regarding the approach to handling large networks in DQN, the DDPG algorithm uses experience replay and separate target network techniques to deal with large-scale neural network approximators in deep reinforcement learning. The DDPG has two basic networks called critic and actor, respectively. The whole process structure of the algorithm is presented in Figure 2 and will be used in the following study of the control strategy.