# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 4 Reinforcement Learning Control of the AUV for the Docking Manoeuvre
### Deep Deterministic Policy Gradient  
A popular model-free, on-line, off-policy RL algorithm known as deep deterministic policy gradient (DDPG) is adopted here [45]. Although DDPG has been found to be sensitive to hyperparameter tuning  
Figure 4: Diagram of the DQN algorithm for the docking control of an AUV. and to suffer from overestimating the Q-values [46; 47], it has been successfully applied to the depth and low-level control of AUVs [30; 31]. DDPG is a type of actor-critic strategy developed by [PERSON] et al. [45]. As shown in Figure 5, the critic evaluates the action-value (or Q-)function, while the actor improves the policy in the direction suggested by the critic. In DDPG, a deep neural network is used by the critic to approximate the expectation of the long-term reward based on the state observation \(\mathbf{s}\) and action \(\mathbf{a}\). Hence, the state-action value function can be expressed through the neural network features \(\mathbf{\theta}_{Q}\) as \(Q(\mathbf{s},\mathbf{a},\mathbf{\theta}_{Q})\). An additional deep neural network is employed by the actor for the approximation of the optimal policy: based on state \(\mathbf{s}\), the network returns the continuous action \(\mathbf{a}\) that is expected to maximise the long-term reward or state-action value. Thus, by including the neural network features \(\mathbf{\theta}_{\mu}\), the policy can be expressed as \(\mu(\mathbf{s},\mathbf{\theta}_{\mu})\). Hence, in the DDPG algorithm, the agent learns a deterministic policy and the Q-function concurrently. Furthermore, similar to DQN, the DDPG algorithm exploits target networks for the critic and actor to improve learning and enhance the stability of the optimisation. These networks, \(Q^{\prime}(\mathbf{s},\mathbf{a},\mathbf{\theta}^{\prime}_{Q})\) and \(\mu^{\prime}(\mathbf{s},\mathbf{\theta}^{\prime}_{\mu})\), respectively, have the same structure and parameterisation as \(Q(\mathbf{s},\mathbf{a},\mathbf{\theta}_{Q})\) and \(\mu(\mathbf{s},\mathbf{\theta}_{\mu})\), respectively. The value function target is thus set as  
\[y=r+\gamma(1-d)Q^{\prime}(\mathbf{s}^{\prime},\mu^{\prime}(\mathbf{s}^{\prime}|\mathbf{ \theta}^{\prime}_{\mu})|\mathbf{\theta}_{Q^{\prime}}), \tag{37}\]  
which indicates the sum of the immediate and discounted future reward. In Equation (37), the agent first obtains the new observation from the policy through the actor. Then, the cumulative reward is found through the critic. As for DQN, the target networks are updated once per main network updated by Polyak averaging [45] with the following equation in addition to Equation (35):  
\[\theta_{\mu^{\prime}}=\tau\theta_{\mu}+(1-\tau)\theta_{\mu^{\prime}}. \tag{38}\]  
Similar to DQN, DDPG is an off-policy strategy as it relies on an experience replay buffer. Using the samples in the data batch \(\mathcal{M}\), the parameters of the deep neural network for the critic are updated by _minimising_ the loss over all \(I\) samples [45]:  
\[L=\frac{1}{I}\sum_{i}\left[y_{i}-Q(s_{i},a_{i}|\mathbf{\theta}_{Q})\right]^{2}. \tag{39}\]  
Conversely, the parameter of the deep neural network for the actor are updated by _maximising_ the expected discounted reward:  
\[\
abla_{\mathbf{\theta}_{\mu}}I\approx\frac{1}{I}\sum_{i}\
abla_{\mathbf{a}}Q(\mathbf{s}, \mathbf{a}|\mathbf{\theta}_{Q})|_{s=s_{i},\mathbf{a}=\mu(s_{i})}\
abla_{\mathbf{\theta}_{\mu} }\mu(\mathbf{s}|\mathbf{\theta}_{\mu})|_{s_{i}}. \tag{40}\]  
Figure 5: Diagram of the DDPG algorithm for the docking control of an AUV. The off-policy nature of the DDPG algorithm is further exploited to deal with the RL long-standing compromise of exploration and exploitation. Especially during the initial stages of learning, it is fundamental for the agent to explore the state-action space to optimise performance and avoid getting stuck in a local optimum. As learning progresses, the agent can focus on exploiting the action that yields maximum overall reward.