# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 4 Reinforcement Learning Control of the AUV for the Docking Manoeuvre
### Deep Q-Network
One common approach with RL schemes with discrete actions is to select three actions per action signal: the maximum, minimum and zero values of the signal [43]. This results in a bang-bang control type, which can be proven to be optimal with [PERSON]'s principle [36]. However, here, a different approach is selected similarly to [50; 51]. The action space is selected as the combination of positive, negative and zero step changes in the motor torque and sternplane angle, as described in Section 5.1. Therefore, at each time step, the motor torque and sternplane angle are changed by a fixed amount, which should be selected as a realistic value based on the physical constraints of the system. To ensure learning occurs, the values of \(Q_{\text{m}}\) and \(\delta_{\text{s}}\) must be tracked. Otherwise, the action information, i.e., a relative change in the control input, is not sufficient for the agent to successfully learn a policy, since information on the absolute (positive or negative) value of the motor torque and sternplane angle is required. Hence, the state space in Equation (26) is modified as follows for the DQN algorithm for the docking control of the AUV in 3 DOF:  
\[\mathbf{s}_{\text{DQN}}= \begin{bmatrix}\mathbf{s}&Q_{\text{m}}&\delta_{\text{s}}\end{bmatrix} ^ {T}. \tag{36}\]Note that upper and lower limits are imposed on the values of the motor torque and sterplane angles to reflect realistic physical boundaries. Hence, even if the action corresponding to a positive change in motor torque is selected for \(Q_{\text{m}}=Q_{\text{m,max}}\), no changes will be applied, since the value of \(Q_{\text{m}}\) is saturated. ```
Output: state-action value function \(Q(s,a|\theta_{Q})\) randomly initialise the critic \(Q(s,a|\theta_{Q})\) network with weights \(\theta_{Q}\); initialise the target critic \(Q^{\prime}\) network with weights \(\theta^{\prime}_{Q}\leftarrow\theta_{Q}\); initialise the replay buffer \(\mathcal{R}\); initialise exploration rate \(\epsilon\); for each episodedo  update exploration rate \(\epsilon\) with decay; start from an initialise state \(s\); for each time step up to maximum number of time stepsdo  select action \(a\) with \(\epsilon\)-greedy policy (Equation (33));  apply action \(a\), get the reward \(r\) and observe the new state \(s^{\prime}\);  store the transition \((s,a,r,s^{\prime},d)\) in \(\mathcal{R}\) if different enough from existing data; if\(s^{\prime}\) is terminalthen  reset the environment state;  end if  sample a random data minibatch \(\mathcal{M}\in\mathbb{R}\); for all transitions \(i=1,I\) in \(\mathcal{M}\)do  set the transition value to \(u_{i}=r_{i}+\gamma(1-d_{i})Q^{\prime}(s^{\prime}_{i},a^{\prime}|\theta_{Q^{ \prime}})\) with Equation (34);  update the critic by minimising the loss \(L=\frac{1}{I}\sum_{i}\left[y_{i}-Q(s_{i},a_{i}|\theta_{Q})\right]^{2}\);  update the target network with Polyak averaging with Equation (35);  end if  update the state \(s\leftarrow s^{\prime}\);  end for  end for
```  
**Algorithm 1**DQN algorithm adapted from [49]. The DQN algorithm for the docking control of the AUV onto a fixed platform in surge, heave and pitch is shown graphically in Figure 4.