# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 4 Reinforcement Learning Control of the AUV for the Docking Manoeuvre
### Deep Q-Network  
The first RL algorithm investigated is deep Q-network (DQN), a popular model-free, on-line, off-policy strategy [44; 49]. DQN trains a critic to approximate the action-value (or Q-) function \(Q(\mathbf{s},a|\mathbf{\theta}_{\mathrm{Q}})\), i.e., to estimate the return of future rewards. Hence, the algorithm relies on a discrete action-space and a continuous state-space. As a result, the action is selected with an \(\epsilon\)-greedy policy [41]:  
\[a=\left\{ \begin{array}{ll}\arg\max_{a^{\prime}\in\mathcal{A}}Q(\mathbf{s},a^{ \prime}|\mathbf{\theta}_{\mathrm{Q}})&\quad\text{if $b\leq\epsilon$},\\ \text{random}(a^{\prime}\in\mathcal{A})&\quad\text{if $b>\epsilon$},\end{array} \right. \tag{33a}\]  
where \(\mathcal{A}\) indicates the action space, i.e., a vector of all possible actions, \(b\in[0,1]\) is a random number and \(\epsilon\in[0,1]\) is defined as the exploration rate. The exploration rate is typically set to a high value close to \(1\) at the start of the training process and as learning progresses a decay function is employed to lower its value. This ensures that the agent focuses on exploring the environment at the beginning and then shifts to exploiting the optimal actions as learning progresses. In DQN, the critic approximator is represented by a deep neural network. Two features ensure that training occurs smoothly. Firstly, the algorithm relies on the concepts of experience replay buffer [44; 45]. With this strategy, at each step, the state state, action, reward, new state and flag for episode end from the previous step \((\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime},d)\) are saved as a transition inside the experience memory buffer \(\mathcal{R}\). During training, a mini-batch of experiences \(\mathcal{M}\in\mathcal{R}\) is then sampled from the memory to update the deep neural networks. If only the latest experience is used, the neural network is likely to suffer from overfitting. To ensure stability, the replay buffer should be large and contain data from a wide range for each variable. However, only a limited number of data points (usually in the order of \(10^{6}\)-\(10^{7}\)) can be stored in memory. Hence, new experience should be stored only if it describes a data point different enough from the existing data. Furthermore, using batches sampled randomly from the experience buffer ensures the best compromise between computational performance and overfitting avoidance. Since learning occurs without relying on the current experience, DQN is described as off-policy scheme. Furthermore, the DQN algorithm exploits the concept of a target network for the critic to improve learning and enhance the stability of the optimisation. This network, \(Q^{\prime}(\mathbf{s},a|\mathbf{\theta}^{\prime}_{Q})\) has the same structure and parameterisation as \(Q(\mathbf{s},a|\mathbf{\theta}_{Q})\). The value function target is thus set as  
\[y=r+\gamma(1-d)\max_{d^{\prime}\in\mathcal{A}}Q^{\prime}(\mathbf{s}^{\prime},a^{ \prime}|\mathbf{\theta}_{Q^{\prime}}), \tag{34}\]  
which indicates the sum of the immediate and discounted future reward. The main problem is that the value function target depends on the same parameters that are being learned in training, thus making the minimisation of the mean-squared Bellman error unstable. For this reason, the target network is used, as it has the same set of parameters are the critic network, but lags it in time. Hence, the target network is updated once per main network update by Polyak averaging [45]:  
\[\theta_{Q^{\prime}}=\tau\theta_{Q}+(1-\tau)\theta_{Q^{\prime}}, \tag{35}\]  
where \(\tau\in[0,1]\) is the smoothing factor. The DQN algorithm is summarised in Algorithm 1, which is taken from [44; 49] with the addition of Polyak averaging as in the MATLAB implementation used here. DQN Docking Control of an AUV in 3 DOF  
Since DQN relies on discrete actions, only a limited set of actions \(\mathcal{A}\) can be selected by the controller.