# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 5 Results  
### Simulation Framework  
The equations of motion in Section 2 were used to simulate the dynamics of a REMUS 100 AUV in surge, heave and pitch. The model parameters for this device were taken from [24] and are included in Appendix A for completeness. Note that for simplicity zero net buoyancy was assumed here. The simulation was implemented in the MATLAB/Simulink interface using specially designed C-coded S-functions. The MATLAB _fminunc_ optimisation function was used by the optimal controller with the Quasi-Newton optimisation algorithm [52]. A fourth-order-accurate Runge-Kutta method with fixed time step was used for the integration [53]. Although the simulation time step was set at 0.01 s to prevent numerical instabilities, the parametric optimal and the RL controllers operate with a more realistic sample period of 0.2 s. This value is a compromise between hardware limitations, power consumption and control performance. Although the sample rate was much higher than typical AUV operations, a shorter time step was required for the docking manoeuvre due to the required level of accuracy. The total simulation time was capped at 50 s (i.e., a time-out), with shorter times possible if docking occurs sooner. The AUV starting state vector was set as as  
\[ \begin{split}\mathbf{x}_{0}=\begin{bmatrix}0\;\text{m}&1\;\text{m}&0 \;\text{rad}&1.10513\;\text{m/s}&0\;\text{m/s}&0\;\text{rad/s}&80.1440\; \text{rad/s}&0.792386\;\text{m/s}\end{bmatrix}^{T},\end{split}  \tag{41}\]  
which corresponds to steady-state conditions for \(\delta_{\text{s}}=0\;\text{rad}\) and \(Q_{\text{m}}=40\;\text{Nm}\). The docking platform was sited at \(x_{\text{p}}=20\;\text{m}\), \(z_{\text{p}}=0\;\text{m}\) and \(\theta_{\text{p}}=0\;\text{rad}\). The measure of docking accuracy was set at \(l_{\eta}=0.1\). Additionally, ideally, the desired velocity vector of the AUV at the docking pointwas set to \(\hat{x}_{\text{f}}=0.1\;\text{m/s}\), \(\hat{z}_{\text{f}}=0\;\text{m/s}\) and \(\hat{\theta}_{\text{f}}=0\;\text{rad/s}\), as these values ensure a gentle manoeuvre. To prevent stall and include realistic constraints on the motor torque, \(Q_{\text{m,max}}=100\;\text{Nm}\), \(Q_{\text{m,min}}=-100\;\text{Nm}\), \(\delta_{\text{s,max}}=15^{\circ}\) and \(\delta_{\text{s,min}}=-15^{\circ}\). The discrete action space for the DQN algorithm was selected as  
\[ \begin{split}\mathcal{A}=\begin{bmatrix}\delta Q_{\text{m}}&0&- \delta Q_{\text{m}}&\delta Q_{\text{m}}&0&-\delta Q_{\text{m}}&0&- \delta Q_{\text{m}}\\ \delta\delta_{\text{s}}&\delta\delta_{\text{s}}&\delta\delta_{\text{s}}&0&0&0&- \delta\delta_{\text{s}}&-\delta\delta_{\text{s}}&-\delta\delta_{\text{s}} \end{bmatrix}^{T},\end{split}  \tag{42}\]  
where \(\delta Q_{\text{m}}=10\;\text{Nm}\) represents the selected step change in motor torque and \(\delta\delta_{\text{s}}=3^{\circ}\) in sterplane angle. Tables 1-4 contain the parameters used by the PID, parametric optimal and RL controllers, respectively. For the RL algorithms, the parameters were initialised from common-sense values based on the literature (e.g., from [32]) and then tuned to achieve successful docking for the analysed cases. Furthermore, the following limits were used to normalise the state vector to help the training of the deep neural networks:  
\[ \begin{split}\mathbf{s}_{\text{min}}=\begin{bmatrix}l_{x,\text{min} }&l_{z,\text{min}}&l_{\theta,\text{min}}&-2\;\text{m/s}&-1\;\text{m/s}&-2\;\text{ rad/s}&-200\;\text{rad/s}\end{bmatrix}^{T},\end{split}  \tag{43a}\] \[ \begin{split}\mathbf{s}_{\text{max}}=\begin{bmatrix}l_{x,\text{max} }&l_{z,\text{max}}&l_{\theta,\text{max}}&2\;\text{m/s}&1\;\text{m/s}&2\;\text{ rad/s}&200\;\text{rad/s}\end{bmatrix}^{T}.\end{split}  \tag{43b}\]The deep neural networks used to approximate the critic for the DQN algorithm and the critic and the actor for the DDPG algorithm can be seen in Figures 6 and 7, respectively.