# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 4 Reinforcement Learning Control of the AUV for the Docking Manoeuvre
### Problem Formulation  
RL has been successfully applied to control applications (e.g., see, [43; 46; 47]). In these tasks, the controller represents the agent, while the system plant corresponds to the environment. The control output is the action, whereas the observable and estimated states may represent the state. The reward is typically expressed through a cost function. Most applications of RL to control tasks to date refer to episodic problems, i.e., the task can be defined as an episode with a specific desired outcome signalling its completion. Expressing the control problem as an episodic task greatly reduces the problem formulation complexity, as it enables the designer to exploit the classical RL framework. For the specific application to the docking of the AUV onto a fixed platform in surge, heave and pitch, it is possible to define the RL state space as follows:  
\[\mathbf{s}= \begin{bmatrix}x_{\mathrm{r}}&z_{\mathrm{r}}&\theta_{\mathrm{r}}&\dot{ x}&\dot{z}&\dot{\theta}&n\end{bmatrix} ^{T}. \tag{26}\]  
It is necessary to add the propeller revolutions, \(n\), to the state-space to provide an indication of the propeller thrust. Changes in motor torque will affect the thrust, but with a delay. Hence, ideally, the controller should learn to reduce the propeller revolutions as it approaches the goal position and orientation. Similarly, the action space is identical to the control output \(\mathbf{u}\):  
\[\mathbf{a}=\mathbf{u}= \begin{bmatrix}Q_{\mathrm{m}}&\delta_{\mathrm{s}}\end{bmatrix}  ^{T}. \tag{27}\]  
However, to prevent stall and because of torque limitations, the selected continuous action should be constrained to minimum and maximum values (\(\mathbf{a}_{\mathrm{min}}\) and \(\mathbf{a}_{\mathrm{max}}\), respectively). Specifying an appropriate reward function is fundamental to have the agent learn the desired behaviour. Although it is possible to inverse-engineer reward functions from examples or expert pilots [48], here a reward function is designed similarly to cost functions for control problems. In particular, the reward function for the docking problem should comprise of different elements: a continuous cost on the action to limit excessive power expenditures, a continuous cost on position so that the AUV is guided towards the docking point, a continuous cost on velocity so that small terminal speed is achieved, a high penalty for exceeding realistic environmental boundaries and high rewardsonce the AUV is close enough to docking point and at low speeds. The penalty and rewards make the reward function discontinuous and reliant on an if-loop. Inspiration for the reward function is taken from [32], where the authors employed RL for the landing of a spacecraft. Similarly, the cost on the velocity is expressed as a function of the distance to the desired docking point. This is because at low speeds the sterplane becomes ineffective due to the drop in lift force. Hence, a higher speed away from the docking point is necessary to achieve the desired level of control. Therefore, the reward function that has been designed for the docking of an AUV to a fixed or moving platform is  
\[c=-\omega_{x}x_{\mathrm{r}}^{2}-\omega_{z}z_{\mathrm{r}}^{2}-\omega_{\theta} \theta_{\mathrm{r}}^{2}-\omega_{Q}Q_{\mathrm{m}}^{2}-\omega_{\delta_{\mathrm{ s}}}\delta_{\mathrm{s}}^{2}-\omega_{u}\frac{u^{2}}{\max(x_{\mathrm{r}}^{2},l_{u})}, \tag{28}\]  
\[r=\left\{ \begin{array}{ll}c&\quad\text{if $f=0$ and $h=0$},\\ -p&\quad\text{if $h=1$},\\ c+r_{\mathrm{f}}+\frac{\omega_{x}}{\max(z_{\mathrm{r}}^{2},l_{x})}+\frac{ \omega_{z}}{\max(z_{\mathrm{r}}^{2},l_{z})}+\frac{\omega_{\theta}}{\max(\theta _{\mathrm{r}}^{2},l_{\theta})}&\quad\text{if $f=1$}.\end{array} \right. \tag{29a}\]  
The constants \(\omega\) indicate weights and \(l\) limiting values. The parameter \(r_{\mathrm{f}}\) is an additional reward achieved at the end of the episode for successful docking. Note that \(f\) is a boolean that expresses whether docking is achieved to the desired level of accuracy:  
\[f=\left\{