# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 3 Classical Control of the AUV for the Docking Manoeuvre
### Optimal Control of the Docking of the AUV  
A parametric optimal control approach is used to find the time-dependent control input vector \(\mathbf{u}\) that minimises a cost function over a predefined time interval [36]. Examples for the application of  
Figure 2: Diagram of the PID control algorithm for the docking control of an AUV. parametric optimal control to nonlinear systems can be found in [37; 38; 39], from which this article takes inspiration. The aim is to have the AUV reach the desired docking position, orientation and velocity,  
\[\mathbf{x}_{\mathrm{f}}= \begin{bmatrix}x_{\mathrm{f}}&z_{\mathrm{f}}&\theta_{ \mathrm{f}}&u_{\mathrm{f}}&w_{\mathrm{f}}&q_{\mathrm{f}}&\mathrm{free}&\mathrm{ free}\end{bmatrix} ^{T}, \tag{18}\]  
at the end time, \(t_{\mathrm{f}}\). Note that no end condition is imposed on \(n\) and \(u_{\mathrm{p}}\). Similarly, initial conditions on the states have to be specified for time \(t_{0}=0\) s:  
\[\mathbf{x}_{0}= \begin{bmatrix}x_{0}&z_{0}&\theta_{0}&u_{0}&w_{0}&q_{0}&n_{0}&u_{ \mathrm{p},0}\end{bmatrix} ^{T}. \tag{19}\]  
The end condition represents the state of the AUV before starting the docking manoeuvre, i.e., when it is near the docking platform. Additionally, the control action is constrained within physical limits \(\mathbf{u}_{\mathrm{min}}\) and \(\mathbf{u}_{\mathrm{max}}\), which are due to the maximum magnitude of the torque that can be delivered by the motor and the maximum magnitude of the sterplane angle to prevent stall. Although it is possible to form the necessary conditions whose two-point boundary value problem solution should yield the optimal trajectory, the necessary conditions for optimality result in a two-point boundary value problem whose solution is the optimal trajectory in surge, heave and pitch [36]. However, no solution was found for the docking problem due to the nonlinear and cross-coupling terms in the dynamic equations and the constraints imposed on the optimisation. Therefore, a direct, parametric sub-optimal solution has been sought instead [36]. With this approach, the constraints were removed and included as quadratic penalty terms in the cost function instead. Fixing the end time \(t_{\mathrm{f}}\) and the time step used in the computation of the cost function greatly reduces the complexity of the problem and improves the robustness of the solution. Additionally, the control input signals, i.e., the motor torque \(Q_{\mathrm{m}}\) and sterplane angle \(\delta_{\mathrm{s}}\), are represented parametrically through predefined curves. This enables the solver to find a solution, since it reduces the number of unknowns from the number of time steps to the much smaller number of parameters. Although constraints or saturation limits cannot be imposed on the control signals, a cost should be imposed on their magnitude to prevent the controller from selecting unrealistically high values. The solution from the PID control was used to provide a more appropriate starting point to guide selection of the shape functions to parameterise the control signals for the motor torque and sterplane angle of Equation (6), respectively:  
\[Q_{\mathrm{m}}=p_{1}(1-\tanh(t-p_{2}))+p_{3}\exp\left[-\left( \frac{t-p_{4}}{p_{5}}\right)^{2}\right], \tag{20a}\] \[\delta_{\mathrm{s}}=\sum_{i=1}^{4}p_{5+i}\sin\left(\frac{i\pi t}{t_{ \mathrm{f}}}\right), \tag{20b}\]  
where \(\mathbf{p}\in\mathbb{R}^{9}\) is the vector of parameters. In Equation (20b), the sterplane signal is modelled as the sum of four sinusoidal basis functions. Conversely, Equation (20a) is more heavily engineered, as it presents the sum of a hyperbolic tangent and a Gaussian function. The hyperbolic tangent creates a smoothly decaying signal, while the Gaussian function creates the dip (for \(p_{3}<0\)) shown by the PID control response in Section 5 when the motor torque inverts its direction to break the vehicle's forward motion. A more general approach can be found in [40].