# Docking Control of an Autonomous Underwater Vehicle Using Reinforcement Learning
## 4 Reinforcement Learning Control of the AUV for the Docking Manoeuvre  
### Reinforcement Learning Statement  
RL is a decision-making framework in which an _agent_ learns a desired behaviour, or _policy_\(\pi\), from direct interactions with the _environment_[41]. At each time step, the agent is in a _state_\(s\) and takes and _action_\(a\). As a result, it lands in a new state \(s^{\prime}\) while receiving a _reward_\(r\). A Markov decision process is used to model the action selection depending on the value function \(Q(s,a)\), which represents an estimate of the future reward. By interacting the environment for a long time, the agent learns an optimal policy, which maximises the total expected reward. This is shown graphically in Figure 3.  
The on-policy _action-value_ function \(Q^{\pi}(s,a)\) yields the expected reward if the agent starts in state \(s\), takes an action \(a\) and afterwards acts according to policy \(\pi\)[41]:  
\[Q^{\pi}(s,a)=E_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_{0}=s \right], \tag{22}\]  
where \(\tau=(s_{0},a_{0},s_{1},a_{1},\dots)\) indicates a sequence of actions and states, \(\gamma\in[0,1]\) is a discount factor and \(E\) expressed the expected, discounted value. Conversely, the optimal action-value function \(Q^{*}(s,a)\) yields the expected reward if the optimal policy is followed after starting from state \(s\) and taking action \(a\)  
\[Q^{*}(s,a)=\max_{\pi}E_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}|s_ {0}=s\right]. \tag{23}\]  
Since the optimal policy will result in the selection of the action that maximises the expected reward starting from \(s\), it is possible to obtain the optimal action \(a^{*}(s)\) from \(Q^{*}\) as follows:  
\[a^{*}(s)=\arg\max_{a}Q^{*}(s,a). \tag{24}\]  
As a result, it is possible to express the Bellman equation for the optimal action-value function as follows [41]:  
\[Q^{*}(s,a)=E_{s^{\prime}\sim P}\left[r(s,a)+\gamma\max_{a^{\prime}}Q^{*}(s^{ \prime},a^{\prime})\right], \tag{25}\]  
where \(P\) indicates state transitions according to a stochastic policy \(\pi\). RL algorithms can be subdivided into _model-based_ and _model-free_ schemes [42]. Whereas model-based strategies enable the agent to plan by exploiting the model of the environment, thus greatly improving their behaviour, exact models of the environment are generally unavailable in practice. Any existing bias in the model is likely to cause significant problems with generalising the agent's behaviour to the actual environment. As a result, RL research has focused on model-free approaches to date. Figure 3: Reinforcement learning diagram. Within model-free schemes, RL algorithms can be further categorised based on what they learn. On the one hand, the strategies in the policy optimisation group directly represent the policy and optimise its parameters. Since the policy update only relies on data collected while following the policy, these approaches are known as _on-policy_. Note that policies can be _stochastic_, defined as \(\pi\), or _deterministic_, defined as \(\mu\). On the other hand, Q-learning strategies learn an approximation for the optimal action-value function based on the Bellman equation. As the update of the optimal Q-value may employ data collected at any point during training, these schemes are known as _off-policy_. Whereas policy optimisation schemes are more robust, since they directly optimise what is being sought, Q-learning methods can be significantly more sample-efficient when they work. Additionally, RL algorithms can be subdivided into _on-line_ and _off-line_ types. With on-line schemes, learning occurs in real time from the data stream. In off-line strategies, the policy is updated only at regular intervals and stored in memory. Both types of algorithms have much lower computational cost at deployment time than during training. The first RL algorithms dealt with discretised state and action spaces [41].