# AUV-Aided Optical--Acoustic Hybrid Data Collection Based on Deep Reinforcement Learning
## 5 Proposed DRL-Based Multi-Modal Data Collection Scheme
### MDP Formulation  
When the network nodes are clustered, the next goal is to find an optimal CHs data collection strategy. The AUV-assisted data collection problem can be formulated as an MDP to be solvable by the DRL approach, which is represented by \(<\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma>\) five tuples. Here, \(\mathcal{S}\) is the state of the environment, \(\mathcal{A}\) is the set of actions of the agent, \(\mathcal{P}\) is the state transition probability, \(\mathcal{R}\) is the reward function, and \(\gamma\) denotes the discount factor. In particular, at time slot \(t\), the agent observes state \(s_{t}\) and chooses an action to be performed. Then, the environment state is transferred with probability \(p_{s_{t},s_{t+1}}\) to \(s_{t+1}\) and the agent obtains a reward \(r_{t}\) from the environment. In this paper, the AUV is considered as the agent to collect data, and the details of each element are defined as follows.  
* State space \(\mathcal{S}\): The status of AUV mobile collection is defined as \[s_{t}=\Big{\{}p_{a,t},\psi_{t},\Delta_{t},\{x_{i,t}^{ac},x_{i,t}^{op},d_{i,t}, \delta_{i,t}\}_{i\in N}\Big{\}},\] (18) where \(p_{a,t}\) and \(\psi_{t}\) are the coordinates and sailing orientation of the AUV at time slot \(t\), and its position can be obtained via ultra-short baseline (USBL) [44]. \(\Delta_{t}\) is the difference between the remaining energy of the AUV and the AUV's arrival at its final destination from its current position. \(d_{i,t}\) records the Euclidean distance of the AUV to CH \(c_{i}\). \(x_{i,t}^{m}\) is the data collection indicator related to the data collection option. When the AUV arrives at the data collection point \(c_{i}\), the AoI of node \(c_{i}\) starts to be updated.  
Action space \(\mathcal{A}\): In state \(s_{t}\), the action selection of the AUV is characterized by the target point \(c_{i,t}\in N_{r}\) with the transmission option \(m_{i,t}\), and the next target point \(c_{j,t}\in N_{r}\backslash c_{i,t}\), where \(N_{r}\) is the set of CHs that have not been collected. Then, the action performed by AUV at state \(s_{t}\) can be expressed as \[a=\big{\{}c_{i,t},m_{i,t},c_{j,t}|s_{t}\big{\}}.\] (19)
* State transition probability \(\mathcal{P}\): \(\mathcal{P}(s_{t+1}|s_{t},a_{t})\) defines the transition probability from state \(s_{t}\) to the next state \(s_{t+1}\) under the action \(a_{t}\), and \(\mathcal{P}(s_{t+1}|s_{t},a_{t})=1\) holds.
* Reward \(\mathcal{R}\): Applying action \(a_{t}\) in state \(s_{t}\), the AUV enters state \(s_{t+1}\) and obtains an immediate reward \(r(s_{t+1}|s_{t},a_{t})\). In the AUV-assisted multi-modal data collection scenario, the immediate reward \(r_{t}\) can be expressed as \[r_{t}=\left\{\begin{array}{ll}x_{i,t}^{ac}k_{1}\epsilon_{i}+x_{i,t}^{op}k_{ 2}\epsilon_{i},&\text{if }b_{i}=1\\ J,&\text{if done}\\ \eta_{i}(dis_{p_{a},p_{i}}+1),&\text{otherwise}\end{array}\right.,\] (20) where \(k_{1}\), \(k_{2}\) are constants and \(k_{1}<k_{2}\) holds, and when the AUV has collected the data of CH \(c_{i}\), the relevant reward is obtained according to the selected modem. \(dis_{p_{a},p_{i}}\) is the Euclidean distance from the current position of the AUV to the target point. \(J\) denotes the reward at the end of the data collection process, including rewards for successful data collection and penalties for failure (e.g., exceeding maximum energy consumption and crossing boundaries). \[J=\left\{\begin{array}{ll}r_{out},&\text{if }\Delta_{t}<0\text{ or }p_{a}\
otin\Omega\\ k_{3}-\frac{1}{N}\sum\limits_{i=1}^{N}A_{i},&\text{if }\text{p}_{a}=\text{p}_{0} \text{ and }\text{N}_{\text{r}}=\varnothing\end{array}\right.,\] (21) where \(k_{3}\) is a constant and \(\Omega\) is the region in which the AUV can move within.
* Discount factor \(\gamma\): \(\gamma\in[0,1]\) is the future reward discount factor.