# AUV-Aided Optical--Acoustic Hybrid Data Collection Based on Deep Reinforcement Learning
## 5 Proposed DRL-Based Multi-Modal Data Collection Scheme
### DRL-Based Multi-Modal Path Planning Scheme  
Due to the uncertainty of reference access points and node data arrivals, the locations of AUV and the AoI of collected data are inherently random, which leads to a proliferation of state space dimensions. In comparison, DRL can handle extremely large state space by estimating the \(Q\) values of states \(s\) and actions \(a\) through neural networks [45; 46]. The training framework of DQN includes a current Q-network and a target Q-network. In order to balance experience and exploration of the unknown, the agent at state \(s_{t}\) selects the action \(a_{t}\) to be performed by the \(\epsilon\)-greedy algorithm [47]. \[a_{t}=\left\{ \begin{array}{ll}random\ a\in\mathcal{A},&\text{with probability}\ \varepsilon\\ \arg\max_{a_{t}}Q(s_{t},a_{t};\theta),&\text{with probability}\ 1-\varepsilon\end{array} \right.. \tag{24}\]  
Immediately after adjusting the navigation angle and execution of action \(a_{t}\), the AUV receives reward \(r_{t}\) and the data acquisition network moves to the next state \(s_{t+1}\). Aiming to reduce the correlation between the online Q-network samples, an experience replay \(\mathcal{B}\) is used to store historical experience samples \((s_{t},a_{t},r_{t},s_{t+1})\). At each training step, a small batch of randomly selected empirical samples \(\Phi_{b}\) from the experience replay is used to update the parameters of the online Q-network. In addition, we denote the parameters of DQN as \(\theta\), and the parameters of the online Q-network are determined by minimizing the loss function. \[L(\boldsymbol{\theta})=E_{\Phi_{b}}\Big{[}\delta(s,a)^{2}\Big{]}, \tag{25}\]  
where \(\delta(s,a)=y_{t}-Q(s_{t},a_{t}|\boldsymbol{\theta})\) is the temporal difference, and \(y_{t}\) is the target Q-value, which can be calculated by  
\[y_{t}=r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1}|\boldsymbol{\theta}^{-}), \tag{26}\]  
where \(\boldsymbol{\theta}^{-}\) denoted the parameter of the target Q-network. Then, the weight of the current network \(\boldsymbol{\theta}\) is updated by the following formula. \[\boldsymbol{\theta}=\boldsymbol{\theta}+\frac{\alpha}{\Phi_{b}}\sum_{t=1}^{ \Phi_{b}}\delta(s,a)\
abla_{\theta}Q(s_{t},a_{t},\boldsymbol{\theta}). \tag{27}\]The proposed AUV-assisted data collection algorithm is shown in Algorithm 2. The algorithm starts by initializing all neural networks as well as the replay buffer \(\mathcal{B}\). The training iterates over \(E\) episodes, and the environment is initialized in each episode by observing the distribution of CHs. The action is first obtained according to the \(\epsilon\)-greedy policy, which is followed by inputting the action to Algorithm 1 to obtain the steering angle. Then, the AUV moves to the next state \(s_{t+1}\) and receives an immediate reward \(r_{t}\). After storing the transition tuple \((s_{t},a_{t},r_{t},s_{t+1})\) in experience replay \(\mathcal{B}\), a randomly selected sample of \(\Phi_{b}\) is utilized to learn the current network \(Q\), and it updates the weights of the current network \(\mathbf{\theta}\) and that of the target network \(\mathbf{\theta}^{-}\). Then, \(c_{i}\) is removed from \(N_{r}\) if the current state is able to collect the data of \(c_{i}\), and the current loop is terminated when \(N_{r}=\emptyset\) holds. ```
1:Input: Initialize the constants \(k_{1},k_{2}\) and \(k_{3}\), maximum number of training sets \(E\), reward discount factor \(\gamma\), learning rate \(l_{r}\), experience replay \(\mathcal{B}\), minimum batch \(\Phi_{b}\), exploration probability \(\epsilon\), and update step \(\chi\);
2: Initialize the current network \(Q(s_{t},a_{t},\mathbf{\theta})\) with weights \(\mathbf{\theta}\) and the target network \(Q(s_{t},a_{t},\mathbf{\theta}^{-})\) with weights \(\mathbf{\theta}^{-}\). 3:forepisode \(=1,\cdots,E\)do
4:for\(t=1,\cdots,\mathcal{T}\)do
5: Initialize the data collection network environment and observe the initial state \(s_{t}\). 6: Select a random action \(a_{t}\) according to the \(\epsilon\)-greedy algorithm.