## 4 DRL Based TDIA Transmission Scheduling Algorithm (DRLSA-IA)
### Formulated as a DRL Problem  
The transmission scheduling problem is formulated as a DRL problem. We give the following definitions: agent, action space, state space and objection function.  
(a) Agent: We regard the node that uses DRLSA-IA as a DRL agent. There are \(N\) nodes in the networks.  
Figure 4: The propagation delays between nodes at time \(t1\).  
Figure 5: The propagation delays between nodes at time \(t2\).  
(b) Action space: In an underwater node collaboration network, nodes communicate with each other. Each node selects an expected node to send a message or remain idle in each time slot \(t\). Therefore, the action space \(A\) of nodes is defined as,  
\[\mathbf{A}=\{0,1,2,\ldots,N\}. \tag{5}\]  
Each node \(i\) selects an action \(a_{i}(t)\) from the action space \(\mathbf{A}\) in every time slot \(t\), i.e., \(a_{i}(t)\in\mathbf{A}\). If \(a_{i}(t)=0\), node \(i\) will be idle in time slot \(t\). If \(a_{i}(t)=k\
eq 0\), node \(i\) will transmit a message to node \(k\) in time slot \(t\).  
(c) State space: Each node \(i\) can only observe part of the network state. The network state \(S_{i}(t)\) observed by node \(i\) in time slot \(t\) consists of all ACK packets received and the actions performed by node \(i\) before time slot \(t\), as shown in Equation (6).  
\[S_{i}(t)=\left\{a_{i}(t),\left\{o_{j}^{i}(t)\right\}_{j=1}^{N}\right\}_{t=1}^{ t}, \tag{6}\]  
where \(o_{j}^{i}(t)\) denotes the ACK packet returned by node \(j\), \(o\in\{-1,0,1\}\). \(o=1\) indicates that the message is successfully received. \(o=0\) indicates that the message conflicts with an unintended message. \(o=-1\) indicates that the message conflicts with an intended message. \(\left\{o_{j}^{i}(t)\right\}_{j=1}^{N}\) denotes all ACK packets received by node \(i\) in time slot \(t\).  
(d) Reward function: To improve the network throughput, we design a TDIA-based reward function. We explain the reward function in detail in Section 4.2.  
(e) Objective function: We formulate the transmission scheduling problem as a DRL problem, so the throughput maximization problem can be formulated as follows:  
\[\max_{\mathbf{S}}\mathbb{E}\left[\sum_{t=1}^{T}\gamma^{t-1}R(t)\right], \tag{7}\]  
where \(\gamma\in(0,1)\) is a discount factor and \(R(t)\) is the reward in time slot \(t\). \(\mathbf{S}\) is the transmission schedule. The objective function is the expected value of the discounted sum of reward over a time horizon of length \(T\).  
A Q-function is used to evaluate the accumulative reward of the system, as shown in Equation (8),  
\[Q(S,a)=\mathbb{E}\left[\sum_{t=1}^{T}\gamma^{t-1}R(t)\mid S_{i}(t)=S,a_{i}(t) =a\right]. \tag{8}\]  
The node follows a policy \(\mathbf{\delta}\) to interact with the environment. The \(\mathbf{\delta}\) maps states to actions. We aim to find the optimal policy \(\mathbf{\delta}^{*}\) that maximizes the Q-function as \(\mathbf{\delta}^{*}(S)=\arg\max_{a}\{Q(S,a)\}\). Equation (9) is the optimal Q-function,  
\[Q^{*}(S,a)=\mathbb{E}\left[R(t)+\gamma\max_{a^{\prime}}Q^{*}\big{(}S^{\prime},a^{\prime}\big{)}|S_{i}(t)=S,a_{i}(t)=a\right], \tag{9}\]  
where \(S^{\prime}\) and \(a^{\prime}\) are the next state and action after the node takes action \(a_{i}(t)\) when in state \(S_{i}(t)\), respectively.