## 2 Related Works  
Recently, there has been a lot of research work on underwater MAC protocols [15; 16; 17]. The MAC protocols of UWANs fall into two categories: schedule-based MAC protocols and contention-based MAC protocols. The schedule-based MAC protocols mean that the channel resources are allocated to each node according to a predetermined mechanism. For example, Time Division Multiple Access (TDMA)-based protocols divide time into several time slots. Nodes use pre-allocatedtime slots to transmit messages. Frequency Division Multiple Access (FDMA)-based protocols divide the available frequency band of acoustic channels, and each user owns their unique frequency channel. Code Division Multiple Access (CDMA)-based protocols allow multiple users to transmit data simultaneously, and the receiving node uses pseudo-random code to distinguish different sending nodes. Contention-based protocols are mainly divided into random access protocols and handshake-based protocols. Among them, the ALOHA protocol [18] is a classic random access protocol. As long as there is a message, the node will send it immediately. Thus, collisions occur frequently. Slotted-Aloha [19] is an improved version of pure ALOHA with the time slot. In Slotted-Aloha, the node will start to transmit a message in the next time slot. This algorithm prevents nodes from sending messages randomly and reduces message conflicts to a certain extent. Random access protocols lead to frequent conflicts in networks with high traffic, but it performs better in low-traffic networks due to the simple access mechanism. To avoid collisions, the handshake-based protocols send control packets to compete for the channel before sending data. However, additional control packets increase network load and network delay. To reduce the number of control packets, [PERSON] et al. [20] proposed a Hybrid Sender and Receiver-initiated (HSR) protocol. In HSR, multiple nodes share control packages by inviting neighbor nodes to join the current handshake. The time slot-based scheduling algorithms convert channel allocation into time slot scheduling. Nodes use the allocated time slots to transmit or receive messages. [PERSON] et al. [21] proposed an evolutionary genetic node scheduling algorithm called GA. However, they assumed that the network was static. In [14], the authors proposed a packet-level slot scheduling algorithm. The slot scheduling problem was formulated into a combinatorial optimization problem. Similarly, the algorithm assumed that the network was static. In [22], the authors considered the message scheduling problem in a linear topology network and proposed a transmission scheduling algorithm. The limitation of these algorithms is that they are not flexible enough to change with the underwater environment. In recent years, the studies on resource allocation in wireless sensor networks using reinforcement learning algorithms received considerable interest [23; 24]. In UWANs, most of the reinforcement learning-based works focus on routing protocols [25; 26]. The studies on the underwater MAC protocol based on reinforcement learning are in their infancy. In [27], the goal was to redesign a protocol (UW-ALOHA-Q) based on the established RL-based protocol (ALOHA-Q). UW-ALOHA-Q was more suitable for UWANs. UW-ALOHA-Q algorithm used the Q-learning algorithm to learn optimal action. However, Q-learning used the Q-table to store the Q-values of action-state pairs. Like [13], UW-ALOHA-Q also faced the computation expense problem. Different from the above algorithms, we propose a deep reinforcement learning-based scheduling algorithm (called DRLSA-IA). DRLSA-IA learns transmission scheduling as nodes constantly interact with the environment. Therefore, it can adapt to the highly dynamic underwater environment. DRLSA-IA does not need to enumerate the state space. Thus, it is not subject to computationally expensive constraints. We compare DRLSA-IA and classical related algorithms in Table 1. Comparison of typical relevant works.