## 1 Introduction  
Information communication, which involves sharing relevant information to fulfill certain activities, has taken various shapes and forms throughout human history [1]. However, with technological breakthroughs in the 20 th century, we humans have chosen many media through which to communicate between two entities [2; 3; 4]. The number of entities involved in the communication process has risen quite abruptly over the past three decades [5; 6]. To cater for all humans with communication possibilities, networks have been created to accommodate this never-ending sharing of data [7; 8]. There are two types of networks: wired and wireless. However, in this article, we focus on wireless communication [9]. Nowadays, WSNs play a key role in communication in all habitats: space communication between satellites and between satellites and base stations, underground communication, underwater communication, etc. [10; 11; 12]. This paper is devoted to UWSNs, in which various peculiar constraints like high propagation delays, limited bandwidth, and scalability of network topologies mean that a Q-learning-based MAC protocol can well handle the communication challenges faced by UWSNs. Among the solutions proposed to solve the MAC problem, Q-learning, from reinforcement learning methods, is extremely efficient and can become a tool to optimize the nodes to respond to changes in these harsh conditions, maximizing network performance in throughput, delay, and energy consumption. For instance, the authors in [13] design a Q-learning-optimized multi-receiver handshake scheme for mitigating packet loss and delay. The Q-learning algorithm constantly updates a Q-table for smart data transmission choices, significantly increasing delay and normalized throughput compared to traditional protocols. The authors in [14] achieve a similar goal by employing Q-learning to improve the utilization of the channel; that is, letting nodes learn how to transmit packets in an optimal way even if they do not have enough knowledge about the network topology. As a consequence, increased network channel utilization benefits from 13 to 26 percent higher throughput in different network environments. Regarding low energy [15], Q-learning improves energy consumption by minimizing collisions, which helps to increase network life and also optimizes MAC slot allocation for better efficiency. Over the years, much progress has been achieved in designing and developing UWSNs, which serve various purposes in different applications such as resource extraction, military surveillance, environmental monitoring, ocean pollution tracking, etc. [16]. And the progress in these new technologies has brought new marine and ocean research opportunities, which in turn is helping to develop a clearer understanding of the aquatic environment [17; 18]. However, the specific nature of underwater communications gives rise to profound barriers that must be overcome to unlock the full potential of UWSNs. Among the challenges, one of the critical issues is related to the ability of water to absorb radio frequency (RF) waves, which leads to the utilization of acoustic waves to transfer information in UWSNs [19]. This reliance on acoustics brings several technical issues, primarily due to the characteristics of acoustic channels/links. Traditional communication paths using acoustic technology in underwater situations have limited bandwidth and slow propagation. This delay in communication between nodes implies the non-reliability of the network [20]. Table 1 illustrates the comparison between different existing MAC protocols and the present research. Furthermore, there are numerous underwater-environment-specific factors, such as propagation conditions, that are continuously varying underwater [23]. Another very important factor affecting the whole communication process in UWSNs is the batteries' storage capacity, which is used to keep the sensor (Rx) nodes running while specific sensor nodes perform a given task remotely [24]. So, if the battery charging and discharging