# Distributed Learning for Dynamic Channel Access in Underwater Sensor Networks
## 1 Introduction
The authors of [54] proposed a DRL-based multiple access protocol which maximizes the occupation of available time slots caused by long propagation delay or not used by other nodes. Despite the extensive amount of existing studies, including the works in [46; 47; 48; 49; 50; 51; 52; 53; 54] on dynamic channel access for UASNs, there is no existing work that considers autonomous channel access with no coordination between underwater sensors and utilizing the prediction information of link qualities of available acoustic channels at the same time. Hereafter, we focus on the detailed data communication scenario from each underwater sensor to its associated data sink (e.g., receiver) in UASNs to explicitly clarify the problems addressed in this work. In UASNs, when transmitting data from underwater sensors to a data sink, two approaches might be considered to manage the channel access between the sensors: The first approach is one wherein a data sink allocates the available acoustic channels to its associated underwater sensors in a centralized manner. However, generally, the data sink does not feature the functionality of radio resource management, unlike a base station in cellular networks [29]. The second approach is one wherein underwater sensors associated with the same data sink can cooperate with each other to share the channels; however, it is impractical to coordinate or exchange messages between the sensors to manage channel access in general UASNs [55]. In addition, it is difficult for each sensor to predict the link qualities of available acoustic channels because of the influence of a variety of factors such as pressure, density, salinity, and temperature, compared to terrestrial radio frequency channels. To overcome the limitations of UASNs described above, our idea is that each underwater sensor enables the simultaneously learning of both the behaviors (i.e., channel choices) of other sensors and the time-varying dynamics of available acoustic channels in a distributed manner, through learning the relationship between its action choice and the corresponding reward (i.e., RL). For this, we formulate a dynamic channel access problem of UASNs as a multi-agent Markov Decision Process (MDP). In particular, for reflecting the above idea, we design a state space of each sensor, which includes its previous action (i.e., previous channel choice), the estimation of channel error probabilities for all available channels, and two-bit feedback information corresponding to the previous action from its associated data sink. Note that adopting the two-bit local feedback mechanism enables each sensor to estimate the channel error probabilities of available acoustic channels itself. Then, we propose a distributed DRL-based algorithm, under which each sensor can choose the proper channel while avoiding not only the same channel selection with other sensors, but also the channels with the bad channel qualities without any coordination of other sensors. We summarize the contributions of this work below. * We formulate the dynamic channel access problem in UASNs as a multi-agent MDP, wherein each underwater sensor is considered an agent whose objective is to maximize the total network throughput without coordinating with or exchanging messages among underwater sensors. * We propose a dynamic channel access algorithm for UASNs, based on deep Q-learning. In the proposed algorithm, each agent (i.e., underwater sensor) exploits partial information, i.e., only the feedback information between a data sink and that particular underwater sensor instead of complete information on the actions of all other agents, to learn not only the behaviors (i.e., actions) of the other sensors but also the physical features, i.e., channel error probability (CEP) of its available acoustic channels. This property ensures that each underwater sensor can implement the proposed algorithm in a distributed manner, i.e., there is no need for cooperation between agents.