# Distributed Learning for Dynamic Channel Access in Underwater Sensor Networks
## 3 Problem Formulation with MDP  
In this section, we formulate a dynamic channel access problem for each underwater sensor, as an MDP. To determine the channel that each underwater sensor accesses in each time slot, we define an MDP with a tuple (\(\mathbf{S}\), \(\mathbf{A}\), \(r\)), where \(\mathbf{S}\) is the state space, \(\mathbf{A}\) is the action space, and \(r\) is the reward. The details of these parameters are explained subsequently. State \(\mathbf{s}_{u}\in\mathbf{S}\) of sensor \(u\) can be expressed as a vector of size \(2K+2\), which is illustrated in Figure 2.  
The first \(K+1\) elements, which are presented as a one-hot vector of state \(\mathbf{s}_{u}\), stand for the action of sensor \(u\) executed at time \(t-1\). Specifically, if sensor \(u\) does not transmit to a data sink in the time slot \(t-1\), the first element is set to 1 and the remaining elements are set to 0. If the sensor transmits on channel \(k\) at time slot \(t-1\), the \(\left(k+1\right)\)-th element is set to 1, and the remaining elements are set to 0. The following K elements of state \(\mathbf{s}_{u}\) are the estimation of the CEP of each channel \(k\in K\) for sensor \(u\), conditioned on the event that the channel is idle. Note that the estimation of CEP on channel \(k\) for  
Figure 2: Illustration of state. sensor \(u\), denoted by \(\tilde{p}_{u}^{k}\left(t\right)\), is updated only after transmission on channel \(k\), by using the feedback information as follows,  
\[\tilde{p}_{u}^{k}\left(t\right)= \begin{cases}a\varphi_{u}^{k}\left(t-2\right)+ \left(1-\alpha\right)0,&\text{if }a_{u}\left(t-1\right)=k,\text{ }o_{u}\left(t-1\right)=0,\\ a\varphi_{u}^{k}\left(t-2\right)+\left(1-\alpha\right)1,&\text{if }a_{u}\left(t-1\right)=k,\text{ }o_{u}\left(t-1\right)=1,\end{cases}  \tag{2}\]  
where \(\alpha\in\left[0,1\right]\) is the moving rate and \(a_{u}\) is the action of sensor \(u\), which will be explained below. Note that when collision occurs (i.e., \(o_{u}\left(t\right)=2\)) on channel \(k\), the estimation of CEP is not updated. As mentioned before, this is because, when a collision occurs, the data sink will not be able to judge whether a channel error has occurred or not. The last element of state \(\mathbf{s}_{u}\) is the feedback information received after transmission at time slot \(t-1\), described in (1). An action \(a_{u}\in\mathbf{A}\) is an element of the action space \(\mathbf{A}\), which is the set of available actions that the underwater sensor \(u\) can choose, as given by  
\[\mathbf{A}\left(t\right)=\left\{0,1,2,...,K\right\}, \tag{3}\]  
where \(a_{u}\left(t\right)=0\) means that a sensor does not transmit a packet at time slot \(t\), and \(a_{u}\left(t\right)=k\) means that the sensor transmits a packet on channel \(k\) at time slot \(t\). Note that the reward of each underwater sensor depends on not only its action, but also other sensors' actions, which constitute the unknown network environment that each sensor must learn. As such, the action profile of sensor \(u\) for all sensors except itself at time \(t\) can be defined as follows,  
\[\mathbf{a}_{-u}\left(t\right)=\left\{a_{i}\left(t\right)\right\}_{i\
eq u}. \tag{4}\]  
Finally, we can define a reward function \(r\left(t\right)\) as follows,  
\[r_{u}(t)= \begin{cases}1,&\text{if }o\left(t\right)=0,\\ 0,&\text{Otherwise},\end{cases}  \tag{5}\]  
where underwater sensors that receive \(o\left(t\right)=0\) from the data sink can achieve a positive reward, i.e., \(r(t)=1\). With the help of (5), the total network throughput, which is the performance metric in this work, can be presented as follows,  
\[r_{net}\left(t\right)=\sum_{u\in U}r_{u}\left(t\right). \tag{6}\]  
As this work focuses on sequential decision making for dynamic channel access in UASNs, both the immediate and future rewards should be considered when making a decision.