# Distributed Learning for Dynamic Channel Access in Underwater Sensor Networks
## 6 Performance Evaluations
### Baseline Schemes  
For a comprehensive performance evaluation, we compare the performance, given in terms of total network throughput, of the proposed algorithm achieved from Algorithm 1 against those of three algorithms: the optimal algorithm, the random algorithm, the exact CEP algorithm. Similar to the work in [65], the optimal algorithm performs optimal channel allocation that maximizes total network throughput in a centralized manner. For this, it should be assumed that the data sink has the functionality of radio resource management and notifies the channel allocation result to its associated sensors. On the contrary, following the random algorithm [66], each underwater sensor chooses its transmission channel among available channels at each time slot in a fully distributed manner. That is, there is no need for a feedback process. The exact CEP algorithm is based on the proposed algorithm with a slight difference as follows. In the exact CEP algorithm, it is assumed that each underwater sensor can know the exact CEPs for all available channels at each time slot and utilizes those in the  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Network Parameter** & **Value** \\ \hline Number of active sensors and data sink & 2, 1 \\ Surface height (depth) & 100 m \\ Height of sensors and data sink & 10 m \\ Transmit power of sensors & 20 W \\ Number of available acoustic channels & 3 \\ Minimum frequencies of available channels & \([10,30,50]\) KHz \\ Bandwidth of each channel & 10 KHz \\ \hline \hline \end{tabular}
\end{table}
Table 1: List of network parameters.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Agent** \\ \hline Batch size & 6 \\ Optimizer & Adam \\ Activation function & Relu \\ Learning rate & \(10^{-4}\) \\ Experience replay size & 1000 \\ Discount factor \(\eta\) & 0.99 \\ \hline \hline \end{tabular}
\end{table}
Table 2: List of DQN hyperparameters.  
state space, and then chooses an action through Algorithm 1. For this, before transmitting sensing data, signal exchanges, such as the pilot transmission from each sensor to the data sink for all available channels, the link qualities estimation of the channels between the sensors and the data sink, and the feedback mechanism to report the results of the estimation, are needed. On the other hand, in the proposed algorithm, each sensor utilizes the estimated CEPs in the state space, which are computed by using only feedback information transmitted from the data sink, as described in (2).  
Figure 3 shows the performance of the proposed algorithm under a change in the number of available channels \(|K|\). In the figure, the x-axis represents an episode consisting of 500 timesteps and the y-axis represents the moving average of the total network throughput average achieved during an episode. From Figure 3, it can be identified that, regardless of \(|K|\), when the proposed learning algorithm is executed at each sensor, the total network throughput improves and then converges at specific values over time. The performance of the proposed algorithm under \(|K|=1\) is much lower than those under \(|K|=2\) and 3. This is because, in the case of \(|U|>|K|\), the probability of collision between underwater sensors increases and the total network throughput decreases, accordingly. In the case where sufficient channels are assigned to the sensors, i.e., if \(|U|\leq|K|\), the performance of the proposed algorithm remains almost the same, regardless of \(|K|\). This is because, although \(|K|\) increases, the probabilities of collision will be same.