# Distributed Learning for Dynamic Channel Access in Underwater Sensor Networks
## 4 Background on Q-Learning and Deep Reinforcement Learning  
In this section, we present a brief background on Q-learning and DRL, which will be utilized to develop the proposed algorithm in next section. To avoid clutter of indices, we assume a fixed typical sensor (say sensor \(u\)) and drop the sensor indices for all the parameters. Q-learning is a reinforcement learning algorithm that can help find the best policies (i.e., a sequence of actions over time) for dynamic programming problems. Because the expected reward can be calculated from among the available actions without prior knowledge on the environment, Q-learning has been widely adopted in a variety of decision-making applications. More specifically, we denote \(Q_{\pi}\left(\mathbf{s},a\right)\) as the Q-function given policy \(\pi\) where a state-action pair \(\left(\mathbf{s},a\right)\) is considered as a variable. The Q-function is defined as the sum of discounted rewards achieved when action \(a\) is taken in the initial state \(\mathbf{s}\) under policy \(\pi\), and it can be presented in a recursive form:  
\[Q_{\pi}\left(\mathbf{s},a\right)=r\left(\mathbf{s},a\right)+\delta\sum_{ \mathbf{s}^{\prime}\in\mathbf{S}}\sum_{a^{\prime}\in\mathbf{A}}P_{\mathbf{ss }^{\prime}}^{a}Q_{\pi}\left(\mathbf{s}^{\prime},a^{\prime}\right), \tag{8}\]  
where \(\delta\in\left[0,1\right]\) is the discount factor determining the effect of the future rewards, \(P_{ss^{\prime}}^{a}\) is the state transition probability from state \(\mathbf{s}\) to state \(\mathbf{s}^{\prime}\) by action \(a\), and \(\left(\mathbf{s}^{\prime},a^{\prime}\right)\) is the next state-action pair when an agent executes action \(a\) in state \(\mathbf{s}\). The agent aims at finding the optimal policy \(\pi^{*}\left(\mathbf{s}\right)\) that maximizes (8) for each state \(\mathbf{s}\). We denote \(Q_{\pi^{*}}\left(\mathbf{s},a\right)\) as the Q-function for the state-action pair \(\left(\mathbf{s},a\right)\) under the optimal policy \(\pi^{*}\left(\mathbf{s}\right)\), such that (8) can be rewritten as  
\[Q_{\pi^{*}}\left(\mathbf{s},a\right)=r\left(\mathbf{s},a\right)+\delta\sum_{ \mathbf{s}^{\prime}\in\mathbf{S}}\sum_{a^{\prime}\in\mathbf{A}}P_{\mathbf{ss }^{\prime}}^{a}Q_{\pi^{*}}\left(\mathbf{s}^{\prime},a^{\prime}\right), \tag{9}\]  
where an optimal policy can be obtained as \(\pi^{*}\left(\mathbf{s}\right)=\arg\max_{a\in\mathbf{A}}\left[Q_{\pi^{*}} \left(\mathbf{s},a\right)\right],\;\;forall\mathbf{s}\in\mathbf{S}\). However, for some systems, we cannot calculate \(Q_{\pi^{*}}\left(\mathbf{s},a\right)\) from (9), because the state transition probability \(P_{\mathbf{ss}^{\prime}}^{a}\) is practically unknown to the agents. Q-learning has been widely adopted as an alternative [56], as it is one of the representative model-free algorithms. Q-learning constructs a \(\left|\mathbf{S}\right|\times\left|\mathbf{A}\right|\) Q-table, each element of which is a Q-value. The agents can update each element \(Q\left(\mathbf{s},a\right)\) in the Q-table using the following equation,  
\[Q\left(\mathbf{s},a\right)=\left(1-\alpha\right)Q\left(s,a\right)+\alpha \left[r\left(\mathbf{s},a\right)+\delta\max_{a^{\prime}\in\mathbf{A}}Q\left( \mathbf{s}^{\prime},a^{\prime}\right)\right], \tag{10}\]  
where \(\alpha\) is the learning rate. Note that Q-learning works well if the state-action space is small, whereas it becomes impractical if the size of the state-action space increases, because of two reasons: First, many state-action pairs in the state-action space are visited very rarely, which degrades the performance considerably. Second, we need a Q-table that can hold all Q-values corresponding to the state-action pairs in the space, which might make the storage complexity intolerable. To overcome these issues, a potential was proven by DRL that combines deep neural network (DNN) with Q-learning, which is also called Deep Q-Network (DQN) [57].