# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 3 The Proposed Deep Multi-User Reinforcement Learning  
To solve the multi-user conflict problem, the multi-user DDQN framework is proposed by integrating cooperative sensing with DDQN. Consider an environment with \(K\) related channels, and each channel has two possible states: idle (1) or busy (0). SU can dynamically  
Figure 3: The framework of deep Q network (DQN).  
access these \(K\) authorized channels and select one channel for communication. Based on this, channel exchange can be modeled as a Markov chain of up to \(2^{K}\) states. Because the environment is unknown to the user, the user can only make a decision after each attempt to perceive a different channel. In the proposed model, \(N\) users can dynamically access \(K\) shared channels. At the beginning of each time slot, each user selects a channel and sends a data packet with a certain trial probability. After each time slot, each user who has sent a data packet receives an observation indicating whether its data packet was successfully sent (i.e., \(ACK\) indicator). If the data packet is delivered correctly, the reward is 1. Otherwise, the collision occurs if the transmission fails, and the reward is set to 0. The model parameters are as follows.  
* \(S\) stands for state space. It is a matrix with dimension \(N\times(2K+2)\), which expresses the state set of \(N\) users, and its expression is shown as: \[state=\left[\begin{array}{c}s_{1}\\ \ldots\\ s_{N}\end{array}\right]\}N.\] (4)  
Among them, \(s_{i}(1\leq i\leq N)\) represents the status of the \(i\)-th user, and the status of each user consists of \(2K+2\) elements. The first \(K+1\) elements represent the user's transmission status (If the user does not send, the first element is 1 and the other elements are 0; If the user selects channel \(K\) for transmission, the \(K+1\) element is 1 and other element values are 0). The \(c_{1}\) to \(c_{k}\) represent the remaining capacity of \(K\) channels. The last element represents the \(ACK\) signal (When the \(ACK\) signal is received, the value is 1; Otherwise, the value is 0). Therefore, \(s_{i}\) is expressed as:  
\[s_{i}=[\xi_{i},\ldots,\xi_{K+1},c_{1},\ldots,c_{K},p_{i}]. \tag{5}\]  
Among them, \(\xi_{i}\) represents the selection of the \((j-1)th\) channel for transmission; \(c_{j}\) represents the remaining amount of the \(j\)-th channel; \(p_{i}\) represents the \(ACK\) signal.  
* \(A\) represents the action set. The actions of \(N\) users in each time slot form the action matrix, which is expressed as: \[action=[a_{t}^{1},a_{t}^{2},\ldots,a_{t}^{N}],\] (6) where \(a_{t}^{i}\in\{1,\ \ldots,j,\ldots,K\}\), indicating user \(i\) selects channel \(j(1\leq j\leq N)\) for accessing.
* \(O\) represents the observation space. It is mainly composed of the \(ACK\) signal, instant reward, and the remaining capacity of the channel, which are listed as: \[obs=[(p_{1},r_{1}),\ldots,(p_{i},r_{i}),\ldots,(p_{N},r_{N}),[c_{1},\ldots,c _{j},\ldots,c_{K}]].\] (7) Among them, \((p_{i},r_{i})\) represents the \(ACK\) signal and instant reward obtained by user \(i\) after accessing the channel \(a_{t}^{i}\). Symbol \(c_{j}\) represents the remaining capacity of the channel \(j\).
* \(R\) represents the reward space. It is a matrix with a dimension of \(1\times N\), representing the reward set of \(N\) users. Its value is shown as: \[reward=[r_{1},\ldots,r_{i},\ldots,r_{N}],\] (8) where \(r_{i}\) represents the reward obtained by selecting the \(i\)-th user.  
In this paper, dueling DQN network (DDQN) is utilized to understand which states are valuable and get better sensing strategies. The channel state with the cooperative signal is learned as the input of the network. Then the dynamic multi-channel cooperative strategy is obtained by training the DDQN network.