# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 3 The Proposed Deep Multi-User Reinforcement Learning
### DMRL Training  
In each time slot, we will record \(<\)\(S\),\(A\),\(R\),\(S\)\(t\)\(>\) into the experience memory, where \(S\) represents state space, \(A\) represents action, \(R\) represents reward and \(S\)\(t\) represents the next state space. Extracting a batch of records from the experience cache, \(S\) and \(S\)\(t\) is normalized to calculate \(Q_{pred}\) and \(Q_{next}\) through \(Q\) value, and then \(Q_{actual}\) can be calculated according to:  
\[Q_{actual}=reward+\gamma\times Q_{next}. \tag{15}\]  
Utilizing the function provided by TensorFlow, the loss can be calculated and its gradient optimization can be performed to update the network parameters. The specific formulas are listed as:  
\[Q_{loss}=Reduce\_Mean(Square(Q_{actual}-Q_{pred})), \tag{16}\]  
\[opt=AdamOptimizer.minimize(Q_{loss}). \tag{17}\]  
To sum up, the specific process of the proposed deep multi-user reinforcement learning algorithm is presented in Algorithm 2. In order to achieve good sensing in multi-user scenarios, two networks with different number of users and channels are simulated. To achieve good results with lower time consuming, the number of iterations is set to 20 and the step size is set to 5000 individually. In the multi-user case, many studies [36] consider the case of 2 users and 3 channels. To verify the case for more users, we also consider the case of 10 users and 6 channels in the experiments. The specific training settings are shown in Table 2.  
Three quantitative indicators are applied to evaluate the performance of the dynamic cooperative spectrum sensing models, including the total reward, the cumulative collision volume and the cumulative reward of all users. The total reward represents the reward value after the user senses the channel in each time slot. If SUs successfully sensed channel, their reward value is set to 1. That is, the value of the total reward is proportional to the number of successful perception by SUs. For example, a total reward value of 0 means that no SUs successfully sense the channel and a total reward value of 4 means that 4 SUs successfully sense the channel. Considering \(r_{i}\) represents the reward, the total reward is:  
\[Total\_Rewards=\sum_{i=1}^{N}r_{i}. \tag{18}\]  
Cumulative collision indicates the cumulative amount of collision times as the time slot increases. When two or more SUs access the same authorized channel in the same time slot, collisions will happen. In general, the number of collisions is the number of users who sense access to the same authorized channel at the same time. The cumulative collision rate is defined as the ratio of the number of cumulative collisions to the cumulative sensing attempts. For example, if there are 3 SUs sensing channel 1 at the same time, the number of collisions is 3. Another way to express cumulative collisions is the total number of channels minus the total reward value. If the number of channels is 5 and the total reward value is 3, the number of collisions is 2. The cumulative collision is:  
\[Cum\_Collision=\sum{(K-Total\_Rewards)}. \tag{19}\] \begin{table}
\begin{tabular}{c c c} \hline \hline
**Symbol** & **Parameters** & **Value** \\ \hline \(K\) & Number of authorized channels & 2, 6 \\ \(num\_slots\) & Total number of time slots & 100,000 \\ \(N\) & Number of secondary users & 3, 10 \\ \(D\) & Experience cache size & 1000 \\ \(M\) & Batch size & 25 \\ \(h\) & Hidden layer size & 150 \\ \(st\) & State-space size & 6, 14 \\ \(\gamma\) & Discount factor & 0.9 \\ \(ac\) & Action space size & 3, 7 \\ \(\varepsilon\) & Exploration rate & 0.15 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Dueling deep \(Q\) network (DDQN) parameters. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Number of** & **Users** & **Number of** & & \\
**Channels** & **Users** & **Iterations** & **Step Count** & **Batch Size** \\ \hline
2 & 3 & 20 & 5000 & 25 \\
6 & 10 & 20 & 5000 & 25 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Experimental parameters.