# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 3 The Proposed Deep Multi-User Reinforcement Learning
### Simulation Results  
In this section, the proposed framework is simulated and analyzed by different experimental setting. Firstly, with the fixed number of authorized channels, the total number of successful packet transmissions are observed as the training time slot increases. Secondly, the total number of successful packet transmissions are analyzed when the number of users significantly increases. Finally, MARL and MADRL are chosen as compared approaches to verify the performance of the proposed DMRL. Figure 5 shows the values of the total reward, the cumulative collision and the cumulative reward obtained by DMRL, in which the number of authorized channels is 2 and the number of users is 3 at the beginning of training. From Figure 5a, it can be easily found that the total reward value is mainly distributed between 0 and 1. This result shows that in most cases, no user successfully accesses the channel or only one user successfully accesses the channel. As shown in Figure 5b, when the number of authorized channels and the number of users remain unchanged, the distribution of the total reward will change as the number of training increases. Most values of the total reward are 1. Comparing Figure 5a,b, the cumulative collision rate decreases, and the cumulative reward increases. It can be found that in the first 5000 time slots, the cumulative collision volume is high and the cumulative reward is low. The main reason is that the neural network with fewer training times can't efficiently learn the strategy of accessing the channel. After training about 100,000 times, the experimental results are shown in Figure 6. It can be easily found that most values of the total reward are 2, which means that two authorized channels can be well allocated when being occupied by three users, and there are few conflicts. The cumulative collision rate has also been reduced from 0.7 to 0.02, which greatly reduces the probability of conflict. The cumulative reward has also increased from 2500 to 10,000. From these indicators, it is quite obvious that the proposed DMRL can effectively solve the problems of multi-user DSA problem, while reducing the collision rate and maximizing the success rate of channel access. Figure 5: When \(K=2\) and \(N=3\), the total reward, cumulative collision, and cumulative reward for (**a**) the first 5000 time slots and (**b**) 50,000 to 55,000 time slots. Figure 7 shows the scenario with 6 authorized channels and 10 users. It can be easily observed that the total reward increases as the increase in training slots, i.e., the reasonable number of authorized channels is increased, but the rate of increase is not fast. It can be obviously found from Figure 7a that the total reward value is roughly distributed around a lower value at the beginning of training, and the cumulative collision volume remains in a high-value range. However, the total cumulative reward remains in a low range. The reason is the same, i.e., too few channel perception records are used to train the neural network. As can be seen in Figure 7b, as the number of training increases, the multi-user cooperative sensing strategy learned by the neural network becomes more and more accurate, and the total reward value gradually increases, but the degree of optimization is relatively slow. However, the cumulative collision volume is still not low. This is because the increase in the number of authorized channels and the number of secondary users will increase the state space, and more difficult problems will take longer to resolve. When the number of training reaches a certain level, the experimental results are shown in Figure 7c,d. Although the total reward distribution remains in the upper middle part, turbulence and unstable points appear. From the cumulative amount of collision maintained at 8000, and the cumulative reward value maintained at 20,000, it can be seen that the amount of knowledge learned by the neural network due to the complexity of the problem has stagnated. This will be one of the problems to be solved in the future.