# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 1 Introduction
Most methods mentioned above are constrained to consider only one user instead of the real multi-user or multiple channels environment, in which collisions are prone to occur. Noticing this, a completely distributed model [35] is adopted for SUs to conduct dynamic multi-user spectrum access. Deep multi-user reinforcement learning method [36] is developed for DSA. It can effectively depict the general complex environment, while overcoming the expensive computational cost caused by the large state space and inefficient observability. Based on Q learning, the spectrum access scheme is proposed in [37], which can allocate idle spectrum holes without interaction between users. The multi-agent reinforcement learning (MARL) is proposed in [38] based on Q learning, which can effectively resist interference and obtain the good sensing results. In this model, each radio treats the behavior of other radios as a part of the environment. Then each radio attempts to evade the transmissions of other wideband autonomous cognitive radios (WACRs) as well as avoiding a jammer signal that sweeps across the whole spectrum band of interest. However, it is different to obtain a stable policy in changeable and complicated environment. The multi-agent deep reinforcement learning method (MADRL) is proposed in [39]. To avoid interference to the primary users and to coordinate with other secondary users, each SU decides its own sensing policy from the sensing results of the selected spectra. The method in [39] applies a channel sorting algorithm based on RL to decrease the number of sensing operations and spectrum handoffs. Compared with traditional RL models, MADRL has the advantages in the reward performance and the convergence speed. In summary, [27; 33] only assume one SU in the network with perfect spectrum sensing outcomes. This is, there is no error in spectrum sensing. On the other hand, [34; 35] consider multi-user access, but [34] does not perform spectrum sensing for multi-user in a time-varying environment and [35] needs to improve performance in convergence speed and collision rate. [PERSON] et al. [36] assumes no PU in the network and the collision with PUs is not considered. MARL is no longer applicable when the number of states is large because it does not consider the coordination among users and reduce the computational complexity. MADRL is simulated in channels where the number of channelsgrows exponentially with the increasing number of users. Meanwhile, it simply takes sensing reward into account. Most mentioned DRL models were \"Sequential\" architectures. That is, a layer could be only connected to the neurons in just one layer before and one layer after its own layer. Utilizing non-sequential architecture of deep learning, dueling deep Q learning (DDQN) [40] achieves the learning process of the Q-learning by exploring separate estimates of the state value function and the state-dependent action advantage function. In general, for dynamic cooperative spectrum sensing, it is necessary to consider the DSA environment where PUs and SUs appear randomly, which is much closer to the real scenarios. In this paper, deep multi-user reinforcement learning is proposed to effectively achieve dynamic cooperative spectrum sensing. After investigating communication strategies of the cooperative DSA environment, ACK signals are utilized to achieve partially observable setting. As the cooperative strategy, ACK signals are then introduced into DDQN to construct the proposed DMRL model, which allows users to learn good policies and achieve strong sensing performance. The main contributions of the paper are presented as follows. We propose a cooperative strategy for the DSA networks, without the need to exchange user spectrum information, online coordination, or carrier sensing. In the network, we consider cognitive MAC protocols for the secondary network with \(N\) users sharing \(K\) orthogonal channels.