# Dynamic Cooperative Spectrum Sensing Based on Deep Multi-User Reinforcement Learning
## 1 Introduction  
Due to the increasing demand for wireless communication, spectrum resources become severely scarce, which has promoted demand of developing high-efficiency dynamic spectrum access (DSA) schemes [1; 2; 3]. To alleviate the existing conflict between increasing spectrum demand and spectrum shortage, DSA allows secondary users (SUs) to opportunistically search and use licensed frequency bands or channels that are not fully utilized by primary users (PUs). Conventional DSA schemes usually focused on model-related settings or near-sighted targets, including short-sighted strategies [4], multi-arm gambling strategies [5; 6], and Whittle index strategies [7; 8]. They are performed under the premise that the channels are independent of each other or the channels follow the same distribution, which ignore the impact of the high correlation between channels. In recent years, many studies have begun to focus on the impact of prior knowledge and channel correlations on multichannel perception. Most conventional studies on channel selection strategies require network status information, but this may be impractical [9; 10; 11]. Machine learning has been applied to the field of dynamic multi-channel perception [12; 13; 14]. As an important branch of machine learning, reinforcement learning (RL) is characterized by frequently interacting with changing environments to gain knowledge, which shows excellent performance in processing dynamic systems [15]. In reinforcement learning, each SU acts as an agent; and the agent who takes actions interacts with the external environment through a reward mechanism and adjusts the action according to the reward value obtained in the environment [16; 17]. In other words, RL allows inexperienced agents to continuously learn through trial-and-error, andit maximizes the reward function to obtain the best strategy. So, RL-based models do not require the complete network status information. Q-learning, one of the algorithms of RL, has been used to depict the SU behavior in cognitive networks [18; 19; 20; 21]. However, Q-learning performs well on small scale models but becomes extremely inefficient when dealing with large state spaces [22]. Deep reinforcement learning (DRL), which uses neural networks to approximate state values, has attracted much attention. It can effectively deal with large dynamic unknown environments [23; 24; 25; 26], including very large state and action space. In recent years, DRL has made many breakthroughs in dynamic spectrum access due to its intrinsic advantages. Reference [27] can be considered as the pioneer work of exploring the DRL. It proposes a dynamic channel access scheme based on the deep Q network (DQN) with experience replay [28]. In each slot, the user selects one of \(M\) channels to transmit its packet. As the user knows the channel state only after the channel is selected, the related optimization decision problem can be expressed as a partially observable Markov decision process (POMDP). DQN can easily search the best strategy from experience [29; 30; 31], so it has been widely utilized in DSA. DQN [32] is explored to jointly address the dynamic channel access and interference management. In [28], the DQL keeps following the learned policy over time slots and stops learning a suitable policy. However, actual environments are dynamic, in which DQN needs to be re-trained. To address the problem, an adaptive DQN scheme [33] is proposed by evaluating the cumulative incentives of current policies in each period. When the reward decreases to less than a given threshold, the DQN will be retrained to find the new good policies. The simulation results show that when the channel state changes, this scheme can detect the changes and start relearning to get high returns. In [34], a DRL framework based on actor-critics is proposed, which can get a better performance in a relatively large number of channels.