## 1 Introduction
For example, the wireless canonical network, in which the channel states are time-varying and there is no information exchange among users, has been investigated in [18]. In addition, the mobility of UAVs also influences the receiver's signal energy, causing the decline of communication quality [19]. In [20], the authors investigated the multi-stage spectrum access problem for a flying ad-hoc network (FANET). The Markov decision process (MDP) [21; 22], as a decision framework under the dynamic channel environment, has been adopted widely. For the purpose of solving the MDP problem, Q-learning [23] methods are usually employed using a \"decision-feedback-adjustment\" structure to obtain the optimal strategy. For instance, in [21; 22; 23], the authors investigated the Markov decision process (MDP), Q-learning and the applications of MDP and Q-learning, but did not study the anti-jamming scenarios. In [4], the author formulated the anti-jamming decision problem as an MDP and obtained the best anti-jamming scheme via Q-learning. However, this work did not take the co-channel interference into consideration. In [24], the author dealt with the channel selection problem for cognitive networks via cooperative Q-learning, and in [25], a dynamic spectrum anti-jamming method in a fading environment was investigated. Furthermore, in [26], a deep Q network was built, and the anti-jamming channel selection problem was solved using a deep Q-learning method. These three works did not consider the multi-user case, which also ignored the existence of co-channel interference. In view of the multi-user scenarios in the anti-jamming field, the MDP problem has been extended to the Markov game [27], and several learning algorithms were designed for multi-user scenarios. Nevertheless, [27] did not concern the UAV communication networks where malicious jamming and co-channel interference existed simultaneously. In [18], a multi-agent learning algorithm was proposed to obtain a stable solution for the dynamic spectrum access problem, while it did not concern the attack of a malicious jammer. In [28; 29; 30], some multi-user Q-learning methods were adopted, where users took actions independently. However, in those methods mentioned in [28; 29; 30], users' states were influenced by each other, which led to unsteady learning environments and poor decision effects. In [31], the author studied the collaborative multi-agent reinforcement learning anti-jamming approach, with no concern about the feature of mobility in UAV communication networks. Taking an overall consideration of the challenges and inspirations brought by the above studies, in this paper, we mainly focus on the anti-jamming channel selection problem under the dynamic environment, where the channel state and UAVs' locations are time-varying. We also investigate two different threats to UAVs: malicious jamming and co-channel interference. The malicious jamming refers to the jamming attack launched by enemies or the opponent, and if the jamming signal occupies a channel, users are not able to communicate in this channel anymore. The co-channel interference refers to the interference signal caused by other users when they are communicating in the same channel. If there exists co-channel interference, users can still communicate via a CSMA or TDMA pattern. Malicious jamming has the purpose of destroying or degrading the communications of users subjectively, while co-channel interference does not have subjectivity. Motivated by [14], we define the user's utility as the trade-off between its throughput and cost. Considering the influence of co-channel interference and malicious jamming simultaneously, we firstly design the users' throughput. While considering the cost of channel switching and the cooperation among users, which have a great impact on the users' utilities, we introduce the channel switching cost unit and cooperation cost unit. Moreover, in our paper, a cooperative anti-jamming mechanism is constructed, in which users can realize information sharing and take actions jointly.