# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
###### Abstract  
Unmanned aerial vehicle (UAV) networks have a wide range of applications, such as in the Internet of Things (IoT), 5G communications, and so forth. However, the communications between UAVs and UAVs to ground control stations mainly use radio channels, and therefore these communications are vulnerable to cyberattacks. With the advent of software-defined radio (SDR), smart attacks that can flexibly select attack strategies according to the defender's state information are gradually attracting the attention of researchers and potential attackers of UAV networks. The smart attack can even induce the defender to take a specific defense strategy, causing even greater damage. Inspired by symmetrical thinking, a solution using a software-defined network (SDN) to combat software-defined radio was proposed. We propose a network architecture which uses dual controllers, including a UAV flight controller and SDN controller, to achieve collaborative decision-making. Built on the top of the SDN, the state information of the whole network converges quickly and is fitted to an environment model used to develop an improved Dyna-Q-based reinforcement learning algorithm. The improved algorithm integrates the power allocation and track planning of UAVs into a unified action space. The simulation data showed that the proposed communication solution can effectively avoid smart jamming attacks and has faster learning efficiency and higher convergence performance than the compared algorithms.  
U  
A _x_(1,2,*),the jamming attack has the lowest technical threshold, the least implementation cost, and the most direct attack effect. The jamming attack directly weakens or blocks the UAV network communication links, so it is urgent to address this kind of smart attack behavior.  
Inspired by symmetrical thinking, a solution using a software-defined network (SDN) to combat SDR may be the best choice. Both SDN and SDR are highly flexible and controllable and can deploy intelligence algorithms. Although the SDN architecture has some successful applications in MANETs (Mobile Ad-hoc NETworks) and VANETs (Vehicular Ad-hoc NETworks), due to the mobility of the UAV network and the rapid change of the network topology, it is necessary to design and deploy a new SDN architecture that meets the mission requirements and security requirements. Similarly, there are some successful experiences in deploying intelligent algorithms to the SDN on the ground. However, in deploying intelligent algorithms to UAVs, especially on UAV networks, a series of problems still need to be solved, such as communication process optimization and convergence optimization.  
The main contributions of this paper can be summarized as follows:  
* We propose a module-level SDN controller design for UAV networks;
* We propose a dual-controller cooperative SDN-based UAV network wireless communication scheme;
* A Dyna-Q-based reinforcement learning algorithm for power allocation and track planning collaborative optimization against smart jamming is proposed.  
The remainder of this paper is organized as follows. We review related work in Section 2. In Section 3, we design the topology architecture and functional architecture of the SDN-based UAV network, present the design of the module-level SDN controller, and establish the network model and jamming attack model. We propose an improved Dyna-Q-based smart defense communication solution in Section 4. Simulation results and discussion are presented in Sections 5 and 6.