# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
## 4 Improved Dyna-Q-Based Smart Defense Communication Solution  
Artificial intelligence can make the UAV network smarter, thus resisting SDR-based smart attacks. The Dyna-Q-based algorithm is a model-based reinforcement learning technique. If we do not consider the limitations of calculation resources, a deep reinforcement learning technique (e.g., DQN (Deep Q Network), fast-DQN) can be adopted to tackle large state space and action space problems, as convolutional neural networks (CNNs) have powerful feature extraction capacity. Unfortunately, it is very uneconomical to deploy the large amount of calculation resources that CNNs need on UAVs. A practical method is to use a grading method that divides continuous values into several quantitative levels to compress the state space and the action space, and then attempt to speed up the convergence of the algorithm. The SDN-based UAV network has strong environmental awareness and control flexibility, and the error between radio channel models and the real radio environment is easily eliminated by multiple iterations. Therefore, we chose the Dyna-Q technique to calculate a better smart defense strategy of the UAV network. The interaction between UAVs and jammers can be formulated as a multistage repeated game. To derive the optimal communication, Dyna-Q-based reinforcement learning techniques can be used to control both the UAV power allocation policy and UAV relocation policy. In the proposed algorithm, the main function of the SDN controller is to aggregate the SINR values of the UAV nodes of the whole network, fit a quadric equation with these SINR values and their GPS coordinates, and use the surface equation to predict SINR values of the eight spatial grids that surround the UAV. Compared to the conventional Dyna-Q algorithm, the proposed algorithm uses global information to predict the SINR values in eight directions around each UAV and uses the predicted values to optimize the Q function. Upon receiving packets from other UAVs, the receiving UAV node extracts the SINR estimated by the sending UAV node and formulates the state as \(\mathbf{s}^{(k)}=\left[\text{SINR}^{(k)},L_{u}^{(k)}\right]\in S\), where \(L_{u}^{(k)}=\left(x_{u}^{(k)},y_{u}^{(k)}\right)\) represents the position coordinates of the sending UAV required by GPS and \(S\) is the state set. The receiving UAV node adopts a Dyna-Q-based reinforcement learning algorithm to choose the transmit power \(P_{s}^{(k)}\) and determines whether to launch the track relocation action with the communication action denoted by \(a^{(k)}=\left[P_{s}^{(k)},L_{st}^{(k)}\right]\in\mathcal{A}\), where \(L_{st}^{u(k)}\) is UAV \(u\)'s flight direction approximated by eight spatial grids and \(\mathcal{A}\) is the action space. In the process of communication, the sending UAV node evaluates the SINR from the feedback packets and calculates the utility based on the SINR and the communication cost, including the cost of frequency hopping \(C_{h}\), the cost of data transmitting \(C_{p}\), and the cost of UAV path replanning \(C_{m}\); the utility at time slot \(k\)\(r^{(k)}\big{(}\mathbf{s}^{(k)},\mathbf{a}^{(k)}\big{)}\) is given by  
\[r^{(k)}\big{(}\mathbf{s}^{(k)},\mathbf{a}^{(k)}\big{)}=\text{S}\hat{\text{I}} \hat{\text{N}}\hat{\text{R}}^{(k)}-C_{p}P_{s}^{(k)}-C_{m}\mathcal{F}\Big{(}L^{( k)}-L_{00}^{(k)}\Big{)}-C_{h}\mathcal{F}\Big{(}f^{(k)}-f^{(k-1)}\Big{)} \tag{2}\]  
where \(\mathcal{F}(. )\) is an bool function that equals 0 if the argument of \(\mathcal{F}(. )\) equals 0; otherwise, \(\mathcal{F}(. )\) equals 1. The value function denoted by \(V(\mathbf{s})\) stands for the maximum value of the \(Q\) function.