# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
## 5 Simulation Results  
Simulations are carried out to appraise the performance of the proposed power allocation policy and trace relocation policy against a smart jammer with a Dyna-Q-based reinforcement learning algorithm. Simulation parameters similar to those used previously [21; 24] are chosen, with \(\alpha=0.95\), \(\gamma=0.7,d_{0}=10m\), \(\varepsilon=0.9\), \(C_{m}=0.8\), \(C_{p}=0.2\), \(C_{h}=0.4\), \(\rho_{free-space}=2\), \(\rho_{two-nuy}=4\), \(T=300\), \(N_{l}=3\), \(N_{l}=2\), \(N=3\), and \(\omega=5\). The simulated flight area is 1000 km \(\times\) 800 km. The horizontal and vertical position coordinates of the moving objects are the remainders devided by 1000 km and 800 km, respectively. The initial position coordinates of the three UAVs are (260,610), (790,110), and (520,270), respectively. The initial position coordinates of the two jamming UAVs are (320,360) and (450,100). The mobility model of the jammer is a random waypoint model. The length of all spatial grids is 1 km. The speed of all the UAVs is 50 km/h. The maximum number of relocating UAVs in one time slot is 1. When calculating the channel gains via Equation (1), the distance \(d\) is calculated from the position coordinates after the remainder operation. We use four algorithms to form the smart defense algorithm of the UAV network. The four algorithms are the WoLF-PHC (Win or Learn Fast-Policy Hill-Climbing) algorithm [21], Q-learning algorithm [25], Dyna-Q algorithm [26], and our improved Dyna-Q algorithm. We performed 100 time  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Algorithm: Dyna-Q-based UAV Networks’ Smart Defense Communication** \\ \hline
**1** & Initialize \(\gamma\), \(\alpha\), \(\varepsilon\), \(\mathrm{SINR}^{(0)}\), \(T\) and model \((\mathbf{s},a)\) for all \(\mathbf{s}\in S\) and \(\mathbf{a}\in\mathbf{A}(\mathbf{s})\) \\
2 & \(\mathbf{for}=1,\cdots,T\mathbf{d}\)do \\
3 & \(L_{u}^{(0)}\leftarrow\) get the coordinates \((x_{u},y_{u})\) of UAV \(u\) by GPS \\
4 & \(\mathbf{s}^{(1)}=\left[\mathrm{SINR}^{(0)},L_{u}^{(0)}\right]\) \\
5 & \(\mathbf{if}=1\), then \\
6 & \(\mathbf{s}\leftarrow\mathbf{s}^{(1)}\), \\
7 & \(\mathbf{or}\mathbf{else}\) \\
8 & \(\mathbf{s}\leftarrow\) current state \\
9 & \(\mathbf{end}\) \\
10 & \(A\leftarrow\varepsilon\)-greedy\((\mathbf{s},Q)\) \\
11 & Execute action \(A\), observe reward \(R\), and state \(\mathbf{s}^{\prime}\) \\
12 & \(\mathbf{if}\mathbf{\,}L_{ll}\
eq L_{00}\), then \\
13 & Report SINR and reward R to SDN controller \\
14 & Update local cyber attack overall map \\
15 & \(\mathbf{end}\) \\
16 & \(Q(\mathbf{s},A)\gets Q(\mathbf{s},A)+\alpha\Big{[}r(\mathbf{s},\mathbf{a}) +\gamma\max_{\mathbf{a}}Q(\mathbf{s}^{\prime},\mathbf{a})-Q(\mathbf{s},A) \Big{]}\) \\
17 & Model \((\mathbf{s},A)\gets r,\mathbf{s}^{\prime}\) \\
18 & \(k=1,\cdots,n\)do \\
19 & \(\mathbf{s}\leftarrow\) random previously observed state \\
20 & \(A\leftarrow\) random action previously taken in \(S\) \\
21 & \(R,\mathbf{s}^{\prime}\leftarrow\) Model \((\mathbf{s},A)\) \\
22 & \(Q(\mathbf{s},A)\gets Q(\mathbf{s},A)+\alpha\Big{[}r(\mathbf{s},\mathbf{a}) +\gamma\max_{\mathbf{a}}Q\Big{(}\mathbf{s}^{\prime},\mathbf{a}\Big{)}-Q( \mathbf{s},A)\Big{]}\) \\
23 & \(\mathbf{end}\) \\
24 & Receive overall cyberattack map from SDN controller \\
25 & Calculate the parameters of the fitted SINR quadratic surface as \\
26 & \(\mathbf{for}\)\(s,t=-1,0,1\)do \\
27 & \(\mathbf{s}\leftarrow\left[\mathrm{SINR}_{SDN},L_{sl}\right]\) \\
28 & \(A\leftarrow\left[L_{sl},\underset{\mathbf{a}}{\mathrm{argmax}}V(\mathbf{s}) \right]\) via Equation (4) \\
29 & \(R,\mathbf{s}^{\prime}\gets r(\mathbf{s},A)\) \\
30 & \(Q(\mathbf{s},A)\gets Q(\mathbf{s},A)+\alpha\Big{[}r(\mathbf{s},\mathbf{a}) +\gamma\max_{\mathbf{a}}Q\Big{(}\mathbf{s}^{\prime},\mathbf{a}\Big{)}-Q( \mathbf{sslot simulations for each algorithm on a computer with 3.6 GHz Intel Core i7-4790 and 8 GB of RAM, and each time slot contains 30 episodes.