# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
## 5 Simulation Results
For instance, at the 300 th time slot, the utility of improved Dyna-Q-based strategy is 1.1626, which is 8.7%, 14.9%, and 22.5% higher than that of the Dyna-Q, WoLF-PHC, and Q-learning-based strategies, respectively. The trend of SINR of the UAV network is basically the same as that of utility when facing smart jamming. At the 300 th time slot, the SINR with improved Dyna-Q is 3.1928, which is 3.4% higher than that of Dyna, 5.7% higher than that of WoLF-PHC, and 10.4% higher than that of the Q-learning strategy. To better observe the impact of changes in the jamming strategy, we averaged the utility and SINR within the same jamming strategy. The results are shown in Figure 7, Table 3, and Table 4. The average performance of the UAV network with the improved Dyna-Q strategy is optimal, the average performance of Dyna-Q and WoLF-PHC are lower and similar, and the average performance of the Q-learning algorithm is the worst. For instance, the mean value of nine average utilities in Table 3 with improved Dyna-Q is 1.0968, 0.9861 with Dyna, 0.9618 with WoLF-PHC, and 0.8380 with Q-learning. The performance advantage of the improved Dyna-Q algorithm is obvious, which is closely related to the situation awareness capabilities of the SDN architecture. The reason for the low performance of the Q-learning algorithm is that the fast-paced attack and defense strategy changes make the algorithm unable to converge in time. It is well known that the Q-learning algorithm can only update the Q value of one state per episode if the delay update mechanism is not used. This slow learning will inevitably lead to slow convergence. For example, in the 60-80 th time slots of Figure 6a, the utility of the Q-learning algorithm shows a continuous but slow growth. The phenomenon of nonconvergence after 20 time slots is not uncommon in the other three algorithms. The advantage of the Dyna-Q algorithm over Q-learning is that it can be learned from models built with historical experience to speed up convergence. The advantage of the improved Dyna-Q algorithm over the Dyna-Q algorithm is that it can be learned from the SINR model fitted by Equation (7) to speed up convergence. The advantage of WoLF-PHC is that it can dynamically adjust the learning rate parameters according to the learning effect to speed up the convergence. Due to the smart jamming attack, Dyna-Q has almost no advantage over WoLF-PHC. The reason is that the changed attack strategy invalidates many experiences in the Dyna model. Learning the wrong experience in the Dyna model leads to the learning effect not rising, but falling. Figure 6: Performance of the SDN-based learning algorithms using power allocation and path re-planning strategy against smart attacks with \(P_{T}=P_{J}=0.4\), \(C_{m}=0.8\), \(C_{p}=0.2\), and \(C_{h}=0.4\) under the condition that jammers change their jamming strategy, including changing jamming channels and changing the positions of jammers, nine times at equal intervals in 300 time slots. (**a**) shows the utility of the UAV network and (**b**) shows the SINR of the UAV network. The last experiment was used to test the effect of changes in key parameters on UAV network performance with \(P_{T}=P_{J}=0.4\), \(C_{p}\in[0,1.5]\), \(C_{h}\in[0.1,0.8]\), and \(C_{m}\in[0.1,0.8]\). The key parameters we chose are three costs, which are frequency hopping cost, \(C_{h}\); UAV path replanning cost, \(C_{m}\); and unit transmission cost, \(C_{p}\). The simulation results are shown in Figure 8 and Tables 5-7. The same effect of the three cost parameters on network performance is that the utility decreases almost linearly as the cost increases. For instance, the utility of the UAV network decreases 68.0% with Dyna-Q if the frequency hopping cost \(C_{h}\) changes from 0.1 to 0.8. The utility of the UAV network decreases 34.1% w improved Dyna-Q if the UAV path re-planning cost \(C_{m}\) changes from 0.1 to 0.8. The utility of UAV network decreases 60.4% with WoLF-PHC if the unit transmission cost \(C_{p}\) changes from 0.1 to 1.5.