# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
## 5 Simulation Results
Since the state space and the action space are all quantified into discretized levels, all the four algorithms end in 10 s.  
We conducted three simulation experiments. The first experiment verified the performance of the four algorithms under the fixed jamming attack strategy. The second experiment verified the performance of the four algorithms under the smart jamming strategy, which means that jammers can use SDR to change their jamming strategy. The third experiment verified the influence of several key parameters on a utility metric. In the first experiment, we compared the performance of the above four reinforcement learning algorithms under a certain jamming strategy randomly selected by jammers in 100 time slots. In this performance analysis, two most important metrics are selected, which are the utility and the SINR of the UAV network. The utility metric can be calculated by Equation (2), and the SINR metric is just the first item on the right side of Equation (2). As shown in Figure 5, the utility and SINR of the UAV network increases over time and gradually converge with different amplitudes. The improved Dyna-Q based strategy has the highest utility, followed by the Dyna-Q. WoLF-PHC, and Q-learning-based strategies. After 100 time slots, the utility of the improved Dyna-Q-based strategy is 1.1252, which is 6.9%, 13.8%, and 18.6% higher than that of the Dyna-Q, WoLF-PHC, and Q-learning-based strategies, respectively. The performance of the WoLF-PHC algorithm has large fluctuations. After 40 time slots, the range of utility of the WoLF-PHC-based strategy varies from 0.88 to 1.11, which is wider than that of the other three strategies. Besides, the SINR values calculated by the four algorithms have a similar trend compared to their utilities. The utility values are calculated by subtracting each cost from the SINR values. However, although three costs (path planning, data transmission, and frequency hopping) vary randomly, the sum of the three will offset a large part of the change, resulting in a stable trend of the sum of the three, which causes the trend of SINR to be very similar to that of utility. By the end of the simulation, the SINR with improved Dyna-Q is 3.1141, which is 2.8% higher than that of Dyna-Q, 5.4% higher than that of WoLF-PHC, and 6.6% higher than that of the Q-learning strategy. The second experiment was used to test the performance of the four algorithms when the jammers used a SDR-based smart jamming strategy. Specifically, the jammers can continuously adjust their attack strategies using the Q-learning algorithm, including reallocating the jamming power and replanning the location of the jammers, but can only adjust the position of one jammer to its surrounding eight grids at one time slot, with \(\varepsilon=0.3\), \(\mu=0.7\), \(P_{T}=P_{J}=0.4\), \(C_{m}=0.8\), \(C_{p}=0.2\), and \(C_{h}=0.4\). The UAV network selects and performs actions from the action space according to the four algorithms, respectively, and finally calculates the utility and SINR. Similarly, at most  
Figure 5: Performance of the SDN-based learning algorithms using power allocation and path re-planning strategy against smart attacks with \(P_{T}=P_{J}=0.4\), \(C_{m}=0.8\), \(C_{p}=0.2\), and \(C_{h}=0.4\) where \(P_{T/J}\) means total power constraints of the UAV/jammer, \(C_{m}\) means the cost of UAV path replanning, \(C_{p}\) means the cost of data transmitting, and \(C_{h}\) means the cost of frequency hopping. (**a**) shows the utility of the UAV network and (**b**) shows the SINR of the UAV network. to move to one of the surrounding eight grids at one time slot. The jammers change their jamming strategy nine times at equal intervals in 300 time slots. The experimental results are shown in Figure 5. It can be seen from Figure 6 that the four algorithms bring about different degrees of performance hopping for each change of the attack strategy. The improved Dyna-Q-based strategy has the highest utility, followed by the Dyna-Q, WoLF-PHC, and Q-learning-based strategies.