## 5 Numerical Results and Discussion
### Performance Analysis of User versus Jammers  
In order to investigate the performance of DRL-based anti-jamming algorithm and verify the theoretical analysis in Section 4, we let the DRL-based users combat different kind of jammers. In Figure 5, the horizontal axis represents cumulative iterations. Each iteration indicates 2 s (200 times) of the users' communication. The longitudinal axis is the normalized throughput in one iteration. In Figure 5, four kinds of jammer jam DRL-based user. It can be seen that in the face of four types of jamming, DRL-based user can almost perfectly avoid the jamming signal after training. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline Current frequency & 2 Mhz & 6 Mhz & 10 Mhz & 14 Mhz & 18 Mhz \\ \hline Next hop & 18 Mhz & 14 Mhz & 2 Mhz & 10 Mhz & 6 Mhz \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Frequency hopping table. Figure 4: Illustration of jamming modes. In order to illustrate the effect of randomness on the DRL-based anti-jamming algorithm, Figure 6 shows the throughput against RL-based jammer with different \(\varepsilon\). As we can see in Figure 6, if the RL used a greedy strategy (\(\varepsilon=0\)) completely, after 50 iterations the jammer failed. As \(\varepsilon\) increased, the performance of DRL anti-jamming dropped dramatically. The deep neural network did not converge due to the randomness, which caused the failure of DRL-based user to learn the pattern of RL-based jamming. From the right of Figure 4b we can see that DRL-based method gets jammed with high probability in face of RL-based jamming. In Figure 7, the performance of RL-based jammer (\(\varepsilon=0.2\)) combating different modes of users are shown. As we can see, the RL-based jammer can effectively jam the frequency-fixed and frequency-hopping users because the communication process of them are ARP according to the definition in [35]. As expected, since the DRL-based user's strategy is changing according to the communication effect, the process of user is not ARP, the RL-base jammer can not get its ideal effect because the RL algorithm is not convergent. Note the performance DRL-based mode gets are still unsatisfactory because of \(\varepsilon=0.2\). Figure 5: Performance of deep reinforcement learning (DRL)-based user versus different kind of jammers. Figure 6: Performance of DRL-based user versus RL-based jammer with different \(\varepsilon\). The Figure 8 shows the normalized throughput of different modes of users with different \(\varepsilon\) after 50 iterations. As we can see, with the \(\varepsilon\) increasing, the probability to randomly choose actions of RL algorithm increased, which had as a result the jammer overly exploring and the actions with good reward could not be fully exploited. However, as \(\varepsilon\) increased, the performance of DRL-based user decreased rapidly due to the randomness as explained above. In Table 2, the performance that different kinds of jammers combat with different kinds of users are summarized. The numbers in the iteration column are the number of iterations that the RL or DRL algorithm needed to converge. The RL-based jamming was used with the setting of \(\varepsilon=0.2\) in simulation. In the normalized throughput column, the average normalized throughput that is calculated after 50 iterations is shown. As we can see in Table 2, the throughput of DRL-based anti-jamming method was restricted by the RL-based jamming with \(\varepsilon=0.2\). Due to the good convergent speed and jamming effect of the RL-based jammer, the effective anti-jamming method needs to be further studied. Figure 8: Performance of RL-based jammer with different \(\varepsilon\) versus DRL-based user. Figure 7: Performance of reinforcement learning (RL)-based jammer with \(\varepsilon=0.2\) versus different modes of user.