## 4 Simulation Results and Analysis
### Analysis of Simulation  
Figure 5 shows the influence of different positions of jammers on the optimal path return. The X-axis and Y-axis represent the position coordinates of the jammer, and the colored area in the figure represents the mean of the maximum return obtained by the path with the minimum transmission hop number in each experiment when the jammer is at this position. Assume that the source node is at (1,1) and the destination node is at (25,25). As can be seen in Figure 5, when the jamming power and transmitting power remain unchanged, the closer the jammer is to the line where the source node and the destination node are, the smaller the return of the optimal path. When the jammer approaches the source node or the destination node, the last hop of the source node and the previous hop of the destination node are both within the serious jamming range of the jammer. At this time, the signal-to-noise ratio of all available nodes is less than the minimum transmission signal-to-noise ratio threshold, so the transmission path may not be found. Figure 4: Schematic diagram of communication nodes. Figure 6 shows the comparison of hops required by different algorithms in each experiment. The \(X\)-axis is the number of experiments, and the \(Y\)-axis is the number of hops in each experiment. Simulations demonstrate the superiority of conventional Q-learning and the MAB proposed in 2019. The Dyna-Q algorithm proposed in this paper can find the path of the minimum hop number at about the 15 th experiment. The classical Q-learning algorithm can find the minimum hop path in about 21 experiments, and the MAB algorithm can find the minimum hop path in 32 experiments. As can be seen in the figure, the proposed Dyna-Q algorithm can converge to the optimal result faster than the classical Q-learning and MAB algorithms. Here, the communication nodes in the Dyna-Q algorithm are planned, while the communication nodes in the classical Q-learning and MAB algorithms are not planned, especially in Q-learning. When n in the n-step Dyna-Q algorithm is 0, it is classical Q-learning. Because the communication nodes are not planned, each experiment will only increase one learning opportunity to learn the strategy. That is, information is learned when it is transmitted to the destination node. The planned communication node, in the first experiment, just like Q-learning, will only have one learning opportunity, but \(\textbf{model}(s,a)\) is initially established, so in the second experiment, multiple backtracking updates can be made to calculate the Q table without waiting for the end of this experiment. Therefore, a planned agent can find the minimum hop path faster than others. The number of experiments required to find the minimum number of hops represents the time spent. The fewer the experiments, the less time, and the faster the speed. Figure 5: Influence of different jammer positions on optimal path return. Figure 6: Comparison of hops required by different algorithms in each experiment. Figure 7 shows the returns obtained by the above three algorithms in each experiment. It can be seen in the figure that the n-step Dyna-Q algorithm can obtain the highest returns faster than the other two algorithms, resulting in faster convergence. As shown in Figure 8, in the n-step Dyna-Q algorithm, the more steps planned, the faster the convergence and the better the effect. Compared with 15-step, 5-step, and 0-step planning, 50-step planning has the best effect. The 0-step Dyna-Q algorithm here is direct reinforcement learning; 50-step planning can converge to the optimal value in the third experiment, the 15-step plan converges after about 30 times, the 5-step plan converges after about 50 times, and the 0-step plan converges after even more experiments. As shown in Figure 9, the algorithms of different step planning are compared. It can be seen in the figure that 50-step planning still achieves the fastest convergence, followed by 15-step planning, 5-step planning, and 0-step planning.