## 3 An Anti-Jamming Algorithm Based on Dyna-Q  
In this work, an anti-jamming path selection algorithm framework based on reinforcement learning (Dyna-Q) was designed. As shown in Figure 3, the continuous interaction between communication nodes and the environment generates practical experience, partly directly improving the value functions and strategies in reinforcement learning. The other part is used to improve the model. The former is called direct reinforcement learning, and the latter is called model learning. In the beginning, the \(Q(s,a)\) table is initialized, where the initial state is the source node, and the action is selected. At first, there is an instant reward, that is, 0; when arriving at \(C_{N}\) communication nodes, there is a corresponding reward of 1 in the \(Q\) table. According to the principle of \(Q\)-learning and Equation (7), the function returns to update the current \(Q\) value. When the \(Q\) value in the \(Q\) table becomes less than the threshold value, the \(Q\) table converges. Communication nodes can gain some \"experience\" in actual communication. These historical experiences can be represented by the quad \((s_{t},a_{t},s_{t+1},r)\), where \(s_{t}\) is the communication node at time \(t\), \(a_{t}\) is the communication node adopted at time \(t\), \(s_{t+1}\) is the communication node selected at time \(t+1\), and \(r\) is the reward obtained. All quads form a historical experience set, denoted as \(\textbf{model}(s,a)\). The Q function is expressed as \(Q(s,a)\). According to the literature [25], \(Q\)-learning is a type of reinforcement learning. The agent constantly interacts with the environment and finds the optimal communication node according to the feedback information of the environment. Equation (7) is derived from the Bellman equation. \(Q\) is updated as follows:  
\[Q(s_{t},a_{t})=(1-\alpha)Q(s_{t},a_{t})+\alpha\bigg{(}\text{r}+\gamma\underset {a_{t+1}\in A}{\text{max}}Q(s_{t+1},a_{t+1})\bigg{)} \tag{7}\]  
where \(\alpha\) is the learning rate, and \(\gamma\) is the discount factor. The \(Q\) table is created by constant interaction. The agent chooses the best action according to the \(Q\) table. The \(Q\) table is the same as the \(Q\) function. It is an array in the simulation. Using Q-learning produces \(Q\) tables. When a new communication experience is obtained, a new quad \((s_{t},a_{t},s_{t+1},r)\) will be obtained. Based on the current state \(s_{t}\), the agent takes action \(a_{t}\), then gets a timely reward \(r\), and moves to the next state \(s_{t+1}\). The quad consists of \(s_{t}\), \(a_{t}\), \(r\), and \(s_{t+1}\). It is determined whether the new quad \((s_{t},a_{t},s_{t+1},r)\) exists in \(\textbf{model}(s,a)\), and if not, the new quad \((s_{t},a_{t},s_{t+1},r)\) is stored in \(\textbf{model}(s,a)\) to update \(\textbf{model}(s,a)\). \(\textbf{model}(s,a)\) will not be updated when there is no new communication experience; that is, no new quad is generated. Figure 3: Structure diagram of Dyna-Q. \(\textbf{model}(s,a)\) was used to learn the simulation process, and a group of quads was randomly selected in \(\textbf{model}(s,a)\). According to the selected quad \((s_{t},a_{t},s_{t+1},r)\), communication users find the corresponding \(\underset{a_{t+1}\in A}{\max}Q(s_{t+1},a_{t+1})\) in the \(Q\) table. Then, \(s_{t},a_{t},r\), and \(\underset{a_{t+1}\in A}{\max}Q(s_{t+1},a_{t+1})\) are substituted into Equation (7), and the \(Q\) table is updated. This process is repeated \(n\) times. After updating the \(Q\) table, the agent adopts the \(\varepsilon\)-greedy algorithm to select action \(a_{t}\). The specific algorithm can be expressed as follows:  
\[a_{t}\ =\ \left\{ \begin{array}{ll}\underset{a\in A}{\max}Q(s_{t},a)& \text{Probability of }1-\varepsilon\\ \forall a\in\mathcal{A}&\text{Probability of }\varepsilon\end{array} \right. \tag{8}\]  
Finally, \(a_{t}\) is output.