# A Dyna-Q-Based Solution for UAV Networks Against Smart Jamming Attacks
## 3 System Model and Network Controller Design
### Network Model  
In the hierarchical mesh network chosen for its flexible scalability, the communication process between the core layer and the edge layer is similar. For simplicity, we only model the core layer of the network, and the modeling method of the edge layer is similar. The notations used in our model are summarized in Table 1.  
There are \(N_{U}\) UAVs in the network, and each UAV flies at a certain height \(h_{u}\) to avoid collisions. The flying height can be adjusted when receiving commands from the network controller or its own flight controller, but the height should be maintained after adjustment. UAV nodes transmit messages over \(N\) radio channels. All the UAV nodes follow the same frequency pattern sets denoted by \(C=\left[C_{p}\right]_{1\leq y\leq\omega}\), where \(\omega\) is the number of frequency patterns and the \(\psi\)-th pattern \(C_{\psi}\) consists of \(\kappa\) \begin{table}
\begin{tabular}{c c} \hline \hline
**Notations** & **Description** \\ \hline \(N_{U}\) & Number of UAVs \\ \(N_{I}\) & Number of jamming UAVs \\ \(h_{u}\) & Flying height of the \(u\)-th UAV \\ \(N\) & Number of radio channels \\ \(\omega\) & Number of frequency patterns \\ \(C_{\psi}\) & The \(\psi\)-th frequency pattern \\ \(f^{(k)}\) & The chosen channel at time slot \(k\) \\ \(p_{u}^{(k)}\) & Transmit power of UAV \(u\) at time slot \(k\) \\ \(P_{T/I}\) & Total power constraints of the UAV/jammer \\ \(d_{u,u_{i}}^{(k)}\) & The distance between UAV \(u_{i}\) and \(u_{j}\) at time slot \(k\) \\ \(L_{u}^{u_{i}(k)}\) & UAV \(u_{i}\)’s destination at time slot \(k,s,t\in\left\{-1,0,1\right\}\) \\ \(\xi\) & Number of SINR quantization levels \\ \(\mathbf{h}_{u,u_{i}}^{(k)}\) & Channel power gains between UAV \(u_{i}\) and \(u_{j}\) at time slot \(k\) \\ \(C_{h}\) & Cost of frequency hopping \\ \(C_{p}\) & Cost of data transmitting \\ \(C_{m}\) & Cost of UAV path replanning \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of main notations used in this work. Figure 3: Design of functional modules in the SDN controller and flow of state information. Msg: Message, Src: Source, Dst: Destination. time slots pattern modes, where the \(i\)-th channel is denoted by \(C_{\psi}=\left[c_{\psi}^{(i)}\right]_{1\leq i\leq N}\), and the chosen channel at time slot \(k\) can be denoted by \(f^{(k)}=c_{\psi}^{k\text{ mod }\kappa+1}\). The transmit power of the \(u\)-th UAV at time slot \(k\) is denoted by \(P_{u}^{(k)}\), and the total transmit power \(P_{r}\) is quantized into \(L+1\) levels. The position of the \(u\)-th UAV at time slot \(k\) can be denoted by \(L_{u}^{(k)}\), and \(\left(x_{u}^{(k)},y_{u}^{(k)},z_{u}^{(k)}\right)\) are the converted Cartesian coordinates. The distance between UAV \(u_{i}\) and UAV \(u_{j}\) at time slot \(k\) is \(d_{u_{i}u_{j}}^{(k)}\). The UAV swarm needs to maintain a relatively stable topology during flight. For simplicity, we assume that only one UAV is allowed to relocate at each time slot and that the other UAVs maintain uniform motion. Within each time slot, the position of the UAV can be relocated to one of the surrounding eight spatial grids, which in the clockwise direction are the north, the northeast, the east, the southeast, the south, the southwest, the western, and the northwest, as illustrated in Figure 4. Since the height difference of the UAV network is much smaller than the UAV's flight range, the spatial grid approximates the planar grid. The side length of the space grid is determined by the maneuverability of the UAV. Specifically, the side length of UAV \(u_{i}\) at time slot \(k\) denoted by \(d_{u_{i}}^{(k)}\) is equal to the minimum displacement of the UAV that flies in eight directions, according to the Dubins path. The entire UAV swarm can be regarded as one large virtual UAV, and the front direction of the eight spatial grids coincides with the flight direction of the virtual drone at time slot \(k\).