## 2 System Model  
A multi-hop CRN comprising \(M\) PUs and \(N\) SUs is considered. Specific spectrum bands are assigned to PUs according to the fixed spectrum allocation regulation. SUs occupy no licensed channels and transmit data opportunistically when finding that frequency bands are not held by the PUs. Every SU node \(n_{i}\) has a set of available channels that consist of Data Transmission Channel (DTC) and Common Control Channel (CCC). The DTC is used for data transmission and SU \(n_{i}\)'s DTC is represented as \(C_{i}=\{c_{1},c_{2},\ldots,c_{m}\}\); whereas the CCC is used by SUs to exchange the negotiation. At any time, a directed communication link can be constructed between SU \(n_{i}\) and \(n_{j}\) if at least one common DTC \(c\in C_{i}\cap C_{j}\) exists. The network model is shown in Figure 1. In the networking scenarios, the multi-hop CRN coexists with two centralized Primary User (PU) networks. As Figure 1 shows, the source SU generates data packets and sends them to the destination node in multi-hop manner through intermediate SUs. Each PU communicates with the PU base station using a licensed frequency, and intermediate SUs in each hop transmit data via the PU channel when the spectrum band is idle.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Abbreviation** & **Full Name** \\ \hline SUs & Secondary Users \\ PUs & Primary Users \\ CRN & Cognitive Radio Networks \\ QoS & Quality of Service \\ CR & Cognitive Radio \\ SG & Stochastic Game \\ DTC & Data Transmission Channel \\ CCC & Common Control Channel \\ PDF & Probability Density Function \\ TL & Transmission Latency \\ PCR & Power Consumption Ratio \\ AWGN & Additive White Gaussian Noise \\ SINR & Signal-to-Interference plus Noise Ratio \\ RL & Reinforcement Learning \\ MSNE & Mixed-Strategy Nash Equilibrium \\ PLR & Packet Loss Ratio \\ \hline \hline \end{tabular}
\end{table}
Table 1: List of Acronyms.  
Figure 1: Multi-hop cognitive networking scenarios.  
We assume that every node in the multi-hop CRN maintains a queuing buffer for the storage of SU packets. For data flow \(f\) generated from source node \(n_{s}\), packet arrivals are considered as a stationary Bernoulli process with mean \(\lambda_{s}^{f}\) that is independent and identical at all time slots [18]. In addition, every SU node has respective queues for each traffic flow, and the packet arrival process of every data stream is independent from each other.  
The PUs' occupation model is considered as an ON/OFF process [19]. The probability density function (PDF) of the OFF periods (when PUs do not occupy the channel) is shown as follows:  
\[f(t)=\left\{\begin{array}{cc}\theta_{d}e^{-\theta_{d}t}&t\geq 0\\ 0&t<0\end{array}\right., \tag{1}\]  
where \(\theta_{d}\) is departure rate of the PU, and \(f(t)\) represents the idle probability of PU channel at time step \(t\). Accordingly, the probability that idle period of PU channel is longer than duration \(\tau\) is denoted as:  
\[P(t\geq\tau)=\int_{\tau}^{\infty}f(t)dt=e^{-\theta_{d}\tau}, \tag{2}\]  
where \(\tau\) is the duration that PU channel is idle. Then, the probability of the collision between PU and SU in the duration \(\tau\) (i.e., the probability of PU reoccupying the spectrum band in the duration \(\tau\)) is given by:  
\[P_{collision}=1-P(t\geq\tau)=1-e^{-\theta_{d}\tau}. \tag{3}\]  
We use the analysis described in [20] to calculate the PU departure rate \(\theta_{d}(\mu,\sigma)\) with expected mean \(\mu\) and deviation \(\sigma\). The spectrum statistic that is parameterized in [21] changes slowly so that it is assumed to be almost static in this work. Every SU can only locate its own position through some positioning equipment.