## 3 Formulation for Joint Design Problem
### Comprehensive Utility Function
#### 3.1.1 Transmission Delay  
The transmission delay consists of queuing waiting time and data transmission time. Firstly, the queuing waiting time is computed based on the packet arrival and service rates of every SU in a multi-hop CRN [18]. The calculation method of queuing waiting time \(\delta_{i}\) was proposed in [24], where the packet arrival and service rates of every SU were used to compute the queuing waiting time. In our work, we further combine this method with the concept of strategy \(\zeta_{k}(s_{k},\ a_{k})\) in RL to obtain the latency for queuing in SU's buffer. A packet from flow \(f\) is placed in the SU \(n_{i}\)'s queuing buffer at time step \(t\) if:  
1. One of the node \(n_{i}\)'s neighboring nodes selects node \(n_{i}\) as its next hop and transmits data via an available channel \(c\);
2. Channel \(c\) is idle during time step \(t\), and
3. There is at least one packet in the queue of the preceding SU node to communicate with node \(n_{i}\)  
Therefore, the arrival rate at SU \(n_{i}\) is the joint probability of all events above, which can be represented as the product of these events' probabilities due to the independence among themselves:  
\[\lambda_{i}^{t}=\sum_{f\in F}\sum_{k\in L_{i}}\sum_{c\in C_{k}}\zeta_{k}(s_{k},\ a_{k})\cdot\frac{\lambda_{f}}{\mu_{f}}\cdot\alpha_{c}\cdot\left(1-P_{ki}^{ out}\right), \tag{5}\]  
where \(\mathbf{F}\) is the set of all data streams; \(\mathbf{L_{i}}\) is the set of \(SU_{i}\)'s previous nodes that select node \(n_{i}\) as their next hop; \(\mathbf{C_{k}}\) is the set of available channels of node \(n_{k}\in\mathbf{L_{i}}\); \(\zeta_{k}(s_{k},\ a_{k})\) is the strategy of node \(n_{k}\in\mathbf{L_{i}}\), i.e., the probability of node \(n_{k}\) choosing action \(a_{k}\), which corresponds to the next intermediate node \(n_{i}\) and the operating channel \(c\in\mathbf{C_{k}}\); \(\lambda_{f}\) is the arrival rate of data flow \(f\); and \(\mu_{f}\) is its service rate. The probability that the queue has at least one packet can be represented as \(\lambda_{f}/\mu_{f}\) according to the theory of discrete time Markov chain. \(\alpha_{c}\) is the probability that channel \(c\) is idle and \(P_{ki}^{out}\) represents the outage probability of the link between node \(n_{k}\) and \(n_{i}\). Like the arrival rate, the service rate is equal to the probability of transmitting a packet successfully at SU \(n_{i}\), which occurs if:  
1. \(SU_{i}\) selects node \(n_{j}\) as its next hop and transmits data via an available channel \(c^{\prime}\), and
2. Channel \(c^{\prime}\) is idle during time step \(t\). Accordingly, the service rate at SU \(n_{i}\) can be calculated as the product of these two events' probability:  
\[\mu_{i}^{t}=\sum_{f\in F}\sum_{j\in N_{i}}\sum_{c^{\prime}\in C_{i}}\zeta_{i} (s_{i},\ a_{i})\cdot\alpha_{c}\cdot\left(1-P_{ij}^{out}\right), \tag{6}\]  
where \(\mathbf{N_{i}}\) is the set of \(SU_{i}\)'s neighboring nodes; \(\mathbf{C_{i}}\) is the set of node \(n_{i}\)'s available channels; \(\zeta_{i}(s_{i},\ a_{i})\) is the strategy of node \(n_{i}\), i.e., the probability of node \(n_{i}\) choosing action \(a_{i}\), which corresponds to the next intermediate node \(n_{j}\) and the data transmission channel \(c^{\prime}\in\mathbf{C_{i}}\); and \(P_{ij}^{out}\) represents the outage probability of link between node \(n_{i}\) and \(n_{j}\). As discussed above, the arrival and service processes of every SU are Bernoulli processes with rates \(\lambda_{i}^{t}\) and \(\mu_{i}^{t}\), respectively. This queuing system is modeled as a Geo/Geo/1 queue [25]. Consequently, the queuing waiting time of SU \(n_{i}\) is calculated as:  
\[\delta_{i}=\frac{\lambda_{i}^{t}}{\mu_{i}^{t}\big{(}\mu_{i}^{t}-\lambda_{i}^{t }\big{)}}. \tag{7}\]  
Large packet size, interference of PUs and bandwidth constraint will lead to limited channel capability. So, the data transmission time has to be considered.