## 3 Construction of Learning Framework  
### Dynamic Adjustment Rating  
In the CRN scenarios, the selection criterion for the intermediate node stipulates that the relay SU lies within the transmission range and has common available channels. In addition, power allocation should be considered to enhance the energy utilization and prevent interference with PUs. In this case, the route selection, channel access, and power allocation constitutes the agent's action, leading to a huge action space. Although DQN achieves excellent performance in problems with a large state space, its calculation complexity increases rapidly as the size of the action space becomes larger, which is a result of the characteristics of its update framework. The network parameters \(\theta_{i}\) are updated through loss minimization using stochastic gradient descent:  
\[L_{i}(\theta_{i})=E\!\left[\left(r+\beta\!\max_{a\in A}\!Q(s,a;\theta_{i}^{-}) -Q(s,a;\theta_{i})\right)^{2}\right] \tag{5}\]  
where \(r\) is the instantaneous return, \(\beta\) denotes the discounted factor, \(Q(s,a)\) represents the Q-value at state \(s\) when choosing action \(a\), and \(A\) is the action space of the agent. It can be seen from Equation (5) that the maximum computation is performed at every time-slot so that the calculation load will be heavy if the size of \(A\) is particularly large. Therefore, a novel concept called dynamic adjustment rating is introduced in this section to handle this problem. A reasonable tradeoff between system energy expenditure and routing delay is demanded in CRN. It is reasonable that if the single-hop latency in the present time exceeds a higher threshold \(th_{1}\), the SU node in the next hop will enhance its transmit power for lower transmission latency to maintain a reasonable and stable end-to-end delay. By contrast, if the current transmission latency is sufficiently short, i.e., less than a lower threshold \(th_{2}\), the next intermediate node inclines in order to reduce its transmit power for system energy conservation. Otherwise, it is assumed that the current transmit power is being maintained within an appropriate range and should remain unchanged. According to this principle, the concept of responsibility rating was introduced in [20; 23]. Nevertheless, as for the responsibility rating, the update framework is defined as follows:  
\[\psi_{i}^{t}=\left\{ \begin{array}{ll}\psi_{j}^{t-1}+1,&if\;\tau_{j}^{t-1}> th\\ \psi_{j}^{t-1}-1,&if\;\tau_{j}^{t-1}\leq th\end{array} \right. \tag{6}\]  
where \(\psi_{i}^{t}\) is the responsibility rating of SU \(n_{i}\) at time step \(t\), and it is a non-negative integer mapping to a level of transmit power. From Equation (6), we can see that if the single-hop delay is larger than the threshold \(th\), \(\psi_{i}^{t}\) will increase by one unit in the next hop; otherwise, it will decrease by one unit. Since each responsibility rating \(\psi_{i}^{t}\) corresponds to a transmit power \(p_{i}\), the transmit power only transfers one level for every timeslot. This leads that the process of power adaptation becoming quite long if the available power set is large, reducing power adjustment efficiency. Thus, a multi-leveltransition measurement called Dynamic Adjustment Rating (DAR) is presented. The DAR of SU \(n_{i}\) at timeslot \(t\) is given by:  
\[DAR_{i}^{t}=\left\{ \begin{array}{ll}DAR_{j}^{t-1}+\varphi_{i}^{t},&\quad if\ \tau_{j}^{t-1}>th_{1}\\ DAR_{j}^{t-1},&\quad if\ th_{2}\leq\tau_{j}^{t-1}\leq th_{1}\\ DAR_{j}^{t-1}-\mu_{i}^{t},&\quad if\ \tau_{j}^{t-1}<th_{2}\end{array} \right. \tag{7}\]  
Herein, \(DAR_{j}^{t-1}\) is the DAR of previous SU \(n_{j}\) at time step \(t-1\), and \(\tau_{j}^{t-1}\) denotes the single-hop latency of previous SU \(n_{j}\) at the last timeslot; \(DAR_{i}^{t}\) is a non-negative integer mapping to a level of transmit power, and two special cases are supplemented in Equations (8) and (9) when \(DAR_{j}^{t-1}=0\) and \(DAR_{j}^{t-1}=\max\{DAR\}\):  
\[DAR_{i}^{t}=\left\{