## 4 Problem Formulation for Cross-Layer Design
### Responsibility Rating  
In this paper, an efficient cross-layer scheme is taken account, which jointly addresses routing, spectrum access and power assignment problem in the multi-hop CRN. Conventional RL methods, i.e., tabular based algorithms, behave poorly when dealing with problem with large state space because of curse of dimensionality. Moreover, it is unrealistic to maintain a large lookup table in every relay node. So it is inclined to adopt DQN to solve the joint design problem with large state space. For energy efficiency and PU protection, we take power adaption into consideration. In this case, the actions of the agent comprise relay selection, channel usage and power allocation, which may result in a huge action space. Though DQN has the capacity of handling problem with large state space, it is not adept in managing problem with huge action space. This is because DQN adopts the Q-learning update framework as the following:  
\[ \begin{split} Q_{t+1}(s,a)&=(1-\alpha)Q_{t}(s,a)+ \alpha\bigg{[}r_{t}+\beta\underset{b\in A}{\max}Q_{t}(s^{\prime},b)\bigg{]}\\ &=Q_{t}(s,a)+\alpha\bigg{[}r_{t}+\beta\underset{b\in A}{\max}Q_{ t}(s^{\prime},b)-Q_{t}(s,a)\bigg{]}\end{split}  \tag{11}\]  
where \(\alpha\in[0,1)\) is the learning rate, and \(\beta\) is the discount factor determining the agent's horizon. The Deep Q-Network is trained by minimizing a sequence of loss function:  
\[L_{i}(\theta_{i})=E_{s,a\sim p(\cdot)}\Bigg{[}\bigg{(}r+\beta\underset{a^{ \prime}\in A}{\max}Q(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i}) \bigg{)}^{2}\Bigg{]} \tag{12}\]  
We can see that both DQN and Q-learning aim to update the Q-value in the direction of \(r_{t}+\beta\underset{b\in A}{\max}Q_{t}(s^{\prime},b)\). And the update of two algorithms needs to search the maximum of \(Q_{t}(s^{\prime},b)\), so that we can say DQN adopts the Q-learning framework indeed. That is, the agent has to choose the action with the largest Q function among all actions available. The maximum calculation will have intensive complexity if the action space is particularly large. To handle this problem, a novel concept named responsibility rating is introduced. The transmission time of the SU packet in node \(i\) is given by:  
\[\tau_{i}=\ R_{packet}\Bigg{/}W\log_{2}\bigg{(}1+\frac{h_{ijc}p_{i}}{\sigma+ \Phi_{ijc}^{PU}}\bigg{)} \tag{13}\]  
where \(R_{packet}\) is the size of SU packet transmitted from source to destination, \(h_{ijc}\) is the channel gain between the transmitter node \(i\) and the receiver node \(j\), \(\Phi_{ijc}^{PU}\) denotes the PU-to-SU interference at the receiver node \(i\), and \(\sigma\) is the AWGN power. If much time is wasted in one hop transmission, it is reasonable that SU node has to increase the transmission power in the next hop in order to reduce the end-to-end delay. In addition, if the delay of one hop is particularly low, then the transmission power can be reduced in the next hop to cut down on power consumption. According to this, the responsibility rating of \(SU_{j}\) is defined as:  
\[\psi_{j}=\left\{ \begin{array}{ll}\psi_{i}+1&\tau_{i}>\tau^{*}\\ \psi_{i}-1&\tau_{i}\leq\tau^{*}\end{array} \right. \tag{14}\]  
where \(\psi_{i}\) is a nonnegative integer and represents the responsibility rating of \(SU_{i}\). In addition, if \(\psi_{i}=\max\{\psi_{i}\}\), then \(\psi_{j}=\psi_{i}\), and if \(\psi_{i}=0\), then \(\psi_{j}=0\). \(\tau^{*}\) is the average transmission time of SUs and can be obtained through history observations. Each responsibility rating \(\psi_{i}\) corresponds to a transmit power \(p_{i}\) (\(p_{\min}\leq p_{i}\leq p_{\max}\)), and the correspondence is described as the following:  
\[p_{i}(\psi_{i})=\left(1-\frac{\psi_{i}}{|\Psi|}\right)p_{\min}+\frac{\psi_{i}} {|\Psi|}p_{\max} \tag{15}\]  
where \(|\Psi|\) denotes the size of \(\{\psi_{i}\}\). If SU costs a lot of time in one hop, the agent will have the responsibility to transmit the packet with larger power in the next link.