## 5 PM-DQN Based Joint Routing and Resource Allocation Algorithm
### PM-DQN-Based Joint Routing and Resource Allocation Algorithm  
In order to solve the cross-layer design problem with large state space and save the system memory, PM-DQN is applied to the routing and resource allocation problem. The joint design scheme requires merely the source node \(n_{0}\) to equip the RL agent and maintain replay memories. It enormously reduces the communication overhead while guaranteeing system performance. The mechanism of PM-DQN based joint routing and resource allocation scheme is shown in Figure 6, and the serial number in Figure 6 denotes the order of stage. Initially, the agent in the source node \(n_{0}\) uses the initial weights of neural network \(\theta_{0}\) to compute Q value and chooses \(n_{1}\) as its next hop. In the first stage, \(n_{0}\) performs data transmission. Before every data transmission, the SU node broadcasts a relay request (RREQ) message using the CCC which includes its node ID. Upon receiving the RREQ, the neighboring nodes which are closer to the destination send back the relay reply (RREP) messages. The neighbor information in RREP will be back propagated with the newest transition to the source node in the subsequent stages, so that the agent can identify unavailable actions during weights updating of neural network. As \(n_{0}\) transmits data using DTC, the agent updates the weights of neural network to \(\theta_{1}\) by sampling transitions from the previous memories and then selects the action \(a_{1}\) with probability [27]:  
\[\pi(s_{t},a_{t})=\frac{e^{Q(s_{t},a)/\eta}}{\sum_{b\in A}e^{Q(s_{t},b)/\eta}} \tag{19}\]  
where \(\eta\) is a positive parameter called the temperature. High temperature causes the action probabilities to be all nearly equal, and low temperature causes huge difference in selection probabilities for actions. After that the action \(a_{1}\) is transmitted to SU node \(n_{1}\) through CCC. After the data transmission is finished, node \(n_{1}\) obtains the reward and generates the newest transition \(t_{1}\). Figure 6: Transmission mechanism of replay memory in joint design scheme. In the second stage, \(n_{1}\) selects \(n_{2}\) as its next hop according to action \(a_{1}\) and performs data transmission using DTC. Then the newest transition \(t_{1}\) is back propagated to the source node through CCC. At the same time, the agent updates the weights to \(\theta_{2}\) by sampling transitions from the same memories as the first stage and selects the action \(a_{2}\) of \(n_{2}\) using weights \(\theta_{2}\). If the selected action is available, i.e., the next relay \(n_{j}\) is in the neighboring set of the current node, the target Q value is set to \(r_{i}+\gamma\text{max}_{\sigma}\hat{Q}(s_{i+1},a^{\prime};\theta^{-})\) in the weights updating stage of neural network; otherwise, the target Q value is set to a certain negative value. After that the action \(a_{2}\) is transmitted to SU node \(n_{2}\). That is, the SU node performs data transmission through DTC and transmits signaling such as transitions and actions using CCC in the meanwhile. Since the size of one transition or action is particularly small, the transition or action can be successfully transmitted before the data transmission is over and will not cause additional transmission delay. Also, node \(n_{2}\) obtains the reward and generates the newest transition \(t_{2}\) when the data transmission is over. The transmission process of data and signaling in the following stage is similar, and the only difference is that the replay memory has been updated step by step. As shown in Figure 6, in the third stage the memories comprise transition \(t_{1}\), and transition \(t_{1}\) with \(t_{2}\) in the fourth, and so on. The details of Centralized Joint Routing and Resource Allocation Algorithm are obtained as described in Algorithm 2.