## 5 PM-DQN Based Joint Routing and Resource Allocation Algorithm
### Prioritized Memories Deep Q-Network  
DQN is adopted to solve the joint design problem because of its huge state space. One of the significant features for DQN is experience replay. The system maintains a replay memory and samples transitions from the memory to update weights of neural network. Using a replay memory leads to design choices at two levels: which experience to store, and which experience to replay. Prioritized Experience Replay addresses only the latter: replaying important transitions more frequently, and therefore learning more efficiently [25]. However, maintaining a replay memory in multi-hop CRN increases storage requirement and communication overhead. Consideration about which transitions to store and when to erase them has become an urgent problem. To reduce replay memory and communication overhead in CRN, the Prioritized Memories Deep Q-Network is proposed. It is able to reduce the redundancy by biasing the contents of the memory to where the errors remain high. The magnitude of a transition's TD-error \(\delta\) is used as a reasonable proxy because it can indicate how far the value is from its next-step bootstrap estimate [26] and how unexpected the transition is. Specifically, the transitions whose absolute TD-errors are below a certain value will be erased, i.e., transitions with high absolute TD-errors can be retained. The implementation is demonstrated in Figure 4.  
First, the format of experience at time step \(t\) is set as \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1},\delta_{t})\), where \(\delta_{t}\) represents the TD-error of the transition. It is assumed that \(\delta_{t}=\infty\) before the transition is replayed, and the true TD-error is assigned to \(\delta_{t}\) after the experience is sampled. If time steps are less than memory size, it means that the learning process has not started. As shown in Figure 4a, the transition is stored following the previous one, and the number of experience is increasing. In the second stage, time steps exceed memory size and the number of replayed transitions is less than \(a\%\) of the memory size. The replayed experience with the lowest absolute TD-error is replaced by the new experience as shown in Figure 4b. In this period, the number of total experiences remains the same. Then we define the Erasing Critical Value (ECV) as:  
\[\text{ECV}_{t}=rand(0,\lambda)*\frac{1}{K_{i}}\sum_{n=0}^{K_{i}}|\delta_{n}| \tag{18}\]  
The ECV is obtained by multiplying a random value in \(\Lambda=(0,\lambda),0<\lambda<1\) with the average absolute TD-error of replayed experiences. If the number of replayed transitions is greater than \(a\%\) of the memory size, the experience whose \(|\delta_{t}|\) is less than the ECV will be erased. The number of experiences decreases in this stage. Fourth, if the number of replayed transitions is below \(b\%(b<a)\) of the memory size, we stop erasing implementation to recover the amount of experiences, and the number of transitions increases gradually. However, the performance of PM-DQN will decrease if the value of \(b\) is too low. If the difference between \(a\) and \(b\) is too small, it is inefficient in saving system memory. So the value of \(a,b\) in this paper is set to 90 and 60 respectively, which proves to be appropriate in the simulation part. The new experience is stored following the previous one as shown in Figure 4c,d, and the agent will perform the third and the fourth step until learning process is end. Figure 5 shows the variation trends of transition quantities in the whole learning process of PM-DQN. We can see that the total number of the transitions fluctuates between \(b\%\) and \(a\%\) of the memory size. The dynamic change of the transition quantities saves the system memory while guaranteeing the performance. The details of PM-DQN are obtained as described in Algorithm 1.