## 5 Experiment and Evaluation
### Simulation Results  
Figure 6 shows the quantity variation trends of self-generated data and expert demonstrations in the self-learning stage. We find that the amount of demonstration data remains constant before 1000 time steps. After that, the agent starts self-training and the size of the demonstration buffer decreases until it reaches 1500. On the other hand, the quantity of self-generated data increases monotonically before 2000 time steps. Then the total amount of the self-generated data fluctuates between \(b\%\) and \(a\%\) of the replay buffer size as memory optimization is performed. The decrease of demonstration data and the dynamic change of the self-generated data save the storage space, which can be used for storing more cache data or sensor data with higher precision. The data efficiency of the natural DQfD and PM-DQfD is investigated in Figure 7. Figure 7 presents the average TD-error of natural DQfD and PM-DQfD as they change with the learning \begin{table}
\begin{tabular}{l c} \hline \hline
**Parameter** & **Default Value** \\ \hline Link Gain & \(g=\varepsilon G(r/r_{0})^{m}\)[34] \\ Available Spectrum & 56 MHz-65 MHz \\ Bandwidth, \(B\) & 1 MHz \\ AWGN power, \(\xi\) & \(10^{-7}\) mW \\ PU-to-SU interference, \(\phi_{ijk}^{PU}\) & \(\phi_{ijk}^{PU}\) \(\sim\)\(\left[10^{-7},10^{-6}\right)\) mW \\ Packet Size, \(S_{packet}\) & \(2\times 10^{5}\)_bit_ \\ Mean OFF period of PU, \(\lambda\) & 2 ms \\ SINR threshold, \(\sigma_{\text{fi}}\) & \(-80\)_dBm_ \\ Higher Latency Threshold, \(t_{\text{H}1}\) & 16 _ms_ \\ Lower Latency Threshold, \(t_{\text{H}2}\) & 12 _ms_ \\ Weighting of the power consumption ratio, \(x_{1}\) & 100 \\ Weighting of the single-hop delay, \(x_{2}\) & 1 \\ Demonstration ratio, \(v\) & 0.1 \\ Discount factor, \(\beta\) & 0.99 \\ Learning rate, \(\alpha\) & 0.001 \\ Weighting of the supervised loss, \(\gamma_{1}\) & 1 \\ Weighting of L2 regularization loss, \(\gamma_{2}\) & \(10^{-5}\) \\ Pre-training step, \(\kappa\) & 200 \\ Replay period, \(p\) & 300 \\ Size of Mini-Batch, \(p\) & 64 \\ Upper Limit of Erasing, \(a\) & 99 \\ Lower Limit of Erasing, \(b\) & 70 \\ Erasing Parameter, \(\varepsilon\) & 0.2 \\ Final Size of Demo Buffer, \(S\) & 1500 \\ Erasing parameter of Demo Buffer, \(q\) & 3 \\ Replay buffer size, \(R\) & 2000 \\ Demonstration buffer size, \(R\) & 2000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. Figure 6: Quantity variation trends of self-generated data and demonstration data. timeslot. It can be seen that the average TD-error of both algorithms increases with timeslot, and the TD-error of PM-DQfD is significantly higher than that of DQfD. This is because data with low TD-error is swept off the demonstration buffer and the replay buffer when using PM-DQfD, which enhances the average TD-error. These results comply with common sense. In addition, the average TD-error of PM-DQfD surges as the self-generated data is erased. After that, the TD-error declines until it stabilizes, and the stable value of TD-error is larger than the value before the surge. The reason for this is that a large amount of high-quality data is retained after executing memory erasing, and it brings accumulation advantage for the subsequent phases. Figure 7b demonstrates the overall loss of the natural DQfD and PM-DQfD changing with time steps. It is observed that the overall loss of two algorithms shows a trend of fluctuating downward, and the loss of PM-DQfD is lower than that of DQfD. Moreover, the overall loss of PM-DQfD shows a sharp decline every time data erasing is performed. This is mainly because PM-DQfD periodically erases transitions with low TD-error, and the average TD-error of the remaining data gets larger. Data with larger TD-error make greater contributions to parameter training for neural network than other data. Thus, PM-DQfD causes lower loss and achieves better performance. Figure 8 demonstrates that with increasing learning episodes, the expected reward increases gradually for all algorithms.