## 5 Experiment and Evaluation
### Simulation Results
The red curve represents the reward of PM-DQfD with DAR. Other curves depict the performance of PM-DQfD, DQfD, PM-DQN, DQN, and CRQ-routing with responsibility rating, respectively. It is found that the expected return of DQfD-based schemes is generally higher than that of CRQ-routing and the two DQN-based algorithms. This is because the network is well trained after pre-training for the DQfD-based schemes, leading to more sophisticated strategies. In addition, the rewards of the two prioritized memories-based schemes, i.e., PM-DQfD and PM-DQN, are larger than those of natural DQfD and DQN, respectively. The reason for this is that the prioritized memory-based schemes periodically erase transitions with low TD-error, retaining high-quality data that has greater value for policy learning. Furthermore, better performance is achieved when the DAR is applied to PM-DQfD. Since the transmit power can be tuned to a reasonable value with fewer hops when applying DAR, lower energy consumption and end-to-end delay are obtained in each learning episode. Figure 7: (**a**,**b**) Data efficiency of the natural DQfD and PM-DQfD versus time step. The effectiveness properties of the energy efficiency and routing latency are investigated in Figure 9. In Figure 9a, we depict the system power consumption ratio changing with the number of episodes. For all algorithms, system PCR experiences a gradual decline as the learning process goes on. From these curves, it can be concluded that the DQfD-based routing protocols achieve lower energy expenditure and faster learning speed than PM-DQN, DQN and CRQ-routing. Moreover, the single-agent framework, DQfD-based schemes have almost the same convergence speed as the multi-agent CBMQ scheme, and achieve better performance than it with regard to energy efficiency. This is because in DQfD-based schemes, the pre-training stage decreases the interactions with the real environment, which decreases trial-and-error cost as well as energy consumption. Furthermore, PM-DQfD holds an advantage over natural DQfD. This is because more valuable transitions are preserved with the memory structure optimization and redundancy reduction, which helps the agent achieve an energy-efficient routing protocol. In addition, since power adaptation is completed with fewer hops when applying DAR, the average PCR on the whole path decreases. Hence the system energy efficiency of DAR-based PM-DQfD is further improved compared with the responsibility rating-based schemes. In Figure 9b, we evaluate the trend of expected end-to-end delay changing with the learning episodes. It is similar to Figure 9a, since the weight of routing latency is equal to that of PCR when constructing the instantaneous reward in Equation (15). PM-DQfD with DAR achieves the shortest routing latency in all algorithms. In addition, it converges as fast as the multi-agent scheme CBMQ, which has a faster convergence speed than CRQ-routing and DQN-based protocols. No more expatiation here, since the reason for this is the same as in Figure 9a. Figure 8: Expected reward versus number of episodes. Figure 9: (**a,b**) Effectiveness properties versus number of episodes. In Figure 10, we investigate the average hops along the path changing with the number of learning rounds. It is found that DQfD-based schemes achieve lower average hops compared with other algorithms. Furthermore, the learning speed of DQfD-based schemes is the same as CBMQ, which converges much faster than CRQ-routing and two DQN-based algorithms. The reason for this is that pre-training with demonstration data accelerates the parameter-tuning of neural network in DQfD-based schemes. It is worth mentioning that the average number of hops of CBMQ are almost the same as that of PM-DQN. This is because the advantage of multi-agent schemes is the reduction in the power consumption and transmission delay of each hop, rather than achieving fewer hops along the path. Similarly, the prioritized memory-based protocols achieve fewer learning steps than the natural algorithms.