## 4 PM-DQfD-Based Energy-Efficient Joint Design Scheme
### Background of Natural DQfD
To avoid this, the supervised loss is introduced as follows:  
\[L_{S}(Q)=\max_{a\in A}[Q(s,a)+l(s,a_{E},a)]-Q(s,a_{E}) \tag{16}\]  
where \(a_{E}\) is the action chosen in the demonstration data, and \(l(s,a_{E},a)\) is a margin function given by:  
\[l(s,a_{E},a)=\left\{ \begin{array}{ll}0&,a=a_{E}\\ C&,else\end{array} \right. \tag{17}\]  
where \(C\) is a large positive constant. It is found that if \(a_{E}\) achieves the maximum Q-function among all actions, the supervised loss \(L_{S}(Q)=0\); otherwise, \(L_{S}(Q)=Q(s,a)+C-Q(s,a_{E})>0\). This ensures the parameters of the neural network update towards the distribution of the expert's Q-function, which reaches its maximum value when performing \(a_{E}\). In addition, to avoid over-fitting the L2 regularization loss, \(L_{L2}(Q)\) is taken into consideration. Consequently, the overall loss function is represented as follows:  
\[L(Q)=L_{B}(Q)+\gamma_{1}L_{S}(Q)+\gamma_{2}L_{L2}(Q) \tag{18}\]  
where \(\gamma_{1}\) and \(\gamma_{2}\) are parameters that regulate the weight between \(L_{S}(Q)\) and \(L_{L2}(Q)\). Once pre-training is completed, the neural network has been trained well without trial-and-error cost. Then the agent enters the self-learning stage, in which it will continue to improve the decision-making level through interactions with the real environment. During the self-learning process, the agent selects the action \(a_{t}=\left\{n_{k},c_{i},p_{DAR},\right\}\) according to the strategy learned in the pre-training phrase, which consists of route selection, channel access and power allocation. Then the agent observes the next state of the environment \(s_{t+1}\) and obtains the reward \(r_{t}\). The self-generated data is formulated as \(d^{self}=(s_{t},a_{t},r_{t},s_{t+1})\) and stored in an independent replay buffer \(D^{ replay}\), which is updated every timeslot. Meanwhile, the demonstration data stored in \(D^{demo}\) remains static. After the amount of self-generated data reaches a certain value, the agent starts sampling a mini-batch of data randomly from \(D^{demo}\) and \(D^{ replay}\) in a specific proportion, which is used to calculate the loss function and update the network parameters by gradient descent. Then the agent will be more intelligent when choosing the next action \(a_{t+1}\), and the routing strategy will be more efficient. In addition, it is important  
Figure 2: Implementation of memory optimization for the replay buffer. to note that the Bellman update loss, as well as the supervised loss, is applied for the demonstration data; while for the self-generated data, only the Bellman update loss is applied. Moreover, if the self-learning process is finished without acceptable results, the agent cannot learn the optimal routing strategy, which leads to lower system energy efficiency and longer end-to-end delay. There are two solutions to this situation. Firstly, we can increase the training steps \(\kappa\) in the pre-training phrase, which leads the agent to obtain a better strategy after pre-training. Secondly, the agent can continue training its neural network using expert demonstrations after the self-learning phrase until its performance is within an acceptable range. The performance of the second method will be better if the demonstration data in the demo buffer is updated using the expert source node after pre-training.