## 4 PM-DQfD-Based Energy-Efficient Joint Design Scheme
### Background of Natural DQfD  
We consider a scenario in which there exists an expert source node that has already learned stable strategies. On this basis, new data flows are derived from another source node, and a small number of intermediate nodes change their location in the network. The strategies of the expert source node cannot be fully adapted to the new network scenario for two reasons. Firstly, the topology has slight changes compared with the previous network. Secondly, if the new source node lies in a remote area of the network, the expert's strategies in the state of the new source node may be imperfect, since few data packets are relayed by it. However, it will take a long time and extensive energy if the self-learning scheme is adopted in this case [26; 27]. Thus, DQfD is considered, which accelerates the learning process with the aid of the expert's demonstration data [28]. Unlike the general learning scheme, DQfD allows the new source node to learn from the mature experience of the expert source node. Therefore, it performs well in tasks at the beginning of the learning process. The key link for DQfD is the pre-training stage. The agent utilizes the expert demonstrations to acquire relatively effective strategies. After pre-training with the demonstration data, interactions with the real environment are reduced, which decreases the cost of trial-and-error. Therefore, both the energy expenditure and the learning period decrease significantly. In the initial stage, the expert source node learns the routing strategies using a self-learning framework called DQN. For the expert source node, the environment state is defined as \(s_{t}=\left\{n_{i},\psi_{i}^{t}\right\}\) and the action is given by \(a_{t}=\left\{n_{k},c_{i},p_{\psi_{i}}\right\}\), where \(\psi_{i}^{t}\) is the responsibility rating of SU \(n_{i}\) defined in Equation (6), \(p_{\psi_{i}}\) is the transmit power of SU \(n_{i}\) corresponding to its responsibility rating, and the meaning of other parameters is the same as that of Equations (13) and (14). In addition, the reward function of the expert source node is the same as Equation (15). As shown in the learning framework depicted in Figure 2, the agent in the expert source node chooses an action \(a_{t}\) comprising route selection, channel access and power allocation, and then it observes the new state \(s_{t+1}\) and obtains the instantaneous reward \(r_{t}\) from the environment. The agent integrates them into the demonstration data \(e_{t}=\left(s_{t},a_{t},r_{t},s_{t+1}\right)\) and stores this in the expert's replay memory. If the reward \(r_{t}\) is larger than a pre-defined threshold, the transition \(e_{t}\) will be stored in the demo buffer of the expert node as demonstration data. At a later point, these demonstration data will be transmitted to the apprentice source node through CCC. Back in the training step of the expert source node, in every timeslot, a mini-batch of transitions are sampled from the replay memory to calculate the loss function and update the network parameters based on gradient descent. At the next time step, the agent selects the action according to the Q-function approximated by the updated neural network. Finally, the expert source node gradually learns the optical strategy of cross-layer routing. In the pre-training stage, the loss function consists of the Bellman update loss, the supervised loss, and the L2 regularization loss [29]. The Bellman update loss \(L_{B}(Q)\) guarantees that the Bellman equation is satisfied, and Q-value update can adopt online TD-learning in the later period. The supervised loss plays an important role in the effectiveness of pre-training. This is because the demonstration data only covers a limited state space and cannot traverse all possible actions. Many state-action pairs did not appear in the pre-training stage, so there is no data to ensure the corresponding Q-function converges to reasonable values [29].