## 4 PM-DQfD-Based Energy-Efficient Joint Design Scheme
### PM-DQfD-Based Energy-Efficient Cross-Layer Routing Scheme  
In this section, the PM-DQfD is applied to the cross-layer routing design for improvement of power efficiency and reduction of transmission delay. To combine the single-agent DRL with wireless network environments, an effective mechanism is designed. In the protocol, only the source node needs to deploy the cognitive engine and the replay memory, which radically reduces the computation load and communication overhead. The implementation is demonstrated as follows:  
The complete learning stage consists of three parallel steps: data transmission, action transmitting and transition back propagation. As shown in Figure 4, we take the learning process of the SU node \(n_{i}\) at stage \(t\) as an example. Firstly, after the data transmission of link \((j,i)\), SU \(n_{j}\) generates the transition \(e_{t}\) of the link \((j,i)\) and back propagates \(e_{t}\) to the source node \(n_{0}\) via CCC. Meanwhile, the SU node \(n_{i}\) chooses the intermediate node \(n_{k}\) as its next relay node and starts transmitting data to \(n_{k}\) through DTC \(c_{i}\) according to the action selection at the previous stage \(a_{t-1}=\left\{n_{k},c_{i},p_{DdR_{i}}\right\}\). During the data transmission, the agent in source node \(n_{0}\) updates network parameters to \(\theta_{t}\) by sampling a mini-batch of transitions. Then it selects action \(a_{t}\) of the next relay node \(n_{k}\) and sends it to \(n_{k}\) through the established route. The three parallel steps are executed hop by hop until the destination node successfully receives the data packet.  
Furthermore, memory structure optimization is adopted in the joint design scheme. When the transition number is increased to a certain extent, it will release the storage space in order to reduce memory occupation. The memory saved can be used to store more data caches. For example, it is possible to store sensor data with higher precision for wireless sensor networks. Therefore, the PM-DQfD-based joint design algorithm is compatible with wireless environments in multi-hop CRN. Although the PM-DQfD-based protocol has many advantages over traditional schemes, a tradeoff is made between system performance and signaling overhead. The signaling overhead of the PM-DQfD-based cross-layer routing protocol consists of two parts. Firstly, the expert demonstrations are transmitted from the expert source node to the apprenticeship source node through the established route via CCC. Secondly, action transmission and transition back propagation are performed every hop in the single-agent-based routing mechanism. However, since the size of both transition and demonstration buffer is limited, the signaling overhead during the demonstration data transmission is in an acceptable range. In addition, the size of one action is sufficiently small compared to the data packet so that the signaling overhead caused by the action transmission and transition back propagation is relatively low. The interactions with the real world decrease despite these overheads, which is worthwhile for the significant decline of the power consumption and transmission latency.  
Figure 4: Implementation of PM-DQfD-based cross-layer routing scheme.