## 4 PM-DQfD-Based Energy-Efficient Joint Design Scheme
### Prioritized Memories Deep Q-Learning from Demonstrations  
Prioritized experience replay can be adopted to sample high-value data more frequently for performance improvement. However, the agent still needs to maintain both of the large replay buffer \(D^{\textit{replay}}\) and large demo buffer \(D^{\textit{demo}}\), which increases the storage occupancy. Therefore, consideration about how to release storage space and utilize the data efficiently has become a crucial issue. To address the problem discussed above, the Prioritized Memories Deep Q-learning from Demonstrations (PM-DQfD) is presented, which optimizes the memory structure of both demo and replay buffer. It erases the ineffective transitions and biases the memory contents to where the data value remains high. In DRL, transition's TD-error \(\delta\) is defined as follows:  
\[\delta=r+\beta\underset{ar\in A}{\max}Q(s,ar;\theta_{i}^{-})-Q(s,a;\theta_{i}) \tag{19}\]  
It measures the potential for performance improvement when using the data for training [30]. So TD-error is adopted as an effective index to evaluate the value of transitions. To be specific, the transitions whose TD-errors are lower than the threshold will be swept off the replay buffer \(D^{\textit{replay}}\), and expert demonstration with the lowest TD-error will be erased from the demo buffer \(D^{\textit{demo}}\). In Figure 3, we illustrate the implementation of memory optimization for the replay buffer \(D^{\textit{replay}}\). The transition at timeslot \(t\) is formulated as \(e_{t}=(s_{t},\,a_{t}\,,r_{t}\,,s^{\prime}_{t},\,\delta_{t})\), where \(\delta_{t}\) denotes the TD-error of experience. For the convenience of memory optimization, \(\delta_{t}\) is set as \(\infty\) before \(e_{t}\) is replayed. The real TD-error will be rewritten after the transition is sampled. Firstly, the learning process does not start before the transitions' amount is more than half the size of the replay buffer. As shown in Figure 3a, transitions are stored in sequence and number of transitions grows. In the second phase, the agent begins learning, and the quantity of transitions is below \(a\%\) (\(a>50\)) of the buffer. We can see from Figure 3b that every timeslot a mini-batch of transitions is replayed, while the total number of transitions simultaneously increases. Then we define the Memory Erasing Threshold (MET) at timeslot \(t\) as:  
\[MET_{t}=rand(0,\varepsilon)\star\frac{1}{R_{t}}{\sum_{n=0}^{R_{t}}}\delta_{n} \tag{20}\]  
where \(R_{t}\) is the number of replayed transitions at timeslot \(t\), and \(\varepsilon\) is a certain value between 0 and 1. If the quantity of replayed transitions exceeds \(a\%\) of the buffer size, the transitions whose TD-errors are lower than MET will be swept off replay buffer \(D^{\textit{replay}}\) until the quantity is less than \(b\%\) (\(b<a\)) of the buffer size. As shown in Figure 3c, the number of transitions decreases sharply in this period, and only transitions with sufficiently high TD-error are preserved. The second and the third step are executed alternately until the learning procedure converges. On the other hand, the expert demonstrations may be subsequently outdated due to the dynamic nature of the radio environment. Thus, the reliability of the expert demonstrations is reduced for the agent, and it is unnecessary to maintain as much demonstration data as the initial phrase. Based on this, we erase \(q\) transitions with the lowest TD-error from the demo buffer \(D^{demo}\) every time when the learning episode is finished. When the amount of data decreases to a particular value \(S\), we stop erasing demonstrations, and the number of demonstrations remains unchanged after that. Memory structure optimization regularly erases worthless transitions, which saves system memory and efficiently utilizes the data. PM-DQfD is shown in detail as Algorithm 1.