## 4 Low-Energy Federated Deep Reinforcement Learning for Caching
### Local DRL Model Design
Therefore, the cache reward \(R_{i,n}(s_{i},\phi(s_{i}))\) is represented in iteration \(i\) as follows:  
\[R_{i,n}(s_{i},\phi(s_{i}))=H_{i,n} \tag{14}\]  
when making caching decisions, it is necessary to ensure that each BS does not cache more files than its capacity. The capacity limits are defined as follows:  
\[\sum_{f\in\mathcal{F}}s_{i,n}^{c}C_{f}\leq C_{n} \tag{15}\]  
MDP is usually defined as a transition \(\delta_{i}=\{s_{i},\phi(s_{i}),R_{i,n}(s_{i},\phi(s_{i})),s_{i+1}\}\) under DRL. By learning in a series of transitions \(\delta_{i}\), DRL can give the best strategy to maximize the return in the current state. The action-value function Q function is used to evaluate the expectation starting at state \(s_{i}\). Then, the Q function is defined as  
\[Q(s_{i},\phi(s_{i}))=\mathbb{E}_{\phi}\left[\sum_{i=1}^{\infty} \gamma^{i-1}R(s_{i},\phi(s_{i}))|s_{i};\phi\right] \tag{16}\]  
where \(\gamma\in(0,1]\) is the factor discounting future rewards. The optimal value \(Q^{*}(s_{i},\phi(s_{i}))=max_{\phi}Q_{\phi}(s_{i},\phi(s_{i}))\), and \(Q^{*}(s_{i},\phi(s_{i}))\) can be written as follows:  
\[Q^{*}(s_{i},\phi(s_{i}))=\mathbb{E}_{s_{i+1},\phi}\left[R_{i}+ \gamma\max_{\phi}Q^{*}(s_{i+1},\phi)|s_{i},\phi\right] \tag{17}\]  
A local BS training model was specifically developed utilizing the double Q-learning (DDQN) algorithm as described in [36], with a focus on states, actions, and rewards as depicted in Figure 2. Two deep neural networks were employed to learn an equation \(Q(s_{i},\phi(s_{i});w)\) that approximates \(Q^{*}(s_{i},\phi(s_{i}))\). The objective of \(Q^{*}(s_{i},\phi(s_{i});w)\) is \(y_{i}=R_{i}+\gamma\max_{\phi}Q(s_{i+1},\phi;w)\). Consequently, the parameters \(w\) can be updated to optimize \(Q^{*}(s_{i},\phi(s_{i}))\). Figure 2: DDQN training process for cache. Figure 2 illustrates the training process of DDQN, where every individual local BS maintains a finite-size experience replay pool, which is refreshed with the latest transitions. The Q network (MainNet) of the deep reinforcement learning (DRL) framework is leveraged to determine system actions, while the \(\hat{Q}\) network (TargetNet) is utilized to evaluate the selected actions. The \(\hat{w}\) in the \(\hat{Q}\) network is periodically updated with \(w_{i}\) in the Q network. During the training process, the agent randomly selects a minibatch \(\delta_{i}\) from the experience replay pool and trains the Q network by minimizing the loss function for each iteration,  
\[loss(w_{i})=\mathbb{E}_{\delta_{i}}\left[\left(R(s_{i},\phi)+\gamma\hat{Q} \left(s_{i},\underset{\phi_{i+1}}{\arg\max}Q(s_{i+1},\phi_{i+1};w_{i});\hat{w}_ {i}\right)-Q(s_{i},\phi_{i};w_{i})\right)^{2}\right] \tag{18}\]  
Thus, we can obtain the gradient update formula for \(w_{i}\),  
\[ \begin{split}\
abla_{w_{i}}loss(w_{i})=\mathbb{E}_{\delta_{i}} \left[\left(R(s_{i},\phi_{i})+\gamma\hat{Q}\left(s_{i},\underset{\phi_{i+1}}{ \arg\max}Q(s_{i+1},\phi_{i+1};w_{i});w_{i+1}\right)-\right.\right.\\ \left.\left.Q(s_{i},\phi_{i};w_{i})\right)\cdot\
abla_{w_{i}}Q(s_ {i},\phi_{i};w_{i})\right]\end{split}  \tag{19}\]  
The online iterative process used by DDQN for caching is represented by Algorithm 1. First, each BS initializes MainNet and TargetNet based on its historical information and obtains the initial reply memory \(\Omega\) (shown as lines 1-5). Next, DDQN is executed to train the caching process and update all parameters provided that there is no requested content \(f\) in \(BS_{n}\) and the \(BS_{n}\) storage is full (shown as lines 8-17). ```
1:for each \(BS_{n}\)do
2: Initialize replay memory \(\Omega\). 3: Initialize action-value function \(Q\) with random weights \(w_{0}\). 4: Pretraining the MainNet and TargetNet by \(w_{0}\) and \(\hat{w}_{0}\) from local historic information. 5: Save \(\{s_{0},\phi_{0},R_{0},s_{1}\}\) in \(\Omega\). 6:endfor
7:for each \(BS_{n}\), each iteration \(i\in L_{n}\)do
8:\(BS_{n}\) receive a request content \(f\). 9:if\(s_{i,n,f}^{c}=1\)then
10: Provide content directly \(f\) to UE \(u\).