## 4 Low-Energy Federated Deep Reinforcement Learning for Caching
### Federated Mechanism  
Owing to its distributed and privacy-preserving attributes, federated learning has garnered widespread adoption in domains such as cloud computing, edge computing, and device-to-device (D2D) computing [37]. By leveraging the federated approach to optimize the training models of multiple agents, superior efficiency can be achieved as compared to centralized training, while ensuring complete privacy. Consequently, this manuscript puts forth a novel dynamic energy-conserving federated mechanism to tackle the challenge of the energy efficiency ratio in cache for cloud-edge systems. Due to the regional characteristics of local BSs, their training strategies are more tailored to local demands. In order to ensure that BSs with local advantages do not lose their cache efficiency due to the federated mechanism, it is also necessary to provide sufficient training for BSs with lower cache efficiency. The basic idea of the proposed mechanism is to determine the aggregation and training of BSs based on their data request quantity and cache hit efficiency. Specifically, the cache hit efficiency of each BS is calculated at the end of each epoch \(t\). If a BS has a lower data request quantity, it will not participate in the aggregation process (which will not affect the effectiveness of the global model) and save sent power, while BSs with higher cache hit efficiency will not participate in online training (as local BSs have their own cache advantages at this time). This approach has two issues. On the one hand,some BSs may have a higher hit efficiency due to receiving fewer requests, but the model is not sufficiently trained. Aggregating the model at this point will reduce the performance of the global model. On the other hand, some BSs may have a lower hit efficiency due to receiving too many content requests, but this type of model has a strong generalization ability and can significantly improve the performance of the global model. To this end, we dynamically partition the local BSs and incorporate a mechanism for adjusting the local iteration count dynamically to ensure cache efficiency while reducing energy consumption. Assuming that each BS possesses adequate computational capabilities to execute the training tasks, the dynamic federated mechanism proposed in this study is deployed across both cloud and edge BSs, as demonstrated in Figure 3 and Algorithm 2. The said mechanism comprises three distinct steps. ```
1:Initialized process in BSs:
2:Initialize \(w_{0,n}=w_{0}\). 3:Initialize number of local iterations \(L_{0}\), global epoch \(T\), and step size \(\eta\). 4:Pretrained parameters \(w_{0,n}\) from Algorithm 1 (lines 1-5). 5:for\(t=1,2,3,\ldots,T\)do
6:Aggregate model and select BSs process in cloud:
7: Receive \(w_{t,n}\) from \(\mathcal{N}_{3}\) BSs. 8: Update global model \(w_{t+1}\) according to (26). 9: Computing weight factor \(\beta\) by (22), and set \(L_{t,n}=[\beta_{t,n}L_{0}]\). 10:\(\mathcal{N}_{1}=\{n|\Re_{n}>\Re_{threshold}\wedge H_{n}<H_{threshold}\}, \mathcal{N}_{2}=N\setminus\mathcal{N}_{1}\). 11: Send training requests and global models to \(BS_{n}\) that need to participate in training (\(\mathcal{N}_{1}\)). 12:Local online training process in BSs:
13:for each \(BS_{n}\)do
14:if\(n\in\mathcal{N}_{1}\)then
15: Processing online training by Algorithm 1 (lines 7-19)
16: Set local parameter \(w_{t,n}\) by (23). 17:else
18: Processing strategy of \(t-1\) epoch in \(BS_{n}\) (\(\mathcal{N}_{2}\)). 19:endif
20: Computing \(\Re_{n}\) and \(H_{n}\) of \(BS_{n}\). 21:\(\mathcal{N}_{3}=\{n|\Re_{n}>\Re_{threshold}\}\). 22:for\(n\in\mathcal{N}_{3}\)do
23: Sending models to the cloud. 24:endfor
25:endfor
```  
**Algorithm 2** Dynamic low-power federated DRL for caching (LFDC). (1) BSs are selected for participation in training by defining two thresholds: the request quantity threshold (\(\Re_{treshold}\)) and the cache hit efficiency threshold (\(H_{threshold}\)) (lines 8,12).