## 4 Low-Energy Federated Deep Reinforcement Learning for Caching
### Federated Mechanism
Commencing from the \(t_{1}\)-th iteration, the cloud selects the BSs (\(\mathcal{N}_{1}\)) that have received more requests than the request quantity threshold in the previous \(t-1\) iterations. Next, the BSs with cache hit efficiency lower than the efficiency threshold are chosen for online training (lines 11-13), while other BSs (\(\mathcal{N}_{2}\)) adopt the cached strategy obtained after the previous \(t_{1}\)-th iteration of training (line 15). This strategy facilitates continued training for BSs with low caching efficiency and fully reflects the local content request characteristics for BSs with higher efficiency, thereby sustaining a high cache performance. Simultaneously, as the number of BSs participating in online training in each round diminishes, the overall energy consumption of the system reduces. \(w_{t,n}\) represents the model parameters in epoch \(t\). For the local dataset \(D_{n,t}\) of each BS, its model loss function is given by  
\[F_{n}(w_{t,n})=\frac{1}{D_{t,n}}\sum_{d\in D_{t,n}}f_{d,n}(w_{t,n,d}) \tag{20}\]  
\[ \begin{array}{l}\text{{subject to}}:\\ \mathcal{N}_{1}\leftarrow\{n|\Re_{t-1,n}>\Re_{threshold},H_{t-1,n}<H_{\text{ threshold}}\}\end{array} \]  
where \(\mathcal{N}_{1}\) represents the set of BSs that require local online training. Therefore, the local problem is  
\[w_{t,n,d}=\arg\min F_{n}(w_{t,n}|w_{t-1,n}) \tag{21}\]  
that is, to minimize the local loss function relative to the previous epoch (lines 10-18). (2) A novel strategy for the dynamic adjustment of local iteration times in BS models that participate in local online training. Our approach aims to obtain the neighboring BSs experience without compromising the unique local characteristics of each BS model. To achieve this, we propose dynamically adjusting the local iteration times based on the difference between the model parameters obtained from the previous global training (\(t-1\)) and the current global model parameters at time \(t\). Specifically, we define a weight factor \(\beta\) as the basis for this dynamic adjustment strategy (line 7). Figure 3: Proposed federated mechanism. \[\beta_{t,n}=\frac{\sqrt{\sum\limits_{i}^{I}(w_{t,n}-w_{t})}}{\frac{1}{N}\sum \limits_{n\in\mathcal{N}}\sqrt{\sum\limits_{i}^{I}(w_{t,n}-w_{t})}} \tag{22}\]  
The dimensionality of the model parameters is denoted by \(I\). Thus, for \(BS_{n}\), the number of local training iterations is determined by \(L_{t,n}=[\beta_{t,n}L_{0}]\), where \([\beta_{t,n}L_{0}]\geq 1\). Models with large differences in parameter values have lower global performance and may not meet the dynamic requirements of requests, therefore requiring more local iterations (\(L_{n}\)) to ensure their dynamic performance. Additionally, reducing the number of local iterations can lead to lower energy consumption of the system, as shown in Equation (8). When a local model of a participant is close enough to the global model, the number of local training rounds for this participant can be appropriately reduced to reduce the training energy consumption. Local parameter updates can be expressed as follows:  
\[w_{t+1,n}=w_{t,n}-\eta\
abla F_{n}(w_{t,n}) \tag{23}\]  
(3) The conventional approach to model aggregation involves weighting the training data size of each model to create a global model. However, in cloud-edge caching environments, the number of content requests and the cache hit rate are more critical technical indicators. This is because, for any \(BS_{n}\), as the number of content requests increases, the caching strategy must adapt more dynamically to maintain request efficiency. Additionally, a higher cache hit rate implies a higher cache efficiency for the current \(BS_{n}\).