## 4 Low-Energy Federated Deep Reinforcement Learning for Caching
### Local DRL Model Design  
When a request for content is received by a local base station (BS) from user equipment (UE), the cloud-edge system conducts a search for the content and subsequently delivers it to the UE. The content may be stored in the local BS, neighboring BSs, or the cloud. The objective is to cache the content proactively prior to user requests in order to ensure high cache hit efficiency. To achieve this goal, the proactive caching policy is updated in each epoch of global federation, based on the local deep reinforcement learning (DRL) training performance of the BS. Therefore, it is essential to design the training process of the local DRL. Generally, the design of DRL involves three steps: (1) designing the system state, action, and reward; (2) defining the caching policy based on the system state, action, and reward; and (3) continuously optimizing the caching policy through online iteration. DRL distinguishes itself from reinforcement learning (RL) by utilizing deep network stacking, which enhances the generalization capability of the trained neural network as the sample data grow larger. The first step in DRL is to define the state \(s_{i}\) in the environment, followed by selecting an action \(a\) based on \(s_{i}\) and the action probability. This is then followed by a new state \(s_{i+1}\) and a reward \(R\). In the context of the cloud-edge collaborative caching environment, the cache state is defined for content \(f\) at each decision \(i\). We model the content replacement process in the cache of BS as an Markov decision process (MDP) [35]. The states, system actions, and rewards of the cache and request are shown below:  
\[s_{i}=(s^{r}_{i,u,f},s^{c}_{i,u}) \tag{12}\]  
Let \(s^{r}_{i,u}:=s^{r}_{i,u,f^{c}}f\in\mathcal{U},f\in\mathcal{F}\) and \(s^{c}_{i,u}:=s^{c}_{i,u,f},n\in\mathcal{N},f\in F\) denote the content request state and content cache state, respectively. \(s^{c}_{i,u,f}=1\) and \(s^{c}_{i,u,f}=1\) indicate that the UE \(u\) requests content \(f\) and content \(f\) has been cached in \(BS_{n}\), respectively. Each iteration \(i\) has \(F\) elements, and the set of epochs is the cache state over a period of time. This also indicates the cache state required for one iteration during online training. Since the UE can obtain the requested content in the local BS, neighboring BSs or the cloud, there are three types of actions for the agent in a decision epoch: \(a^{local}_{i}\) replacing the content to the local edge node, \(a^{BS-BS}_{i}\) neighbouring edge nodes collaborating on the content request, and \(a^{cloud}_{i}\) processing the content request in the cloud. Then the action policy under state \(s_{i}\) is defined as  
\[\phi(s_{i})=\{a^{local}_{i},a^{BS-BS}_{i},a^{cloud}_{i}\} \tag{13}\]where \(a_{i}^{local}=\{a_{i,1}^{local},a_{i,2}^{local},\ldots,a_{i,f}^{local}\}\) denotes the local processing action, \(a_{i,f}^{local}\in\{0,1\}\), \(f\in\ \mathcal{F}\), and \(a_{i,f}^{local}=1\) indicates that content \(f\) needs to be replaced by the requested content in iteration \(i\); otherwise, \(a_{i,f}^{local}=0\). If the requested content is not cached in the local BS, it needs to be processed in the neighboring BS. \(a_{i}^{BS-BS}=\{a_{i,1}^{BS-BS},a_{i,2}^{BS-BS},\ldots,a_{i,N}^{BS-BS}\}\) denotes the processing action of the neighboring BSs, \(a_{i,n}^{BS-BS}\in\{0,1\}\), and \(a_{i,n}^{BS-BS}=1\) indicates that the requested content is provided in \(BS_{n}\). If the UE is unable to get its requested content in either BS, the local BS has to decide whether to forward the request to the cloud for processing. Let \(a_{i}^{cloud}\in\{0,1\}\), and \(a_{i}^{cloud}=1\) means the request is handed over to the cloud for processing. After an action is performed under state \(s_{i}\), a reward is obtained. From formula (7), it can be seen that the reward of the cache should maximize the cache hit efficiency.