# From Cyber-Physical Convergence to Digital Twins: A Review on Edge Computing Use Case Designs
## 7 Edge Computing Digital Twins--Special Use Cases
### Federated Deep Reinforcement Learning-Based Digital Twin
#### 7.2.1 The Federated DRL Computational Offloading  
Contrary to the traditional FML, techniques such as the federated DRL (FDRL) can be designed into edge intelligence frameworks for enabling edge devices to remember what they have learned together with other edge devices [122]. Therefore, edge intelligence can be applied in cases where there are multiple edge devices that need to make decisions in different environmental contexts. This means that in each environmental context, each edge device can build its own learning framework based on its context with assistance from other edge devices. Here, the basic assumption is that edge devices with limited computation capabilities and limited storage resources, but having various compute-intensive, time-critical, and privacy-sensitive applications, offload their data to the edge server [123]. Using the FDRL computational offloading scheme, the privacy-sensitive data are protected from malicious attacks by the federated intelligence shown in Figure 6 below. In Figure 6a above, a single round of FDRL in a CRN is illustrated, where edge devices on the device layer compute their local updates \(\,\mathrm{w}\,\) using their local data without sharing them then upload their local updates to the edge server via the gNB in the form of a  
Figure 6: (**a**) A FDRL architecture for secure computational offloading in CRNs, and (**b**) a FDRL-inspired digital twin-empowered edge platform. computed gradient. Then, the edge server in the edge layer combines all the gradient updates from the different edge devices and performs aggregate computation of global update \(\Delta w\). Upon computing the global update, the edge server then broadcasts it back to the edge devices via the gNB. The processing delay of this FDRL computational offloading scheme is usually determined using the number of iterations required for the algorithm to converge [121]. The two-layer abstraction of the FDRL offloading architecture in Figure 6a above is discussed as follows:  
* The Device Layer: The device layer operates according to vertical FML, where each edge device begins by partitioning a DNN model according to the current environmental factors, such as channel conditions and available bandwidth [123]. Then, an assumed set of edge devices share their local datasets with one another to cooperatively train their associated gNB by uploading gradients \(\Delta w\). The transmission link for information exchange between devices is governed by device-to-device (D2D) protocols. In return, the edge server computes global updates and broadcasts an aggregated gradient as \(\ w\) through the gNB to globally train its associated devices [124]. * The Edge Layer: The edge layer consists of a gNB that is managed with a DNN model that is partitioned according to the workload of the edge server and executes according to device resource and intermediate data to the server. Here, features that describe the state of the edge servers and the requirements of the edge devices are fed to learn the DNN [125]. Through its generalization capabilities, the DNN trains the agent towards yielding general scheduling policies, which are not just tuned states that are encountered during the training process but are adaptive states that can be applied even to unknown states in the processes of predicting/prescribing. The FDRL technique, as well as federated communication, can solve issues related to data privacy and latency in edge computing. However, a trade-off between the local computation delay and the communication delay is very crucial in minimizing the overall FDRL processing latency. In this context, the processing latency can be defined as a measure of the responsiveness of the computing devices (edge devices and the edge server), while responsiveness is used to measure the quality of the network by the number of round trips per minute. Therefore, processing latency results from the slowness of the hardware devices and the number of hops along the network.