## 4 Deep Reinforcement Learning-Based Efficient Federated Learning Policy Optimization
### Model Offloading Decisions  
Edge FL leverages EC capacities to assist local co-training, reduce client-cloud communication rounds, and offer edge aggregation. However, it comes with several challenging drawbacks in terms of multi-service offloading schedules and resource placement. The interaction of client-edge requires optimal aggregator selection in model label matching for both model update offloading and averaged model downloading. A central FL server with DRL-based decisions on offloading rules and aggregation policies is an optimal technique for ensuring adequately allocated edge resources and appropriate server selection in different IoT network congestion states (Figure 9).  
Figure 8: Experience-driven DRL (e.g., actor–critic) in setup environment of FL parameter server.  
In [79], the authors set up a system architecture on intelligent ultra-dense EC with formulated system models on communication (cellular link, D2D link, service caching, and task transmission) and computation offloading. An optimization problem on joint offloading, resource allocation, and service placement was initiated to aim for the main objective of minimizing task execution delays and resource consumption. Modified DRL was used to deal with the problem formulation by labeling two timescale approaches, namely fast and slow timescales. Fast-timescale DRL covered the handling scheme of offloading decisions and resource orchestration. Slow-timescale DRL dealt with a service caching scheme. To securely train these agents, the FL framework is later converged for a personalized model builder. The primary components of the agent and ultra-dense EC environment consist of:  
* _State_: at the particular timeslot \(t\), the setup environment samples state features as follows: (1) available signal frequency (transmission channels), (2) weight matrices of D2D and uplink transmission rates, (3) caching statuses between small-cell base station and ED, (4) task queue, (5) available computational resource, (6) local resources, and (7) required services;
* _Action_: to alter the environment conditions, the agent consists of three main variables for system modification, namely (1) application partitioning, (2) communication resource allocations, and (3) policy for caching placement;
* _Reward_: to determine the consequence of applied action, the authors evaluate the positive feedback or negative errors by formulating a correlation model to return the joint task execution delay and resource consumption of the system.  
Considering that IoT participants mostly face a resource-constrained status, improved communication/computation efficiencies of on-device DNN training/optimization are greatly beneficial in collaborative FL. The model offloading will be lessened and directed to only essential co-training requirements. Ref. [80] offered masking techniques for efficient FL by allowing participants to execute localized DNN with higher performances and  
Figure 9: Embedded DRL-based optimization engine for offloading rules and aggregation policies in the central FL server.  
achievable levels. The proposed scheme optimized the computation costs of gradient execution and federated masking in the local tier. After the end participant optimizes the masking process, the local model owner offloads the masked models to the central FL server, which greatly strengthens the privacy details.