## 3 Correlations between Federated Learning and Deep Reinforcement Learning
The resource states (e.g., remaining capacities at the current timeslot) and FL model task sizes can be observed as state features for configuring the allocation amount and edge node offloading destination [65]. The reward can be formulated by the transmission latencies between participant-edge-central and convergence points in the achieved round index. In another aspect of a synchronous FL-IoT condition, high training delays can happen because of a straggler effect. To tackle these issues, [66] introduced a promising architecture that is capable of mapping the client operation with real-time states to a virtual entity and capturing the features of devices for interacting with the agent. Furthermore, an asynchronous FL  
Figure 5: Key interaction and components between DRL agent and FL environment. was proposed with actor-critic DRL-assisted device selection. The DRL components and observable features in this FL environment are listed as follows:  
* _State_: in the particular timeslot \(t\), the environment samples a set of states including transmission power, the device's available resource for computation, model statuses, and previously selected device statuses at timeslot \(t-1\);
* _Action_: in timeslot \(t\), the agent indicates which device to select for participating in the training process;
* _Reward_: to evaluate the state-action (device selection efficiency) in timeslot \(t\), the objectives of minimizing the energy utilization and maximizing the FL accuracy are formulated as the reward evaluation metric. In the process of implementing a DRL-based algorithm for device selection, the parameters of actor, critic, target actor, and target critic networks are required to execute the model iteratively until optimal DNN parameters are defined. In each iteration, the sampling experience replays of an FL environment setup are stored and input into different networks until the final policy satisfies the expectation of the end goal. After selecting the device, k-class(es) of devices are clustered based on the power levels and data contribution sizes. By identifying specific classes of device performance stats, the convergence speed and parallel processing threshold are significantly improved. The straggler effects of massive IoT scenarios are handled in this systematic framework. Local training and global aggregation are executed afterward. Local updates from massive clients can be optimized in the uploading process within a cellular system [67]. In [68], a resource-optimized and trust-aware DRL scheduling for optimal client selection over FL-based IoT environment was studied. Monitoring modules were used for detection and information abstraction between IoT devices and edge nodes. Moreover, edge nodes can play an important role as a management entity of IoT nodes. A stochastic optimization formulation was used as a primary statement to correspond with the DRL-based selection algorithm. To execute a functioning agent, the states were collected from IoT participant's features, and the actions were adjusted to handle the different labels of current-iteration resource/trust at that particular timeslot. The edge nodes were also responsible for initializing the deep model with privacy-free data before declaring the model structures/values. The DRL-based approach can later sample a subset of clients to participate in the procedure. The efficiency of expected performance has relied on the upper-bound execution delay, training latency, resource utilization, and trust levels. The solution on [68] was deployed in a healthcare scenario on top of IoT infrastructure. A DRL-based FL was further converged for efficient and intelligent orchestration in EC-enabled IIoT applications. Due to the drawback of a centralized resource-allocation scheme, [69] introduced an optimization approach to ideally observe and modify three state spaces, namely (1) task offloading ratio, (2) bandwidth allocation ratio, and (3) transmission power in FL-enabled IIoT environments. Action spaces covered a discrete vector of state modification.