## 3 Correlations between Federated Learning and Deep Reinforcement Learning  
In this section, the correlations between the DRL agent and FL environment are given by emphasizing the usage of DRL-assisted zero-touch management to optimize the learning procedures in IoT networks. Figure 5 is given as an overall concept of modeling the interaction interfaces between the DRL agent and FL environment, including the key features from the main entities such as parameter server, edge aggregator, and local participants. Different scenarios define different condition-aware states, actions, and reward formulations. In DRL-based eFL, there are complementary studies that outline various scenarios with defined state-action relativity and long-term orchestration goals from agent. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Domains** & **Summary of Contributions** & **Ref.** & **Year** \\ \hline \multirow{6}{*}{_Applications of DRL in communications and networking_} & \multicolumn{3}{c}{The overview, in-depth} \\  & components, analysis, taxonomies, & \\  & and comparisons of DRL & & \\  & techniques for optimizing & & \\  & communications and networking & & \\ \hline \multirow{6}{*}{_DRL for IoT applications_} & DRL algorithms for handling the problems of communication, computation, caching, control, domain-oriented applications, and privacy in IoT environment & [57] & 2021 \\ \cline{1-1} \cline{2-4}  & The overview of wireless networks & & \\  & and DRL, taxonomy of IoT & & \\  & problem models, DRL-based approaches in IoT, and challenging issues of DRL in IoT networking & [58] & 2021 \\ \cline{1-1} \cline{2-4}  & DRL-based optimization & & \\  & approaches for renewable energy, & & \\  & network slicing, spectrum scarcity, & & \\  & resource management & improving transmission rate, and & \\  & big data in 5G & & \\  & heterogeneous networks & & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Contributions of Existing Selected Surveys on Deep Reinforcement Learning. In the simulation perspective, federated averaging uses an iterative process and selects a random subset of participants for each iteration training step. Then, the initialization function is executed to sample the server's states. The server distributes the model to selected participants, which later computes the model training locally. The aggregation toward the new state is the output of adding between the initial state and the average of total states from clients. To build federated averaging, the models, clients, and optimizers are required to be known. In the testing phase, the data must be centralized and the validation function must be declared. To evaluate the federated optimization, the developers/researchers must specify the settings in terms of cross-silo/cross-device, differential privacy, or communication types [60, 61, 62, 63]. The framework can be evaluated after finalizing the identical hyperparameter setup, the number of selected clients, and communication conditions. In this implementation flow, we can identify the correlation parts with the DRL agent as follows:  
* The initialization function of FL can be converged with DRL as state features within the setup environment (e.g., capacities of edge aggregators);
* The actions can be merged with the selected values of participant subset within each FL round communication;
* Reward is the evaluation of federated optimization. By fully integrating the functions together, simulation is highly applicable and opens many key ideas for real-world application scenarios. DRL has the potential in managing dynamic resources and making smart decisions in IoT environments. Once the application services have computation-intensive tasks to execute, EC is optimally suggested for dealing with adaptive virtual resource allocation [59, 64]. Similarly, in FL model tasks, EC can serve as an enabler for resource efficiencies.