## 2 Preliminary Studies on Federated Learning and Deep Reinforcement Learning  
### Federated Learning for Privacy-Preserving Assurance  
Currently, IoT applications consist of privacy-sensitive information belonging to exclusive users, organizations, or business companies, which necessitate obeying GDPR in data gathering or third-party transmission in AI-based services. FL was first introduced in \begin{table}
\begin{tabular}{c c} \hline \hline
**Abbreviation** & **Description** \\ \hline AI & Artificial Intelligence \\ APIs & Application Programming Interfaces \\ APs & Access Points \\ CI & Cloud Intelligence \\ DL & Deep Learning \\ DNN & Deep Neural Networks \\ DRL & Deep Reinforcement Learning \\ E2E & End-to-End \\ EC & Edge Computing \\ ED & End Devices \\ eFL & Efficient Federated Learning \\ EI & Edge Intelligence \\ FA & Federated Analytics \\ FL & Federated Learning \\ GDPR & General Data Protection Regulation \\ GNN & Graph Neural Network \\ IID & Independent-and-Indettically-Distributed \\ IIoT & Industrial Internet of Things \\ IoT & Internet of Things \\ IoV & Internet of Vehicle \\ KPI & Key Performance Indicator \\ MDC & Micro-Data Center \\ ML & Machine Learning \\ NFV & Network Functions Virtualization \\ QoS & Quality of Service \\ SDN & Software-Defined Networking \\ SGD & Stochastic Gradient Descent \\ SON & Self-Organizing Networks \\ VM & Virtual Machine \\ VNFs & Virtual Network Functions \\ VNF-FGs & Virtual Network Function Forwarding Graphs \\ ZSM & Zero-Touch Network and Service Management \\ \hline \hline \end{tabular}
\end{table} 
Table 1: List of Important Acronyms. Figure 2: Structure of the paper. 2016 by Google researchers [29; 30; 31] to emphasize the concept of distributing the models to be trained by local data owners instead of uploading and storing raw data centrally. FL is a privacy-preserving, collaborative, and resource-efficient learning framework to aggregate multi-dimensional decentralized local models in a centralized global parameter server for constructing a reliable learning model. Table 2 presents the contributions of existing selected surveys on FL, which is drawing major attention from researchers, deployment engineers, and organizations. There are six main phases to execute iteratively in FL for the IoT environment, including (1) the architecture of collaborative learning system, (2) global model distribution, (3) local model computation, (4) loss optimization, (5) model transmission, and (6) global model aggregation [31; 32]. Conventional FL procedures are modified and enhanced throughout various research studies for optimizing the performance in communication perspectives within resource-constrained IoT environments. The consideration of client prioritization, accuracy performance, and specific data types, e.g., independent- and identically distributed (IID) and non-IID, must be comprehensively emphasized with detailed specifications in network controller and federated settings. Moreover, local model training/updates are not completely sufficient to execute without possible termination before fully interacting with the edge server. System heterogeneity on local participants (e.g., interface or channel) requires further handling schemes. Due to the resource constraints of local devices, deep models cannot completely compute and upload for a satisfying accuracy objective. Therefore, to improve the FL performance, an intelligent computational framework for edge applications with multi-exit-based and greedy-approach-based algorithms is discussed by splitting the main models into sub-models with various scale sizes to prioritize the local participants that have inadequate computing capacities [17; 37]. To formulate and handle the problems efficiently, latency-constrained and optimization models must correspond with the observable features of training-time metrics, allocation, and scheduling procedures. FL introduces a sufficient framework that strongly takes client personalization into consideration, and it is highly efficient for multi-level IoT privacy environments.