## 2 Preliminary Studies on Federated Learning and Deep Reinforcement Learning
### Deep Reinforcement Learning for Experience-Driven Autonomy  
In this subsection, we provide a background review of DRL, which specifically refers to the handling of long-term network performance and real-time agent interactivity in the service deployment of IoT networks. This background presents the assisting capability of DRL and leads to the key integration with FL for self-organizing capabilities.  
[PERSON] decision process models an optimization problem with four-tuple components, namely \(\{S,A,R,P\}\), which represents the set of states, actions, rewards, and transition probability [53; 54]. At a particular time \(t\), state \(s_{t}\) represents the characteristic features of the environment, which are observed for feeding the agent. Action \(a_{t}\) consists of the updating parameters or configuration settings that the agent applies for purposes of improving the state \(s_{t}\) condition. Within general DRL, \(a_{t}\) is selected by function approximator using DNN. By modified DRL configuration (e.g., deep Q-networks), function approximators can be split into two primary networks, namely online and target. The separation procedure prevents an error of overoptimism. In the experience replays, denoted as \(e_{t}(s_{t},a_{t},r_{t},s_{t+1})\), the agent feeds \((s_{t},a_{t},r_{t})\) for current loss optimization and \(s_{t+1}\) for q-maximized action approximation. Over numerous iterations of exploration and exploitation, the non-optimal or near-optimal values are iteratively altered through the gradient process. Eventually, the optimal agent policy will be obtained. In every defined step, the weights are exchanged between online and target networks. To evaluate how well action \(a_{t}\) is applied in state \(s_{t}\), the agent used the reward function that was formulated by the specific performance targets for the state-action pair. The transition probability indicates the estimation after configuring \(a_{t}\) into \(s_{t}\), then transits the environment into \(s_{t+1}\). To classify the diversity of DRL types, Figure 4 lists the differences between used/unused criteria (e.g., value, policy, and model).  
Figure 4: Overview taxonomies of DRL.  
The features and functions are varied depending on application scenarios and performance objectives. The concept of a dynamic problem-solving algorithm, DRL, is used for modeling an optimal self-learning agent by interacting throughout exploration, understanding the possible maximum expected long-term rewards, and later exploitation. Optimal policy \(\pi^{*}\) is the policy-maximized long-term reward expectation. The value function can be formulated by the expectation of increasing reward summation following the policy \(\pi\) from state \(s\). The standard optimal q-value function follows the Bellman equation and epsilon \(\varepsilon\) indication [53].  
Table 3 presents the selected existing surveys on DRL that have been applied in communications and IoT applications. The review studies show how efficient DRL is in optimizing and self-organizing network control and orchestration. In [55], a multi-agent model-free DRL approach was proposed for assisting the framework of joint multi-channel and traffic control in SDN-based IoT environments. The system models were defined to show the optimization problems and DRL applicability. Within DRL specifications, the agent observed the state features as follows: (1) weights for channel capacity, transmission, and utilization, (2) channel/task numbers at a particular timeslot, (3) size of a particular task, (4) signal-to-interference-and-noise ratio between channels, and (5) state information of a particular timeslot's channel. The agent applied the actions of task index determination and the factorial of task numbers. A reward valuation was formulated by jointly considering the delay, throughput, and packet loss. As a result, the performance of a (multi-agent) DRL-based approach is highly efficient for experience-driven core network autonomy.