## 4 Federated System
### Simulation Setup  
To carry out the implementation of the proposed FDRL system, the open source framework Flower has been used. Flower is a novel FL framework that supports large-scale training and evaluation, both on real edge devices and on single or multi-node computational clusters. This enables scalable algorithmic investigation of real-world system conditions [20]. One of the main advantages provided by Flower, in addition to being language and framework independent of ML by design, is that it is fully extensible and can incorporate emerging algorithms, training strategies, and communication protocols. This makes it highly flexible and allows it to effectively incorporate the Stable Baselines3 DQN algorithms implemented to develop the centralized stage models [21]. To carry out this integration, it was necessary to study the data structures used by each of these frameworks to represent the neural network  
Figure 5: Proposed implementation scenario for the federated stage. parameters (the only information present in the communication between clients and server) in order to create an effective compatibility. This enabled leveraging Flower's functionalities to manage the federation and execute commands on clients deploying DQN models built using the Stable Baselines3 library. In preparation for the simulation phase, a set of slice requirements was defined, as outlined in Table 3. These requirements associate two distinct slice types with each typical use case in the 5G landscape (eMBB, URLLCs, and mMTC). Each slice type specifies the required resources in terms of MEC parameters (CPU, RAM, storage, and bandwidth) and RAN parameters (minimum transmission rate and maximum latency, denoted as \(R\) and \(L\), respectively, in Table 3). To generate the slice requests that form the synthetic dataset used for model training, two types of slice requests were defined for each slice type, resulting in 12 unique demand types. Each slice request was generated using an arrival rate that determines the time interval between successive requests, following an exponential distribution, while the request life cycle adhered to a Poisson distribution. The distribution parameters were kept constant for each instance of the implemented O-RAN system. During the development of the training phase, various hyperparameter combinations were tested to identify the most effective configuration in terms of convergence. The number of neurons in the hidden layer was determined after evaluating models trained with 4, 8, 16, and 32 neurons.The exploration rate was adjusted between 0.25 and 0.6 to optimize the training phase dedicated to exploitation. Additionally, the learning rate was evaluated over the range of 0.01 to 0.0001, utilizing discrete values corresponding to negative powers of 10 within this interval. In the \(\epsilon\)-greedy strategy, the balance between exploration and exploitation is controlled by the \(\epsilon\) parameter, which determines the probability of selecting a random action versus choosing the action with the highest estimated Q-value. To analyze the impact of different exploration-exploitation trade-offs, a series of tests were conducted with initial exploration values of \(\epsilon_{0}=0.25,0.5,\) and 0.75, with each defining the proportion of the training phase allocated to exploration. The training process was divided into two main phases:  
* Exploration Phase:
* The agent randomly explores different actions to gather diverse experiences. * The \(\epsilon\)-value follows an exponential decay function to gradually shift from exploration to exploitation. * Exploitation Phase:
* Once \(\epsilon\) reaches zero, the agent exclusively selects actions with the highest Q-values, leveraging the knowledge accumulated during exploration.