# A DQN-Based Multi-Objective Participant Selection for Efficient Federated Learning
## 5 Simulation Analysis  
### Experimental Settings  
In this study, we simulate and validate the algorithm using Python 3.7 and TensorFlow 2.2.0 environments. Our experiments emulate a distributed FL training scenario with various categories of devices. The setup consists of an aggregation server and 10-80 devices.  
To demonstrate the robustness of the proposed approach, malicious nodes are introduced in the experiments to simulate devices with subpar training quality. During server aggregation, the parameters of a node can be modified to random values, simulating the presence of a malicious node in local training [39]. We initially select the MNIST dataset for training. The dataset is uniformly partitioned and allocated to the nodes as the local dataset. In addition, the CIFAR dataset and the Fashion Mnist are used in this study to validate the efficacy of the proposed algorithm.  
In our simulation, the client device's CPU cycle frequency \(f_{d}\) adheres to a uniform distribution \(U[0,1]\) while the wireless bandwidth \(r\) follows a uniform distribution \(U[0,2]\), \(\alpha_{i}=2\)[40] This setup allows for a diverse range of computational and communication capabilities among the devices.  
A convolutional neural network (CNN) serves as the FL training model, featuring a structure that consists of six convolutional layers, three pooling layers, and one fully connected layer. The DQN algorithm uses four threads to interact with the environment and collect empirical data. The reward discount factor is set to 0.9, and the learning rate of the Q network (value network) is configured at 0.0001. The target network is updated with the parameters of the Q network after every 100 rounds of agent training. In addition, the buffer size for experience replay is set to 10,000. The settings of specific experimental parameters are shown in Table 3.  
In this study, we compare the proposed algorithm (FL-DQN) with three alternative approaches:  
1. FL-Random: This algorithm does not utilize deep reinforcement learning for node selection during each iteration of FL training. Instead, it selects nodes at random.
2. FL-Greedy: The algorithm selects all participating nodes for model aggregation in each iteration of the FL training.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Parameter Type** & **Parameter** & **Parameter Description** & **Parameter Value** \\ \hline \multirow{8}{*}{Equipment and model parameters} & E & Number of terminals & 100 \\  & \(f_{d}\) & CPU cycle frequency & [0, 1] \\  & \(r_{d}\) & Wireless Bandwidth & [0, 2] \\  & \(H_{d}\) & Local datasets & 600 \\  & \(\zeta\) & Local Iteration & 2 \\  & N & Minimum sample size & 10 \\  & \(\alpha\) & Learning Rate & 0.01 \\  & Node & Number of nodes involved & [10, 80] \\  & \(f_{bt}\) & Number of CPU cycles required for & 7000 \\  & & training per data bit & \\  & \(|R_{i}|\) & Global Model Size & 20 Mbit \\ \hline \multirow{8}{*}{DQN parameter} & A & Agents & 4 \\  & s & Training steps & 1000 \\  & Target Q & Q Network & 0.0001 \\ \cline{1-1}  & \(a^{t}\) & Bonus Discount Factor & 0.9 \\ \cline{1-1}  & circle & Strategy Update Steps & 100 \\ \cline{1-1}  & \(E_{r}\) & Experience replay buffer & 10,000 \\ \cline{1-1}  & B & Batch-size & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Simulation parameter setting.  
* Local Training: This approach does not incorporate any FL mechanism, and the model is solely trained on individual local devices [36]. These comparisons help to assess the relative effectiveness and efficiency of the FL-DQN algorithm.