# A DQN-Based Multi-Objective Participant Selection for Efficient Federated Learning
## 3 System Model
### Workflow for Federated Learning Training  
The core objective of FL is to enable efficient model training among multiple terminals, while ensuring the security and privacy of data communication. After training the model using the local data, the terminals need to upload the local model to the central server for aggregation aggregating the global model. The steps are as follows:  
(1) Participants are selected for this training round. For a federated learning task \(i\in I\), the dataset collection within that task is denoted by \(\sum\limits_{d\in N}H_{d}\). (2) The selected end devices are trained using local data in a local training process, and the locally trained model is obtained by minimizing the loss function of the local training. The loss function \(l_{d}^{i}(x_{d},y_{d};\omega_{d})\) of end device \(d\) during the local training process for federated learning task \(i\) is defined as the distinction between its predicted and actual values on sample dataset \(H_{d}\). Therefore, the loss function of federated learning task \(i\) on all datasets can be defined as \(L^{i}(\omega)=\frac{1}{|R_{i}|}\sum\limits_{d\in N}l_{d}^{i}(x_{d},y_{d};\omega _{d})\), where \(\omega\) denotes the weight of \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameters** & **Meaning** \\ \hline \(H_{d}\) & The dataset of terminal \(d\) participating in federal learning \\ \(R_{i}\) & Total dataset related to FL task \(i\) \\ \(|H_{d}|\) & The size of the dataset for terminal \(d\) participating in FL tasks \\ \(|R_{i}|\) & The size of the total dataset for this federal learning task \\ \(Z_{i}\) & Collection of attributes for FL task \(i\) \\ \(C_{i}\) & The number of CPU cycles required for calculating a set of data in dataset \\ \(M_{i}^{0}\) & Initial model of FL task \(i\) \\ \(l_{d}^{i}\) & Loss function of device \(d\) when performing local training of FL task \(i\) \\ \(\omega_{d}^{n}\) & The model parameters of device \(d\) at the nth training session \\ \(\eta\) & Learning rate \(d\) of device when performing local training of FL task \(i\) \\ \(A_{i}\) & Sum of loss functions for the FL task \(i\) test dataset \\ \(T_{d}^{\text{local}}\) & The local computation time of FL task \(i\) \\ \(T_{d}^{trans}\) & The transmission time of FL task \(i\) \\ \(f_{d}\) & The CPU cycle frequency of device \(d\) in executing FL task \(i\) \\ \(r_{d}\) & The transfer rate of task \(i\) in FL between the device and the server \\ \(G\) & The number of CPU cycles required per data for device in the FL task \(i\) \\ \(s_{i}^{l}\) & The environmental state at time \(t\) \\ \(H_{i}^{t-1}\) & The dataset of terminal at the previous time step \\ \(a_{i}^{l}\) & The action space at time \(t\) \\ \(\beta_{i,d}\) & Whether device \(d\) is selected to participate in FL task \(i\) \\ \(T_{i}\) & The max delay of clients participating in this iteration \\ \(\pi\) & A policy is a mapping from a state space to an action space \\ \(a^{t}\) & Discount factor \\ \hline \hline \end{tabular}
\end{table} 
Table 2: System parameters. the current model to be trained and \(|H_{i}|\) denotes the size of the total dataset for the task, also \(|R_{i}|=\sum\limits_{d\in N}|H_{d}|\). (3) The devices involved in the training process upload their locally generated models to the server, which aggregates them to produce a global model. (4) After aggregating the local models, the central server redistributes the resulting global model to the end devices, which then update their own model parameters and initiate the subsequent round of local training. The objective of federated learning (FL) is to optimize the global model parameters, denoted as \(\omega=\arg\min L^{i}(\omega)\), by minimizing the loss function \(L^{i}(\omega)\) associated with the task. In this paper, we adopt the stochastic gradient descent (SGD) method for updating the parameters of the FL model, where one randomly selected data point \(\{x_{d},y_{d}\}\) from the dataset is used for each update.