# A DQN-Based Multi-Objective Participant Selection for Efficient Federated Learning
## 4 FL Node Selection Algorithm Based on DQN
### DQN-Based Algorithm for FL Node Selection  
To identify the best course of action, standard Q-Learning is commonly used. Q-Learning constructs a constantly updated Q-value table with action states, and then looks up the Q-value table to get the best decision for \(Q(s^{t}_{i},a^{t}_{i})\)[37]. The edge server updates the Q-values based on the empirical replay as follows:  
\[Q^{\prime}\Big{(}s^{t+1}_{i},a^{t+1}_{i}\Big{)}=(1-\eta)Q\big{(}s^{t}_{i},a^{t }_{i}\big{)}+\eta\left[R\big{(}s^{t}_{i},a^{t}_{i}\big{)}+\alpha^{t}\max_{d^{t }_{i}\in A}Q\big{(}s^{t}_{i},a^{t}_{i}\big{)}\right] \tag{12}\]  
where \(\eta\) denotes the learning rate and \(\alpha^{t}\) denotes the discount rate. Following an update to the \(Q(s^{t}_{i},a^{t}_{i})\), the server can utilize it to carry out its operations. Using any given state \(s^{t}_{i}\), the serve can select the optimal decision \(\pi^{*}\) by choosing the action with the highest cumulative reward \(a^{t}_{i}\)[38]. However, the Q-table can be resource-intensive, and the search time for the optimal policy within the table can be prolonged. To address this, we propose the use of the DQN which employs a single neural network (NN) for optimal decision-making, thereby reducing the storage requirements for the Q-value table and accelerating the search process. Convolutional neural networks (CNNs) serve as a function approximation for Q-Tables in high-dimensional and continuous state. However, in function optimization problems, the conventional approach of supervised learning involves first determining the loss function, followed by finding the gradient and updating the parameters using techniques such as stochastic gradient descent. In contrast, the DQN algorithm is built on Q-Learning to determine the loss function. Here, we define the loss function as follows:  
\[L(\theta)=E\Big{[}\big{(}\text{Target}\,Q-Q\big{(}s^{t}_{i},a^{t}_{i};\theta \big{)}\,\big{)}^{2}\Big{]} \tag{13}\]  
where \(\theta\) is the network parameter. The definition of Target \(Q\) is :  
\[\text{Target}\,Q=R\big{(}s^{t}_{i},a^{t}_{i}\big{)}+\alpha^{t}\max_{d^{t}_{i}} Q\Big{(}s^{t^{\prime}}_{i},a^{t^{\prime}}_{i};\theta\Big{)} \tag{14}\]  
The DQN algorithm utilizes the Experience Replay mechanism to store learned data in a cache pool, which is then randomly sampled for subsequent training. When new nodes join the federated learning, we add a new list to store the new nodes. Each training session preferentially selects a node from this list, and deletes the new node from the new list after its selection. When a new node is added, the new node executes a startup script to add itself to the new list. Node failure in this experiment is defined as failure to return training results after training. After each training session, faulty nodes are judged and processed offline, that is, the nodes are removed from the Q-Table. Upon conducting an in-depth analysis of the DQN, we present a DQN-inspired federated learning (FL) node selection algorithm. This approach consists of two main phases: multi-threaded interactions and global network updates. 1. Multi-threaded interactions Step 1
2. Each worker thread is assigned a replica of the environment and a local copy of the DQN network. In the context of FL tasks, the environment emulatesthe behavior and performance of client devices, while the network serves to implement policies within the local environment. Each worker thread independently interacts with a replica of its environment to gather empirical data including states, actions, rewards, and new states. This information is used to train the DQN network. Threads can interact concurrently with their assigned environments, thereby speeding up the data collection process. The experience data collected by individual threads is stored in a shared experience replay buffer. This buffer can be used to randomly sample batch data.