## 4 Federated Client Selection Scheme
### DRL-Based Client Selection  
The goal of the MDP model is to obtain an optimization policy \(\pi^{*}:S\to A\), i.e., to maximize the long-term cumulative return of the whole system after adopting the corresponding action according to this policy in the corresponding state. To find the best decision, standard Q-Learning is usually used. Q-Learning uses action-state pairs to construct a continuously updated table of Q-values, then it looks up the Q-table for optimal decisions. The QL algorithm is only applicable to networks with a finite state and action space. When the number of network participants increases, the problem of assigning training tasks to devices becomes highly dimensional, and the Q-tables require significant storage resources. To solve high-dimensional problems, Deep Q Network (DQN) algorithms (a combination of QL and deep neural networks) come into play. A DQN takes the current state of the environment as input and outputs the Q-value \(Q(s,a)\) as well as the weight matrix of the neural network \(\theta\) through its network structure. In order to obtain an approximate value, the DQN needs to be trained using the experience \((s,a,R(s,a),s^{\prime})\), where \(s^{\prime}\) represents the new state the agent moves to from the current state \(s\) after performing action \(a\). The server updates the Q-value based on empirical playback, gradually improving the efficiency and accuracy of client selection. The specific updates are as follows:  
\[Q^{\prime}(s,a)=(1-\beta)Q(s,a)+\beta\left[R(s,a)+\gamma\max_{a^{\prime}\in A }Q\big{(}s^{\prime},a^{\prime}\big{)}\right] \tag{20}\]  
where \(\beta\) is the learning rate and \(\gamma\) is the discount factor. By training the neural network, we can obtain an approximate Q-function for decision. During the training of the neural network, we employ empirical playback and goal networks to enhance the stability of the algorithm. The empirical playback technique avoids correlations between successively sampled data, thus improving the effective utilization of samples. The target network is updated every certain number of training steps or training batches to reduce the correlation between the target Q value and the current Q value, thus making the model more stable and reliable. In order to define the loss function, we use the mean square error (MSE) as an evaluation metric. The DQN algorithm reduces the value of this loss function by optimizing the Bellman equation. The specific form of the loss function is as follows:  
\[L(\theta_{i})=E\Big{[}(TargetQ-Q(s,a;\theta_{i}))^{2}\Big{]} \tag{21}\]  
where \(\theta_{i}\) denotes the online network parameters for the \(i\)-th iteration, \(\theta_{i}^{\prime}\) denotes the target network parameters for the \(i\)-th iteration, and \(E[. ]\) denotes the expected value. The target \(Q\) is as follows:  
\[targetQ=R(s,a)+\gamma\arg\max_{a^{\prime}\in A}Q(s^{\prime},a^{\prime};\theta_ {i}^{\prime}) \tag{22}\]  
Deep Q Network (DQN) algorithms utilize an empirical replay mechanism to store learned data in a cache pool, which is then randomly sampled for subsequent training. The pseudo-code based on the DQN algorithm is provided in Algorithm 1. Before training begins, the FL server initializes the network, devices, system state, and experience playback buffer. In each training cycle, the agent selects an action \(a\) with probability \(\epsilon\) or action \(a\) with probability \((1-\epsilon)\) and observes all possible actions corresponding to the client's reward \(r\) and the next state \(s\) to find the best decision while storing the experience \((s,a,R(s,a),s^{\prime})\) in the experience playback buffer M. Then, \(m\) small batch samples are randomly selected from the experience playback buffer to update the network parameters. The algorithm terminates when the iteration reaches the upper bound ITER. Policy \(\pi\) selects the action that maximizes Q. ```
Input: Initial state of the Q-network Output: Discrete action selection strategy \(\pi\)  Initialize network, device, system state and experience playback buffers.