# Node Screening Method Based on Federated Learning with IoT in Opportunistic Social Networks
## 3 Methods
### Description of the Transmission Process
As shown in Figure 3, PPO, as a reinforcement learning algorithm based on the AC framework, limits the policy update range by means of regular terms,which solves the problem that the traditional policy gradient update step size is difficult to determine [45; 46; 47; 48; 49; 50]. To further improve the convergence speed, the DPPO-based node selection algorithm uses multiple threads to collect data in the environment, and multiple threads share a global PPO network. \[U_{\lambda}=\sum_{m\in M_{\lambda}}\sum_{n\in N_{\lambda}}U_{m,n} \tag{4}\]3 Restriction strategy update range indication. \[U_{\lambda}=\sum_{m\in M_{\lambda}}\sum_{n\in N_{\lambda}}U_{m,n} \tag{4}\]4 DRL-based node selection framework. In this paper, the federated learning node selection problem is expressed as an MDP model, and then a DPPO-based node selection algorithm is designed to solve the problem. The specific design is described in detail below. State space. The environmental state \(E_{\lambda}^{t}\) at time t can be represented by a four-tuple \(E_{\lambda}^{t}=\left\{I_{\lambda},R_{\lambda}^{t},U_{\lambda}^{t-1},z_{ \lambda}^{t-1}\right\}\), where \(I_{\lambda}\) represents the federated learning task \(\lambda\) information, \(R_{\lambda}^{t}\) represents the resources that the terminal device can use for the federated learning task \(\lambda\) at time t, \(U_{\lambda}^{t-1}\) represents the data set of the terminal device at the previous time, and \(z_{\lambda}^{t-1}\) represents the previous Node selection scheme at the moment. Action space. In the action selection of each step, the agent is only allowed to adapt one node selection scheme, and the node selection scheme of the federated learning task \(\lambda\) at time t is modeled as a 0-1 binary vector \(\gamma_{\lambda}^{t}=\{b_{\lambda,1},b_{\lambda,2},b_{\lambda,3},\ldots,b_{ \lambda,n}\},b_{\lambda,n}\in\{0,1\}\), where \(b_{\lambda,n}=1\) means that the device numbered n is selected in this node selection, and 0 means it is not selected. Therefore, the weight aggregation after node selection is expressed as  
\[\mu_{a}^{*}=\mu_{a}^{\lambda}+\sum_{m\in M_{\lambda}n\in N}\sum_{ \begin{subarray} {c}|U_{m,n}|\left(\mu_{m,n}^{*}-\mu_{m,n}\right)\\ |U_{\lambda}|\end{subarray} }b_{\lambda,n} \tag{15}\]  
Reward function. When the agent executes a certain action according to a certain node selection strategy, the environmental information will change accordingly and obtain a reward value for evaluating this behavior. This paper considers the design of the reward function based on the test accuracy of federated learning and sets the maximum delay as the constraint of each action selection. The reward function can be expressed as  
\[v_{\lambda}^{t}=\frac{-1}{\sum_{n\in N}b_{\lambda,n}}D^{\lambda}\Big{(}x_{ test},y_{test};\mu_{a}^{\lambda}\Big{)} \tag{16}\]  
The source of the executive action above is a strategy \(\theta\), which is a mapping from the state space to the action space, it can be expressed as  
\[F_{\lambda}^{t}=\theta\big{(}E_{\lambda}^{t}\big{)} \tag{17}\]  
The goal of the MDP model is to obtain an optimization strategy, i.e., after taking corresponding actions according to the strategy in the corresponding state, the goal of reinforcement learning-the expectation of cumulative return will be the largest, i.e., solving  
\[\theta^{*}=argmax_{\theta}\text{T}[\sum_{t=0}^{\infty}\phi^{t}v_{\lambda}^{t}] \tag{18}\]  
where \(\phi^{t}\) is the discount factor, and its value decreases with the increase of time. The following describes the node selection algorithm based on DPPO-based federal learning. There are two Actor networks (Actor1 and Actor2) and a Critic network in the global PPO network. Actor1 represents the current latest strategy \(\theta\) and is responsible for guiding each thread to interact. The Critic network evaluates the current strategy according to the rewards obtained after the agent performs the node selection action, and realizes the update of Critic network's parameters through the back propagation. Actor2 represents the old strategy \(\theta_{old}\).