## 4 Industrial Federated Deep Deterministic Policy Gradient
Finally, the algorithm performs a gradient descent to renew the parameters \(\mathbf{w}_{c}\) of the value network, which is given by  
\[\mathbf{w}_{c}\leftarrow\mathbf{w}_{c}-\theta\cdot\delta_{t}\cdot\frac{ \partial q(s_{t},a_{t};\mathbf{w}_{c})}{\partial\mathbf{w}_{c}}, \tag{30}\]  
where \(\frac{\partial q(s_{t},a_{t};\mathbf{w}_{c})}{\partial\mathbf{w}_{c}}\) is the gradient, and \(\theta\) is the network learning rate. By performing gradient descent, we can reduce the TD error and close the difference between the forecasts of the value network and the TD target. The convergence analysis of the gradient update is shown in Appendix A. Inspired by FL, in the Industrial Internet system composed of multiple devices, the network parameters \(\mathbf{w}_{a,i}\) and \(\mathbf{w}_{c,i}\) corresponding to the terminal device \(i\) can be uploaded to the cloud node connected to the edge server. The network parameters \(\mathbf{w}_{a,i}\) and \(\mathbf{w}_{i}\) are called local models. The cloud node processes the uploaded local model using federated averaging to obtain the global model:  
\[\mathbf{w}^{+}_{a}=\frac{1}{M}\sum_{i=1}^{M}\mathbf{w}_{a,i}, \tag{31}\]  
\[\mathbf{w}^{+}_{c}=\frac{1}{M}\sum_{i=1}^{M}\mathbf{w}_{c,i}, \tag{32}\]  
where \(\mathbf{w}^{+}_{a}\) and \(\mathbf{w}^{+}_{c}\) are the policy network parameters and value network parameters after the federation average, \(M\) is the number of terminal devices in the system, and each terminal device corresponds to a DDPG algorithm. Next, the cloud node transmits the network parameters of the global model, i.e., \(\mathbf{w}^{+}_{a}\) and \(\mathbf{w}^{+}_{c}\), to the policy and value networks of each DDPG algorithm running on the edge server, replacing the original network parameters \(\mathbf{w}_{a}\) and \(\mathbf{w}_{c}\). At the same time, target networks also need to update their parameters. Set a hyper-parameter \(\tau\) and \(\tau\in(0,1)\). Take the weighted average of \(\mathbf{w}_{c}\) and \(\mathbf{w}^{-}_{c}\) to obtain a new \(\mathbf{w}^{-}_{c}\):  
Figure 2:0  
Similarly, take the weighted average of \(\mathbf{w}_{a}\) and \(\mathbf{w}^{-}_{a}\) to obtain a new \(\mathbf{w}^{-}_{a}\):  
Figure 2:1In the end, after the algorithm obtains better value network parameters \(\mathbf{w}_{c}\) and policy network parameters \(\mathbf{w}_{a}\), it outputs better actions \(a_{t+1}\) in the next iteration. At the conclusion of the current iteration, the algorithm assigns the current state \(s_{t+1}\) to \(s_{t}\) to complete the closed loop of the algorithm. Algorithm 1 shows the IF-DDPG algorithm. ```
1:Initialize the state \(s_{t}\)
2:if the specified number of epochs has not been completed then
3: Input the state \(s_{t}\) into the policy network
4: The policy network outputs the noise-added determinate action \(a_{t}=\pi(s_{t};\mathbf{w}_{a})\)
5: Input \(s_{t}\) and \(a_{t}\) into the environment results in \(r_{t}\)
6: Initialize the state \(s_{t+1}\)
7: Get a resource transition: \(s_{t},a_{t},r_{t},s_{t+1}\) and store it in the experience replay buffer
8:if the buffer capacity is full then
9: Extract a resource transition \(s_{t},a_{t},r_{t},s_{t+1}\) from the buffer
10: Use the deterministic policy gradient to calculate the gradient \(\mathbf{g}_{t}=\frac{\partial\mathbf{q}[s_{t},r(s_{t};\mathbf{w}_{a})\mathbf{ w}_{c}]}{\partial\mathbf{w}_{a}}=\frac{\partial a_{t}}{\partial\mathbf{w}_{a}}\).