## 4 Industrial Federated Deep Deterministic Policy Gradient
The first step in iterative algorithms is to initialize the state \(s_{t}\). For a given state \(s_{t}\), the policy network will output a deterministic action \(a_{t}\). Each iteration inputs \(s_{t}\) and \(a_{t}\) into the environment to obtain \(r_{t}\), initializing the state \(s_{t+1}\) and then obtaining a resource transition \(s_{t},a_{t},r_{t},s_{t+1}\). Experience replay is utilized to store resource transitions in the experience replay buffer. When the network is updated, when the buffer reaches full capacity, the algorithm extracts resource groups from the buffer for training purposes. For a fixed-input state of the value network, the only factor that affects the value is the parameter \(\mathbf{w}_{a}\) of the policy network. To accomplish the objective of  
Figure 2: Core of IF-DDPG. increasing the value network's output, we make use of the deterministic policy gradient for updating the policy network. The method requires computing the gradient of \(q(s_{t},a_{t};\mathbf{w}_{c})\) with respect to \(\mathbf{w}_{a}\) and then using gradient ascent to update \(\mathbf{w}_{a}\). This gradient is called the policy gradient \(\mathbf{g}_{t}\). The \(\mathbf{g}_{t}\) can be expressed as  
\[\mathbf{g}_{t}=\frac{\partial q[s_{t},\pi(s_{t};\mathbf{w}_{a});\mathbf{w}_{c }]}{\partial\mathbf{w}_{a}}\cdot\frac{\partial q(s_{t},a_{t};\mathbf{w}_{c})}{ \partial a_{t}}. \tag{26}\]  
This is the gradient of the value \(q\) with respect to the parameters \(\mathbf{w}_{a}\) of the policy network. The gradient could be calculated using the chain rule. This is equivalent to passing the gradient from the value \(q\) to the action \(a_{t}\) and then from \(a_{t}\) to the policy network. Finally, use gradient ascent to renew the parameters of the policy network \(\mathbf{w}_{a}\), which is given by  
\[\mathbf{w}_{a}\leftarrow\mathbf{w}_{a}+\beta\cdot\mathbf{g}_{t}, \tag{27}\]  
where \(\beta\) is the parameter that determines the speed of learning for the policy network. Updating \(\mathbf{w}_{a}\) allows the value \(q\) to become larger, i.e., the value network believes the policy will become better. After updating the policy network to improve decision making, the subsequent step is to update the value network to make its evaluation more accurate. The value network is also known as a critic. First, the value network is updated using the temporal difference (TD). Figure 3: Diagram of IF-DDPG. The purpose of TD is to make the results on both sides of the equation as close as possible. The equation for the TD can be expressed as  
\[q(s_{t},a_{t};\mathbf{w}_{c})\approx r_{t}+\gamma q(s_{t+1},a_{t+1};\mathbf{w}_{ c}). \tag{28}\]  
The value network predicts the action value at epoch \(t\), which is denoted by \(q_{t}=q(s_{t},a_{t};\mathbf{w}_{c})\). Next, the value network will predict the action value at the epoch \(t+1\) by incorporating the soft update using the target network. Given the state \(s_{t+1}\) at epoch \(t+1\), the input \(s_{t+1}\) is fed into the policy network \(\pi\) combined with the target network, which then computes the next action \(a^{\prime}_{t+1}\), denoted \(a^{\prime}_{t+1}=\pi(s_{t+1};\mathbf{w}^{-}_{a})\). The action \(a^{\prime}_{t+1}\) is not the action that the agent performs; it only updates the value network. Then, \(s_{t+1}\) and \(a^{\prime}_{t+1}\) are entered into the value network combined with the target network to compute the action value at epoch \(t+1\) and obtain \(\delta_{t}=q_{t}-(r_{t}+\gamma\cdot q_{t+1})\). Calculating the TD error is the subsequent step. The TD error is denoted as \(\delta_{t}\)  
\[\delta_{t}=q_{t}-(r_{t}+\gamma\cdot q_{t+1}), \tag{29}\]  
where \(r_{t}+\gamma\cdot q_{t+1}\) is the TD target, consisting of a portion that is the observed reward \(r_{t}\) and another portion that is the value network's prediction \(q_{t+1}\), where \(\gamma\) denotes the discount factor. The TD target is closer to the real value than a simple prediction of \(q_{t}\), which encourages \(q_{t}\) to approach the TD target to decrease the TD error.