## 4 Industrial Federated Deep Deterministic Policy Gradient
The optimal action value function is given by  
\[Q_{*}(s_{t},a_{t})=\max_{\pi}Q_{\pi}(s_{t},a_{t}). \tag{24}\]  
The optimal policy is given by  
\[\pi_{*}(a|s)=\left\{ \begin{array}{ll}1&\text{if }a=\operatorname*{arg\,max} _{a_{t}\in\mathcal{A}}Q_{*}(s_{t},a_{t})\\ 0&\text{otherwise}\end{array} \right.. \tag{25}\]  
If \(Q_{*}\) can be known, the action \(a_{*}=\operatorname*{arg\,max}_{a_{t}\in\mathcal{A}}Q_{*}(s_{t},a_{t})\) that should be performed by the state \(S_{t}\) at any epoch can be obtained. However, the functional expression of \(Q_{*}\) is unknown in reinforcement learning; therefore, it is necessary to approximate \(Q_{*}\). One of the most common methods is the DQN algorithm [34]. It has a deep neural network \(Q(s_{t},a_{t};\mathbf{w}_{t})\) to approximate \(Q_{*}(s_{t},a_{t})\) and is trained using the temporal difference method. By using deep neural networks, the DQN algorithm solves problems with large state and action spaces. However, the DQN algorithm has low efficiency and is unable to handle continuous action space problems. An effective way to apply DRL methods such as DQN to continuous domains is to discretize the action space, but this introduces a significant problem, known as the curse of dimensionality: training becomes arduous due to the exponential rise in the number of actions that must be considered with each added degree of freedom. Furthermore, simply discretizing the action space will remove information about the structure of the action domain. The IF-DDPG algorithm is a DRL algorithm based on policy gradients. It consists of a policy network and a value network. The policy network controls the agent to take actions, and it makes actions \(a\) based on the state \(s\). The value network does not directly control the agent but instead provides a score for action \(a\) based on the state \(s\) to guide the policy network to make improvements. In addition, IF-DDPG enhances the stability and robustness of the algorithm by adopting concepts from DQN, such as experience replay and a separate target network, to minimize data correlation  
As shown in Figure 2, the policy network is called the Actor and is a deterministic function denoted by the function \(a=\pi(s;\mathbf{w}_{a})\), where \(\mathbf{w}_{a}\) corresponds to the policy network's parameters. The input of the policy network is the state \(s\), and the output is a specific deterministic action \(a\). Based on the state \(s\), the value network evaluates the quality of the action \(a\), which is also the output of the value network. The IF-DDPG algorithm requires training two neural networks so that they improve together. Enhancing the accuracy of the value network's scoring and the decision-making performance of the policy network is the desired outcome. The IF-DDPG algorithm is shown in Figure 3. First, it is necessary to update the policy network to make better decisions. Learning the policy network requires the deterministic policy gradient. The policy network calculates action \(a\) based on the input state \(s\) to control the movement of the agent, and thence, the policy network is also called an actor. Training the policy network requires the help of the value network. The value network \(q(s,a;\mathbf{w}_{c})\) evaluates the quality of the action \(a\) to guide the policy network to make improvements. The policy network is not equipped to determine the quality of an action, as only the value network provides the evaluation. A higher output value from the value network indicates a better action. Therefore, the goal is to improve the parameters of the policy network \(\mathbf{w}_{a}\) so that the output of the value network \(q(s,a;\mathbf{w}_{c})\) is larger. The input of a value network is the state \(s_{t}\) and action \(a_{t}\) at the epoch \(t\). The action \(a_{t}\) is computed by adding Gaussian noise to the output of the policy network \(\pi(s_{t};\mathbf{w}_{a})\). The reason for adding Gaussian noise is to improve policy exploration and increase the randomness of action selection.