## 4 Industrial Federated Deep Deterministic Policy Gradient
The system state \(s_{i,t}\) of the terminal device \(i\) in the current epoch \(t\) can be defined as \[s_{i,t}=(T_{t}^{\max},\mathcal{P}_{i},\mathcal{P}_{j},I_{i,r}^{a},I_{i,t}^{a}),\] (17) where \(T_{t}^{\max}\) is the maximum delay threshold for completing the task, \(\mathcal{P}_{i}^{u}\) is the location information of the mobile terminal device, \(\mathcal{P}_{j}^{u}\) are the location information of the mobile edge servers, specifically represented by \(\mathcal{P}_{j}^{u}=\{P_{1},P_{2},\ldots,P_{N}\}\), \(I_{i,r}^{a}\) is the remaining data size of the task chain at the current epoch \(t\), and \(I_{i,t}^{a}\) denotes the magnitude of task data that must be computed in the current epoch \(t\). 2. Action Space: In our scenario, the agent selects actions based on the environment observed by the MEC system and its current state. The actions include task offloading ratio, number of subcarriers, subcarrier transmission power, and computing frequency of the CPU in mobile smart terminal devices. The action \(a_{i,t}\) of the terminal device \(i\) in the current epoch \(t\) can be expressed as \[a_{i,t}=(\mathbf{O}_{i,t},\mathbf{W}_{i,t},\mathbf{P}_{i,t}^{Up},\mathbf{\ell }_{i,t}^{u}),\] (18) where \(\mathbf{O}_{i,t}\) is a vector of offloading ratios for offloading to edge servers, which can be specifically represented as \(\mathbf{O}_{i,t}=\{O_{i,1,t},O_{i,2,t},\ldots,O_{i,N,t}\}\), corresponding to different edge servers. \(\mathbf{W}_{i,t}\) is the vector of subcarrier numbers selected by the terminal device for offloading, which can be specifically represented as \(\mathbf{W}_{i,t}=\{w_{i,1,t},w_{i,2,t},\ldots,w_{i,N,t}\}\), corresponding to different edge servers. \(\mathbf{P}_{i,t}^{Up}\) is the vector of transmission powers used for offloading to the edge server, which can be specifically represented as \(\mathbf{P}_{i,t}^{Up}=\{\mathcal{P}_{i,1,t}^{Up},\mathcal{P}_{i,2,t}^{Up}, \ldots,\mathcal{P}_{i,N,t}^{Up}  
objective is to maximize the reward. As a result, energy consumption should be inversely associated with the reward function. The reward function is given by  
\[r_{i,t}=r_{i,t}(s_{i,t},a_{i,t})=-C\cdot E_{i,t}^{sum}, \tag{19}\]  
where the reward parameter \(C\) is a constant greater than zero. Its purpose is to enhance the effect of the reward and to highlight the quality of the current action more clearly. The DDPG algorithm can be used to find the action that maximizes the value function \(q\) and achieve lower energy consumption for individual terminal devices. The policy \(\pi\) maps each state to a corresponding action. Given a particular state, the policy defines a probability distribution over the actions that the agent may take:  
\[\pi(a|s)=\mathbb{P}(A_{t}=a|S_{t}=s). \tag{20}\]  
The policy is stationary and does not change over time. As a result, energy consumption should be inversely associated with the reward function. It can be expressed as  
\[U_{t}=\sum_{i=t}^{n}\gamma^{t-t}R_{i}, \tag{21}\]  
where the discount factor \(\gamma\in[0,1]\), and \(R_{i}\) represents the reward for the epoch \(t\) that has not yet been observed. The action value function in MDP calculates the expected sum of rewards that will be obtained by taking a particular action \(a\) in the state \(s\). It represents the value of the action. The action value function is given by  
\[Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}[U_{t}|S_{t}=s_{t},A_{t}=a_{t}]. \tag{22}\]  
According to the Bellman equation, the recurrence relationship of the action value function can be expressed as  
\[Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}[R_{t+1}+\gamma Q_{\pi}(S_{t+1},A_{t+1})| S_{t}=s_{t},A_{t}=a_{t}]. \tag{23}\]  
The significance of the action value function is that it can evaluate the future total value after taking action \(A_{t}\) in the state \(S_{t}\). Therefore, the algorithm finds the best policy by choosing the action that yields the highest value of \(Q\). Thus, the optimal action value function \(Q_{*}(s_{t},a_{t})\) finds the best course of action in each state.