## 4 DDPG-Based Offloading Design  
Note that only the discrete actions can be handled by the DQN-based offloading design, where the reward acquisition mainly depends on solving the formulated optimization problems at MEC servers, which may increase the extra computing burden at the MEC servers. In this section, we rely on the DDPG to design offloading policy, considering that DDPG can deal with discrete and continuous value actions. Different from DQN, DDPG uses the Actor-Critic network to improve the accuracy of the model. In this section, we directly regard \(a(t)\), \(\alpha_{k}(t)\), \(p_{k}^{l}(t)\), \(p_{k}^{o}(t)\), and \(p_{m,k}^{c}(t)\) as the output action instead of disassembling the problem into two parts. **System State Space**: In the DDPG offloading paradigm, the system state space action is the same as the DQN-based offloading paradigm, which is given by  
\[S(t)=\big{\{}\mathbf{h}(t-1),\mathbf{C}(t)\big{\}}, \tag{27}\]  
where \(\mathbf{h}(t-1)\) and \(\mathbf{C}(t)\) are defined in (6). As in the DQN offloading paradigm, the agent can only observe the delayed version of CSI due to channel estimation operations and feedback delay. **System Action Space**: In the DDPG offloading paradigm, the value of \(p_{m,k}^{c}(t)\) is utilized to indicate the MEC server selection, where \(p_{m,k}^{c}(t)=0\) represents that there is no partial task at intelligent terminal \(k\) offloaded to the MEC server \(m\). In other words,  
Figure 5: The structure of the DQN-based offloading algorithm. the MEC server \(m\) is not chosen by intelligent terminal \(k\). If \(p_{m,k}^{c}(t)\) is not equal to 0, it means that the intelligent terminal \(k\) decides to offload partial tasks to the MEC server \(m\). Since the intelligent terminal can only connect to one MEC server at one time slot, only one \(p_{m,k}^{c}(t)\) in any time slot is not 0, and the remaining ones are 0. The action space of the DDPG offloading paradigm can be expressed as  
\[a(t)=\big{\{}a_{k}(t),p_{k}^{l}(t),p_{k}^{o}(t),p_{m,k}^{c}(t)\big{\}},\forall k,m. \tag{28}\]  
It is noted that here the continuous actions \(p_{k}^{l}(t),p_{k}^{o}(t),p_{m,k}^{c}(t)\) can be obtained based on state \(S(t)\) with delayed CSI \(\mathbf{h}(t-1)\). **System Reward Function**: In the DDPG offloading algorithm, \(\alpha_{k}(t)\), \(p_{k}^{l}(t)\), \(p_{k}^{o}(t)\), and \(p_{m,k}^{c}(t)\) can be obtained from a continuous action space. With the decisions, the agent tells each intelligent terminal \(k\) the selected MEC server and delivers \(p_{k}^{l}(t)\) and \(p_{k}^{o}(t)\) to it to perform the offloading. Moreover, the agent needs to send \(p_{m,k}^{c}(t)\) to each server to allocate computing resources. After that, the reward is obtained as in (26) by collecting \(t_{k}\) observed at the MEC servers or intelligent terminals. Compared to the DQN-based offloading paradigm, the DDPG-based offloading paradigm does not need the MEC servers to solve the optimization problems, which can release the computation burden at the MEC servers. However, as the DDPG algorithm is generally more complex than the DQN algorithm, the computation complexity unavoidably increases at the agent. The structure of the DDPG-based offloading algorithm is illustrated in Figure 6. We provide the pseudocode in Algorithm 2.