## 3 DQN-Based Offloading Design
### Reward Function  
In the DQN paradigm, the reward is defined as the maximum time delay required to complete all the tasks received at all intelligent terminals. After taking the actions, a dedicated MEC server can calculate the time delays required for the intelligent terminals choosing this MEC server to offload, as all MEC can observe the local CSI. With the loss of generality, we assume that intelligent terminal \(k\) with \(k\in\mathcal{O}_{k}\) offloads the tasks to MEC \(m\), where set \(\mathcal{O}_{k}\) defines the indexes of the intelligent terminals selecting MEC server \(m\) to offload tasks. To minimize the required time delays, the MEC server needs to formulate an optimized problem to find the optimal \(\alpha_{k}(t)\), \(p_{k}^{l}(t)\), \(p_{k}^{o}(t)\), and \(p_{m,k}^{c}(t)\). It is worth noting that as the MEC server knows the instantaneous CSI at time \(t\), the solution can be obtained based on \(\mathbf{h}(t)\), which is different from the MEC server selection taken based on \(\mathbf{h}(t-1)\). For the intelligent terminals which do not offload tasks to the MEC servers, the required time delays for local task processing can be known by these intelligent terminals. The agent collects all the time delay consumptions from the intelligent terminals and the MEC servers to obtain the final reward. We detail how to compute the time delay for intelligent terminal \(k\), assuming that it selects MEC server \(m\) to offload. The total time consumption for completing the task processing at intelligent terminal \(k\) is denoted as \(t_{k}\), which equals to \(t_{k}=\max\{t_{k}^{l},t_{m,k}^{o}+t_{m,k}^{c}\}\) where \(t_{k}^{l}\), \(t_{m,k}^{o}\), and \(t_{m,k}^{c}\) denote the times required for intelligent terminal local task processing, task offloading transmission from intelligent terminal \(k\) to MEC server \(m\), and task processing at MEC server, respectively. With the computation rate \(r_{k}^{l}\) defined in Equation (4), time \(t_{k}^{l}\) can be represented as  
\[t_{k}^{l}=\frac{\alpha_{k}C_{k}(t)}{r_{k}^{l}(t)}. \tag{9}\]As the size of the offloaded task is \((1-\alpha_{k})C_{k}\) with the communication rate defined in Equation (2), time \(t^{o}_{m,k}\) can be calculated as  
\[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]  
With the computation rate \(r^{c}_{m,k}\) allocated by MEC server \(m\) to intelligent terminal \(k\) in Equation (4), time \(t^{c}_{m,k}\) can be computed as  
\[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]  
To maximize the reward, we need to minimize the time delay for each intelligent terminal under the total energy constraint at intelligent terminals and MEC servers. To illustrate the way to find optimal \(\alpha_{k}(t)\), \(p^{l}_{k}(t)\), \(p^{o}_{k}(t)\), and \(p^{c}_{m,k}(t)\) for different types of MEC server selection, we next present two typical offloading scenarios, that is, an MEC server serves one intelligent terminal and an MEC server serves two intelligent terminals. It is noted that the proposed way of solving \(\alpha_{k}(t)\), \(p^{l}_{k}(t)\), \(p^{o}_{k}(t)\), and \(p^{c}_{m,k}(t)\) can be extended to the case where an MEC server serves arbitrary number of intelligent terminals. _Scenario 1: one MEC server serves one intelligent terminal_  
The energy consumption at intelligent terminal \(k\), denoted by \(E_{k}\), includes two parts, i.e., one part for local partial task processing and another for partial task transmission. Therefore, \(E_{k}\) can be written as  
\[E_{k}=p^{l}_{k}(t)t^{l}_{k}+p^{o}_{k}(t)t^{o}_{m,k}. \tag{12}\]  
The energy consumption at the MEC server \(m\) for processing the partial task offloaded from intelligent terminal \(k\) is denoted by \(E_{m,k}\), and can be represented as  
\[E_{m,k}=p^{c}_{m,k}(t)t^{c}_{m,k}. \tag{13}\]  
The optimization problem formulated to find optimal \(x_{k}(t)=\{\alpha_{k}(t),p^{l}_{k}(t),p^{o}_{k}(t),p^{c}_{m,k}(t)\}\) is given by  
\[\min_{x_{k}(t)} t_{k} =\max\{t^{l}_{k},t^{o}_{m,k}+t^{c}_{m,k}\}\] (14a) s.t.