## 3 DQN-Based Offloading Design
### Reward Function
\right.. \tag{22}\]  
By alternating three subproblems with the solutions given in (19), (20), and (22) until convergence, we obtain the final solution. _Scenario 2: one MEC server serves two intelligent terminals_, e.g., intelligent terminal \(k\) and intelligent terminal \(k^{\prime}\), then the optimization problem can be formulated as follows  
min \[\max\left\{t_{k^{\prime}}^{o},t_{m}^{o}+t_{m,k^{\prime}}^{c},t_{k^ {\prime}}^{l},t_{m,k^{\prime}}^{o}+t_{m,k^{\prime}}^{c}\right\}\] (23a) s.t. \[p_{k}^{l}(t)t_{k}^{l}+p_{k}^{o}(t)t_{m,k}^{o}\leq E_{\max,k}\] (23b) \[p_{k^{\prime}}^{l}(t)t_{k^{\prime}}^{l}+p_{k^{\prime}}^{o}(t)t_{ m,k^{\prime}}^{o}\leq E_{\max,k^{\prime}}\] (23c) \[p_{m,k}^{o}(t)t_{m,k}^{o}+p_{m,k^{\prime}}^{o}(t)t_{m,k^{\prime} }^{c}\leq E_{\max,m}.\] (23d)  
The previously proposed iterative algorithm can still be applied here to solve \(\alpha_{i}(t)\), \(p_{i}^{l}(t)\), \(p_{i}^{o}(t)\) and \(p_{m,i}^{c}(t)\) with \(i=\{k,k^{\prime}\}\). Here the only difference lies in solving \(p_{m,k}^{c}(t)\) and \(p_{m,k^{\prime}}^{c}(t)\). The corresponding optimization problem can be formulated as  
min \[\max\left\{t_{m,k}^{o}+\frac{(1-\alpha_{k}(t))C_{k}(t)}{\frac{ \sqrt[3]{\frac{P_{m,k}^{c}(t)}{P_{m}}}}},\right.\] (24a) \[t_{m,k^{\prime}}^{o}+\frac{(1-\alpha_{k^{\prime}}(t))C_{k^{\prime }}(t)}{\frac{\sqrt[3]{\frac{P_{m,k^{\prime}}(t)}{P_{m}}}}{D_{m}}}\bigg{\}}\] s.t. \[\frac{(1-\alpha_{k}(t))C_{k}(t)}{\frac{\sqrt[3]{\frac{P_{m,k^{ \prime}}(t)}{P_{m}}}}{D_{m}}}+\frac{(1-\alpha_{k^{\prime}}(t))C_{k^{\prime}}( t)}{\frac{\sqrt[3]{\frac{P_{m,k^{\prime}}(t)}{P_{m}}}}{D_{m}}}\leq E_{\max,m}.\] (24b)  
It is worth noting that the optimal solution must activate the constraints and make the two terms within the objective function equal to each other. Therefore, the optimal \(p_{m,k}^{c}(t)\) and \(p_{m,k^{\prime}}^{c}(t)\) can be obtained by solving the following equations  
\[t_{m,k}^{o}+\frac{(1-\alpha_{k}(t))C_{k}(t)}{\frac{\sqrt[3]{\frac {P_{m,k}^{c}(t)}{P_{m}}}}{D_{m}}}=t_{m,k^{\prime}}^{o}+\frac{(1-\alpha_{k^{ \prime}}(t))C_{k^{\prime}}(t)}{\frac{\sqrt[3]{\frac{P_{m,k^{\prime}}(t)}{P_{m}} }}}\] (25) \[\frac{(1-\alpha_{k}(t))C_{k}  
The structure of the DQN-based offloading algorithm is illustrated in Figure 5, and the pseudocode is presented in Algorithm 1. ```
1: Initialize the experience replay buffer \(B\);
2: Initialize action-value function \(Q\) with random weights \(\theta\);
3: Initialize target action-value function \(Q^{\prime}\) with random weights \(\theta^{-}=\theta\);
4:for each episode \(n=1,2,\cdots,N\)do
5: Reset simulation parameters for the environment;
6: Randomly generate an initial state \(s_{1}\);
7:for each time slot \(t=1,2,\ldots,T\)do
8: Generate an action \(a_{t}=\mu(s_{t}|\theta^{\mu})+\
abla\mu\) to determine which MEC server to connect to;
9: Execute action \(a_{t}\) and solving corresponding optimization to obtain reward \(r_{t}\);
10: Receive reward \(r_{t}\) and observe the next state \(s_{t+1}\);
11: Store the tuple \((s_{t},a_{t},r_{t},s_{t+1})\) into \(B\);
12: Sample a random mini-batch of \(N\) transitions \((s_{t},a_{t},r_{t},s_{t+1})\) from \(B\);
13: Perform gradient descent and update Q-network;
14: Every C steps reset \(Q^{\prime}=Q\);
15:endfor
16:endfor
```  
**Algorithm 1** The DQN-based Offloading Algorithm