## 3 DQN-Based Offloading Design
### Reward Function
\[E_{k}\leq E_{\max,k},\forall k\in\mathcal{K} \tag{14b}\] \[E_{m}\leq E_{\max,m},\forall m\in\mathcal{M}, \tag{14c}\]  
where \(E_{\max,k}\) and \(E_{\max,m}\) denote the maximum available energy at intelligent terminal \(k\) and MEC server \(m\), respectively. Problem (14) can be rewritten as  
\[\min_{x_{k}(t)} \max\left\{\frac{\alpha_{k}(t)C_{k}(t)}{\sqrt[3]{\frac{p^{l}_{k} (t)}{l_{k}}}},\frac{(1-\alpha_{k}(t))C_{k}(t)}{\sqrt[3]{\frac{p^{l}_{mk}(t)}{ l_{m}}}}\right.\] \[\left.+\frac{(1-\alpha_{k}(t))C_{k}(t)}{B\log_{2}(1+\frac{p^{o}_{ k}(t)|n_{mk}(t)|^{2}}{N_{0}})}\right\}\] s.t. \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]0 \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]1 (15c) \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]2To solve problem (15), we first find that at optimal solution, constraint (15c) must be active, which can minimize the the objective value (15a). We thus have  
\[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]3  
which produces  
\[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]4  
Substituting (17) to problem (15), we have  
\[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]5 \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]6 s.t. \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]7 (18b) \[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]8  
It is noted that problem (18) is a non-convex optimization problem. We propose an alternating algorithm to solve \(p_{k}^{l}(t)\), \(p_{k}^{o}(t)\), and \(\alpha_{k}(t)\) in different subproblems separately to find an efficient solution. In the first subproblem, we solve \(p_{k}^{l}(t)\) for given \(\alpha_{k}(t)\) and \(p_{k}^{o}(t)\). To minimize the objective function, the optimal solution of \(p_{k}^{l}(t)\) should activate constraint (18b), that is, \(p_{k}^{l}(t)t_{k}^{l}=E_{\max,k}-p_{k}^{o}(t)t_{m,k}^{o}\), which implies  
\[t^{o}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{o}_{m,k}(t)}. \tag{10}\]9  
In the second subproblem, we solve \(p_{k}^{o}(t)\) with given \(p_{k}^{l}(t)\) and \(\alpha_{k}(t)\). The corresponding optimization problem is given by  
\[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]0 (20a) s.t. \[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]1  
Problem (19) is a convex optimization problem and can be efficiently solved, such as the interior point algorithm, etc. In the third problem, \(\alpha_{k}(t)\) is solved with given \(p_{k}^{l}(t)\) and \(p_{k}^{o}(t)\). The corresponding optimization problem is given by  
\[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]2 \[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]3 s.t. \[t^{c}_{m,k}=\frac{(1-\alpha_{k})C_{k}(t)}{r^{c}_{m,k}(t)}. \tag{11}\]4For the min-max problem (21), by denoting \(f_{1}(\alpha_{k}(t))=\frac{\alpha_{k}(t)C_{k}(t)}{\frac{\sqrt[3]{\frac{P_{m,k}^{c}( t)}{P_{m}}}}{D_{m}}}\), \(f_{2}(\alpha_{k}(t))=\frac{(1-\alpha_{k}(t))^{3/2}C_{k}(t)^{3/2}}{\frac{\sqrt[3]{ \frac{P_{m,k}^{c}(t)}{P_{m}}}}{D_{m}}}+\frac{(1-\alpha_{k}(t))C_{k}(t)}{D_{m}}\) and \(f(\alpha_{k}(t))=f_{1}(\alpha_{k}(t))+f_{2}(\alpha_{k}(t))\), it is known that the optimal \(\alpha_{k}(t)\), denoted by \(\alpha_{k}^{*}(t)\), occurs in the following three cases, that is, \(\alpha_{k}^{1}(t)=0\), \(\alpha_{k}^{2}(t)=1\), or \(f_{1}(\alpha_{k}^{3}(t))=f_{2}(\alpha_{k}^{3}(t))\). Note that the solution of the third case can be obtained by solving a cubic equation. The final solution is given as  
\[\alpha_{k}^{*}(t)=\left\{ \begin{array}{ll}\alpha_{k}^{1}(t)&\text{if }f(\alpha_{k}^{1}(t))\leq\{f(\alpha_{k}^{2}(t)),f(\alpha_{k}^{3}(t))\}\\ \alpha_{k}^{2}(t)&\text{if }f(\alpha_{k}^{2}(t))\leq\{f(\alpha_{k}^{1}(t)),f( \alpha_{k}^{3}(t))\}\\ \alpha_{k}^{3}(t)&\text{if }f(\alpha_{k}^{2}(t))\leq\{f(\alpha_{k}^{1}(t)),f( \alpha_{k}^{2}(t))\}\end{array}