## 3 Proposed DRL-Based Optimization Algorithm  
In order to address the above issue, we propose a DRL-based optimization algorithm in this section. Specifically, we utilize the importance sampling based parameterized policy gradient approach (PPGA) DRL algorithm [28]. In order to apply the DRL-based algorithm, we first give the system state, action, reward, and the policy of the MEC system as follows:  
**(1) System state \(\mathbf{S}(t)\):** The system state at time-slot \(t\) is characterized by the channel power gain, the size (in bits) of computation task data, and the corresponding task complexity, i.e., \(\mathbf{S}(t)=\big{\{}(h_{k,m,t},l_{k,t},c_{k,t}),k\in\mathcal{K},m\in \mathcal{M},t\in\mathcal{T}\big{\}}\). **(2) Action \(\mathbf{A}(t)\):**\(\mathbf{A}(t)\) is the set of the offloading decisions of the IoT devices, i.e., \(\mathbf{A}(t)=\left\{\mu_{k,m,t},k\in\mathcal{K},m\in\mathcal{M},t\in\mathcal{T}\right\}\). **(3) Reward \(R(t)\):** After executing an action \(\mathbf{A}(t)\) under system state \(\mathbf{S}(t)\) at time-slot \(t\), the VSO will receive a reward \(R(t)\). The reward of a DRL model is direct related with the optimization objective of the system. Therefore, the reward of our DRL model is determined by the objective function value of the problem \(\mathrm{P}_{1}\) at time-slot \(t\). With given \(\left\{\mu_{k,m,t},k\in\mathcal{K},m\in\mathcal{M},t\in\mathcal{T}\right\}\) (\(\mathbf{A}(t)\)), the optimization problem for reward \(R(t)\) are given as  
\[\mathrm{P}_{2}:-R(t)=\] \[\left\{\eta_{k,m,t}\right\},\left\{\eta_{k,m,t}\right\},\left\{ \eta_{k,m,t}\right\}\gamma^{t-1}\Bigg{\{}\omega_{0}\psi_{0}\sum_{k=1}^{K}\mu_ {k,0,t}\] \[+\omega_{1}\bigg{[}\psi_{1}\sum_{m=0}^{M}\sum_{k=1}^{K}\mu_{k,m,t }\left(E_{m,k}^{0}[t]+E_{m,k}^{c}[t]\right)+\psi_{2}\sum_{m=0}^{M}\sum_{k=1}^{ K}\mu_{k,m,t}\Gamma_{k,t}l_{k,t}(\beta_{k,t})\bigg{]}\Bigg{\}} \tag{6a}\] \[s.t.\ \ (4\mathrm{b})-(4\mathrm{h}). \tag{6b}\]  
The standard form of the DRL-based optimization problem is to maximize the accumulated reward, hence, we add a minus sign before the \(R(t)\). Obviously, when \(\left\{\mu_{k,m,t},k\in\mathcal{K},m\in\mathcal{M},t\in\mathcal{T}\right\}\) are determined, the computing pressure of the PMS is determined too. Therefore, solving the problem \(\mathrm{P}_{2}\) is equal to solve the problem \(\mathrm{P}_{3}\) below  
\[\mathrm{P}_{3}: \min_{\{\eta_{k,m}\},\},\left\{\eta_{k,m,t}\right\}\] \[\omega_{1}\bigg{[}\psi_{1}\sum_{m=0}^{M}\sum_{k=1}^{K}\mu_{k,m,t} \left(E_{m,k}^{0}[t]+E_{m,k}^{c}[t]\right)+\psi_{2}\sum_{m=0}^{M}\sum_{k=1}^{ K}\mu_{k,m,t}\Gamma_{k,t}l_{k,t}(\beta_{k,t}) \tag{7a}\] \[s.t.\ \ (4\mathrm{c})-(4\mathrm{g}). \tag{7b}\]  
Based on the primal decomposition theory [29], the problem \(\mathrm{P}_{3}\) can be decomposed into \(K\) sub-optimization problems. Specifically, for the IoT device \(k\) and the MEC server \(m\), \(k,m\in\left\{\,k,m|\mu_{k,m,t}=1\right\}\), the corresponding sub-optimization problem can be expressed as  
\[\mathrm{P}_{3,km}:\min_{\left\{\eta_{k,m,t}\right\},\left\{\eta_{ k,m,t}\right\},\left\{p_{k,m,t}\right\}}\psi_{1}\left[E_{m,k}^{0}[t]+E_{m,k}^{c}[t] \right]+\psi_{2}\Gamma_{k,t}l_{k,t}(\beta_{k,t}) \tag{8a}\] \[s.t.\ \ (4\mathrm{c})-(4\mathrm{g}). \tag{8b}\]  
If the problem \(\mathrm{P}_{3,km}\) is solvable, i.e., there exist feasible solutions of \(\eta_{k,m,t}\), \(f_{m,k,t}\), and \(p_{k,m,t}\) to meet the constraints (4c)-(4g). The optimal value of \(\mathrm{P}_{3,km}\) is equal to the optimal objective value of \(\psi_{1}\left[E_{m,k}^{0}[t]+E_{m,k}^{c}[t]\right]\). On the other hand, if there exist no feasible solutions of \(\tau_{k,m,t}\), \(f_{m,k,t}\), and \(p_{k,m,t}\), we let the objective value of \(\mathrm{P}_{3,km}\) be equal to \(\Gamma_{k,t}\). It is worth noting that, if not all the IoT devices' tasks can be completed, our work can not be applied to minimize the sum of energy consumption of the system.