## 3 Proposed DRL-Based Optimization Algorithm
Therefore, if \(\tau_{k,m,t}^{\min}\leq\tau_{k,m,t}^{\max}\), solving the problem \(\mathrm{P}_{4,km}\) is equivalent to solving the problem \(\mathrm{P}_{6,km}\), which has one optimization variable \(\tau_{m,k,t}\). If \(\tau_{k,m,t}^{\min}>\tau_{k,m,t}^{\max}\), the problem \(\mathrm{P}_{6,km}\) is unsolvable. **Theorem 2**. : _The optimization problem \(\mathrm{P}_{6,km}\) is convex._  
Proof. : See Appendix A. Based on the convexity illustrated in Theorem 2, we can adopt the bisection method to solve the problem \(\mathrm{P}_{6,km}\). The bisection based optimization algorithm for solving problem \(\mathrm{P}_{6,km}\) is summarized in Algorithm 1, where \(g(\tau_{k,m,t})\) denote the objective function of \(\mathrm{P}_{6,km}\). **(4) Policy \(\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]\)**;** The policy \(\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]\) denotes the mapping from the state \(\mathbf{S}(t)\) to the action \(\mathbf{A}(t)\) of the MEC system, i.e., \(\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]:\mathbf{S}(t)\to\mathbf{A}(t)\), where \(\mathbf{\theta}\) is the parameter of the policy. The parameter of the policy \(\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]\) is obtained through gradient based method. The performance measure \(Y(\mathbf{\theta})\) of the PPGA is defined as [28]  
\[+\omega_{1}\bigg{[}\psi_{1}\sum_{m=0}^{M}\sum_{k=1}^{K}\mu_{k,m,t }\left(E_{m,k}^{0}[t]+E_{m,k}^{c}[t]\right)+\psi_{2}\sum_{m=0}^{M}\sum_{k=1}^{ K}\mu_{k,m,t}\Gamma_{k,t}l_{k,t}(\beta_{k,t})\bigg{]}\Bigg{\}} \tag{6a}\]3  
where \(V_{\pi_{\mathbf{\theta}}}[\mathbf{S}(0)]\) is the value function for policy \(\pi_{\mathbf{\theta}}\) starting from initial state \(\mathbf{S}(0)\) and \(\mathbf{\theta}\) is the parameter of the policy. An analytic expression for the gradient of \(Y(\mathbf{\theta})\) is provided by policy gradient theorem [28], which is given as  
\[+\omega_{1}\bigg{[}\psi_{1}\sum_{m=0}^{M}\sum_{k=1}^{K}\mu_{k,m,t }\left(E_{m,k}^{0}[t]+E_{m,k}^{c}[t]\right)+\psi_{2}\sum_{m=0}^{M}\sum_{k=1}^{ K}\mu_{k,m,t}\Gamma_{k,t}l_{k,t}(\beta_{k,t})\bigg{]}\Bigg{\}} \tag{6a}\]4  
where \(\mu[\mathbf{S}(t)]\) is the on-policy distribution over states, \(q_{\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]}[\mathbf{S}(t),\mathbf{A} (t)]\) is the value of taking action \(\mathbf{A}(t)\) in state \(\mathbf{S}(t)\) under policy \(\pi_{\mathbf{\theta}}[\mathbf{A}(t)|\mathbf{S}(t)\,]\). ```
1:Initialization:
2: The bisection algorithm iteration index \(i=1\), maximum number of iterations \(I_{\text{max}}\), \(\left[\pi^{\text{min}}_{k,m,t}\,\tau^{\text{max}}_{k,m,t}\right]\), \(\forall k\in\mathcal{K},m\in\mathcal{M}\), the tolerance errors \(\xi\). 3:for:\(i=1:I_{\text{max}}\)
4: Update \(c=\frac{\pi^{\text{min}}_{k,m,t}+\pi^{\text{max}}_{k,m,t}}{2}\). 5:if\(g^{\prime}(c)=0\)then
6: The optimal value of \(\tau_{k,m,t}\) is \(\tau^{\text{opt}}_{k,m,t}=c\);
7: break;
8:endif
9:if\(g^{\prime}\Big{(}\tau^{\text{min}}_{k,m,t}\,\Big{)}g^{\prime}(c)<0\)then
10: Update \(\tau^{\text{max}}_{k,m,t}=c\);
11:endif
12:if\(g^{\prime}(c)g^{\prime}\Big{(}\tau^{\text{max}}_{k,m,t}\,\Big{)}<0\)then
13: Update \(\tau^{\text{min}}_{k,m,t}=c\);
14:endif
15:if\(\left|\tau^{\text{max}}_{k,m,t}-\tau^{\text{min}}_{k,m,t}\right|<\xi\) or \(i==I_{\text{max}}\)then
16: The optimal value of \(\tau_{k,m,t}\) is \(\tau^{\text{opt}}_{k,m,t}=c\);
17:endif
18:endfor
```  
**Algorithm 1**A Bisection Algorithm for Solving \(\mathrm{P}_{6,km}\)  
An action-independent baseline \(b[\mathbf{S}(t)]\) is always introduced to decrease the variance in the training process.