## 4 Approach
### Proximal Policy Optimization  
Trinity collaboratively optimizes the partition and schedule network using PPO. The goal of PPO is to maximize the expectation of the reward (i.e., MDPE model) and update the policy network parameters. Therefore, the objective function can be expressed as below:  
\[J(\pi)=J(\pi_{g},\pi_{s})=E_{p(g,g;\theta)}(R) \tag{10}\]  
Convert expectations to probability distributions, which can also be written as  
\[J(\pi;\theta)=\sum_{\pi_{g},\pi_{s}}\left[p(g,s;\theta)R\right] \tag{11}\]  
The essence of the optimization algorithm is to control the parameters of the policy network and change the probability distribution of the policy to maximize the expected reward. \(p(g,s;\theta)\) represents the probability distribution of the distributed parallel strategy under the given network parameter conditions. \(R\) is the reward, which is calculated by the partition strategy \(g\) and the schedule strategy \(s\) according to MDPE. The expectation in Formula (10) is approximated by Monte Carlo sampling and iterative optimization using gradient ascent. However, when the parameters are optimized using the gradient, the probability distribution \(p(g,s;\theta)\) will change. Even small parameter changes can cause drastic changes in \(p(g,s;\theta)\), requiring resampling after parameter update. The violent jittering of the probability distribution is also not conducive to algorithm convergence. Therefore, based on importance sampling, we adopted PPO and rewrote the objective function as the following formula:  
\[\max_{p}\mathbb{E}_{\pi_{old}\sim p_{old}}\left[\frac{p(g,s;\theta)}{p_{old}( g,s;\theta_{old})}(R-b)\right]. \tag{12}\]  
\[s.t.\ KL[p_{old}\ (g,s;\theta_{old}),p(g,s;\theta)]\leq\epsilon. \tag{13}\]  
where \(\theta_{old}\) is the vector of policy parameters before the update. We take the probability distribution \(p_{old}(g,s;\theta_{old})\) of the old policy as the proposal distribution and still sample from the old probability distribution. The Formula (13) maintains the difference between \(p_{old}\) and \(p\) within \(e\); \(b\) is the mean moving baseline. The optimization problem with constraints can be solved by the conjugate gradient algorithm, but the cost is high. Let \(r_{t}(\theta)\) donate the probability ratio \(r_{t}(\theta)=\frac{p(g,\varepsilon;\theta)}{p_{old}(g,\varepsilon;\theta_{old})}\). We modify the objective to Formula (14) to penalize \(r_{t}(\theta)\) for being far from \(\varepsilon\):  
\[J_{\theta}=\max\mathbb{E}_{\pi_{old}}[r(g,s;\theta)(R-b),clip(r(g,s;\theta),1- \epsilon,1+\epsilon)(R-b)]. \tag{14}\]  
where \(\epsilon\) is a hyperparameter that controls the difference between the old and new distributions, and \(clip\) is the truncation function used to truncate the maximum and minimum values of the objective function to ensure that the control always remains between \([1-\epsilon,1+\epsilon]\). Finally, the objective function is maximized by the stochastic gradient ascent. The complete algorithm execution is shown in Algorithm 1. ```
Data: devices: \(\mathcal{D}=\{d_{1},d_{2},\ldots,d_{n}\}\), policy network parameters \(\{\theta_{g},\theta_{s}\}\) Result: optimal parallel strategy: \(\pi^{*}\), policy network parameters: \(\left\{\theta^{*}_{g},\theta^{*}_{s}\right\}\) Initialization min \(\rightarrow\infty\) and \(R=0\); for\(i=1,2,3,\ldots,N\)do \(\pi_{g}\rightarrow\{g_{1},g_{2},\ldots,g_{m}\}\) ; for\(g_{i}\) in \(\{g_{1},g_{2},\ldots,g_{m}\}\)do \(group.append(g_{i})\) ;  end for \(\pi_{s}(group)\rightarrow\{(g_{1},d_{1}),(g_{2},d_{2}),\ldots,(g_{m},d_{n})\}\) ; apply \(\pi_{s}\) to networks and obtain the reward \(R\) ; if\(R<\min\)then \(\pi^{*}=(\pi_{g},\pi_{s})\) ; \(\min=R\) ; \(f_{\theta}=\mathbb{E}_{\pi_{old}}[r(g,s;\theta)(R-b),clip(r(g,s;\theta),1- \epsilon,1+\epsilon)(R-b)]\) ; \(f_{\theta}=f_{\theta}+\
abla J_{\theta}\) according to Formula (14) ;  end for return \(\pi^{*}\) and \(R\)
```  
**Algorithm 1**Parallel policy automatic search algorithm based on PPO. In this paper, Adam [37] is used to complete the gradient descent.