## 3 Preliminaries: Network Model and Problem Definition
### TOMAC-PPO Algorithm Process
Then, in (18), the \(Q_{\pi}(s_{t},a_{t})\) approximation is replaced by \(u_{t}\), and the \(V_{\pi}(s_{t})\) approximation is replaced by the value network output \(v(s_{t};\omega)\), so that the approximate advantage \(Adv_{t}^{\prime}\) can be obtained:  
\[Adv_{t}^{\prime}=r_{t}+\gamma\cdot r_{t+1}+\cdots+\gamma^{T-t}\cdot r_{T}-v(s_ {t};\omega) \tag{19}\]  
Figure 8: Value network structure. So far, the variables in the expression of the approximate objective function \(L^{\mathrm{CLIP}}(\mathbf{\theta})\) can be obtained, and the gradient ascent method can be used to continuously update the strategic network parameter \(\mathbf{\theta}\) of each agent to increase \(L^{\mathrm{CLIP}}(\mathbf{\theta})\). \[\mathbf{\theta}\leftarrow\mathbf{\theta}_{\mathrm{now}}+\beta\cdot\
abla_{\mathbf{\theta}} L^{\mathrm{CLIP}}(\mathbf{\theta}_{\mathrm{now}}) \tag{20}\]  
where \(\beta\) is the learning rate, which is the training value network, and the value network loss function is defined as follows:  
\[L^{V}(\mathbf{\omega})\triangleq-\sum_{t=1}^{T}\left(Adv_{t}^{\prime}\right)^{2} \tag{21}\]  
The larger the approximate advantage \(Adv_{t}^{\prime}\) in (21), the smaller the loss of the value network. The purpose of defining the loss function in this way is to encourage the agent to try to increase the dominance value and choose the action that increases the dominance as much as possible. The value network parameter \(\mathbf{\omega}\) can be updated by the gradient descent method to reduce the loss \(L^{V}(\mathbf{\omega})\):  
\[\mathbf{\omega}\leftarrow\mathbf{\omega}_{\mathrm{now}}-\alpha\cdot\
abla_{\mathbf{ \omega}}L^{V}(\mathbf{\omega}_{\mathrm{now}}) \tag{22}\]  
where \(\alpha\) is the learning rate. The target value network parameter \(\hat{\mathbf{\omega}}\) can be updated in a similar way as shown in (23). \[\hat{\mathbf{\omega}}\leftarrow\sigma\cdot\mathbf{\omega}+(1-\sigma)\cdot\hat{\mathbf{\omega}} \tag{23}\]  
where parameter \(\sigma\in(0,1)\) is the update ratio for the target network. The specific steps of the TOMAC-PPO algorithm are shown in Algorithm 1. ``` Input: Training rounds \(e^{max}\), pruning parameters \(\varsigma\), value network \(\alpha\) learning rate, strategy network learning rate \(\alpha\), discount rate \(\gamma\), target value network update ratio \(\sigma\). Output: Optimal task offloading and resource allocation strategy \(\pi(a|\mathbf{s};\mathbf{\theta})\). 1. Randomly initialize the parameter \(\mathbf{\theta}\) of each strategy network, as well as each value network parameter \(\mathbf{\omega}\), and the target value network parameter \(\hat{\mathbf{\omega}}\);
2. For \(episode=1,2,\dots,e^{max}\) do
3. For all agents \(j\), where \(\mathbf{1}\leq j\leq\mathbf{n}\) do in parallel
4. According to the current strategy \(\pi(A|s_{i};\mathbf{\theta}_{\mathrm{now}}^{i}),\ \mathrm{collect}\ T\ \mathrm{step}\ \mathrm{trajectory}\ \left\{s_{t},a_{t},r_{t}\right\}\);
5. According to (19), calculate the approximate advantage \(Adv_{t}^{\prime}\);
6. Update the policy network parameter \(\mathbf{\theta}^{j}\) of agent \(j\) according to (20);
7. According to (22) and (23), the value network parameter \(\mathbf{\omega}^{j}\) and the target value network parameter \(\hat{\mathbf{\omega}}^{j}\) of \(j\) are updated. End for End for

Figure 9: Clipped summarize objective schematic diagram. In the following, we give the complexity analysis of the TOMAC-PPO algorithm. The main time spent on this algorithm is in the second step, which is a training round loop with \(e^{max}\) training rounds and a time complexity of O (\(e^{max}\)). The third step is that the agent operates in parallel, with a time complexity of O (\(n\)). So, the overall time complexity is O (\(ne^{max}\)). ## 5 Experimental Results and Analysis

To evaluate the optimization effect of the proposed scheme on MEC network performance, we compare the following four offloading decision schemes with TOMAC-PPO. 1. TOMAC-A2C (Task-Oriented Multi-Agent Cooperative Advantage Actor-Critic).