## 3 Preliminaries: Network Model and Problem Definition
### TOMAC-PPO Algorithm Process
A2C algorithm is one of the classic strategy learning algorithms in the RL field. 2. TO-A3C (Task-Oriented Asynchronous Advantage Actor-Critic). TO-A3C belongs to the parallel RL method and does not use multi-agent systems. The A3C algorithm improves its performance by establishing multiple independent single agent A2C training environments, enabling them to train in parallel [24]. 3. CCP (Cloud Computing Priority). CCP adopts the principle of \"deliver tasks to upper level processing as much as possible\", and prioritizes offloading all tasks to the cloud for processing. 4. LC (Local Computing). After the task is generated, skip the information reporting process and directly calculate locally by the user. We build the algorithm environment for active queue management, offloading scheduling rules, and multi-agent systems described in this paper to run the TOMAC-PPO algorithm. We give convergence analysis, and test average cost, average delay, average energy consumption, and drop rate. ### Experimental Environment and Parameter Settings

The experimental environment is implemented using Python language, and the real dataset used includes the longitude and latitude of mobile users and the maximum CPU frequency as user device data. This paper stipulates that the probabilities of users generating high-priority tasks, critical tasks, low-priority tasks, and regular tasks are [0.2, 0.2, 0.2, 0.4], respectively. The specific parameters are shown in Table 1.

### Convergence Analysis

For 50 users and a probability of failure \(p_{en}=p_{fiber}=0.1\%\), we give convergence results for the TOMAC-A2C algorithm and the TOMAC-PPO algorithm. The cumulative reward convergence of TOMAC-A2C and TOMAC-PPO during training is shown in \[Adv_{t}=Q_{\pi}(s_{t},a_{t})-V_{\pi}(s_{t}) \tag{18}\]0 We can see that the moving average reward curves of both algorithms gradually converge with the increase in training episodes, proving the effectiveness of TOMAC-A2C and TOMAC-PPO algorithms. TOMAC-PPO not only has a significant advantage in convergence speed compared to TOMAC-A2C but also improves the final convergence value by about 23.9% compared to MAC-A2C, demonstrating its outstanding performance advantage. From the stability of the convergence curve, although the final convergence value of TOMAC-PPO is significantly higher than TOMAC-A2C, its vibration amplitude is also obviously larger. This may be a phenomenon caused by the introduction of entropy in the approximate objective function \(L^{\text{CLIP}}(\mathbf{\theta})\) of TOMAC-PPO, which leads to a higher desire for agents to explore randomly. ### Optimization Performance Evaluation

To evaluate the performance of various offloading schemes under different network load levels, this experiment tested the variation of average task cost with the number of users under the \(p_{en}=p_{fiber}=0.1\%\) setting of failure probability. As shown in Figure 11, the TOMAC-PPO method always maintains the lowest task overhead under various load conditions, and its performance advantage continues to expand with the increase in the number of users. When the number of users is 50, TOMAC-PPO reduces the average cost by 19.4% to 66.6% compared to other solutions. To test the response capability of various offloading schemes to line faults, this experiment tested the average task cost with the variation of the failure probability of edge servers and lines under the setting of a large number of users (50). As shown in Figure 12, when the failure rate is less than 70%, the task overhead corresponding to TOMAC-PPO and TOMAC-A2C methods is significantly lower than other schemes. When the failure rate reaches over 70% and continues to increase, the task