## 3 Preliminaries: Network Model and Problem Definition
### TOMAC-PPO Algorithm Process  
The PPO algorithm, proposed by Open AI and DeepMind, is widely regarded as one of the most successful DRL algorithms due to its excellent performance, high efficiency, and stable characteristics [21]. PPO is a strategy learning method, based on the Actor-Critic framework, which can be well adapted to the multi-agent system proposed in this paper. Therefore, this paper combines PPO with Transformer and applies it to the TOMDP model and multi-agent system built in this paper, ultimately forming the TOMAC-PPO algorithm  
Figure 7: Policy network structure. framework. During the training process of the TOMAC-PPO algorithm, constant \(\theta_{\text{now}}\) is used to represent the current parameters of the policy network. \(\theta\) indicates the parameters of the policy network during the next update, which is the optimization variable of the algorithm. The objective function in TOMAC-PPO theory is denoted as \(J(\theta)\). Due to the \(J(\theta)\) difficulty in obtaining the expression, when \(\theta\) is in the confidence domain \(\mathcal{N}(\theta_{\text{now}})\), an expression \(L(\theta)\) can be constructed that is close enough to approximate \(J(\theta)\) and easy to solve, and \(L(\theta)\) can be used instead of \(J(\theta)\) as the approximate objective function to solve [22]. To satisfy the confidence domain constraint \(\theta\in\mathcal{N}(\theta_{\text{now}})\), the divergence needs to be used to measure and limit the difference between the new strategy \(\pi(A|S;\theta)\) and the old strategy \(\pi(A|S;\theta_{\text{now}})\). The TOMAC-PPO algorithm proposes a scheme of approximate optimization objective pruning. The approximate objective function \(L(\theta)\) is clipped so that \(\theta\) does not exceed the confidence domain \(\mathcal{N}(\theta_{\text{now}})\) when the gradient ascent algorithm is run. In this scheme, the constraints in the objective function are removed, and the constraint function on \(\theta\) is retained, which makes the optimization problem easier to solve. Here, \(s\left(x_{t}^{i}\right)\), \(a\left(x_{t}^{i}\right)\) and \(r\left(x_{t}^{i}\right)\) will be simplified as \(s_{t}\), \(a_{t}\) and \(r_{t}\). Note that \(ratio_{t}(\theta)=\frac{\pi(a_{t}|s_{t};\theta)}{\pi(a_{t}|s_{t};\theta_{ \text{now}})}\); the derivation process in reference [23] yields the approximate objective function \(L^{\text{CLIP}}(\theta)\) after being trimmed by the approximate optimization target clipping method in the TOMAC-PPO algorithm. \[ \begin{split} L^{\text{CLIP}}(\theta)=\mathbb{E}_{t}\{& \min[ratio_{t}(\theta)\cdot Adv_{t},\ \text{clip}(ratio_{t}(\theta),1-\varsigma,1+\varsigma)\cdot Adv_{t}+c\\ &\cdot H(s_{t};\theta)]\}\end{split}  \tag{17}\]  
where \(\varsigma\) and \(c\) are hyperparameters. \(\text{clip}(ratio_{t}(\theta),1-\varsigma,1+\varsigma)\) means that the maximum value of \(ratio_{t}(\theta)\) is truncated in the interval \(\theta\in[1-\varsigma,1+\varsigma]\), so that the \(L^{\text{CLIP}}(\theta)\) function must have a global maximum value in the interval \([1-\varsigma,1+\varsigma]\) as shown in Figure 9. \(H(s_{t};\theta)\) is based on the entropy reward introduced in references [17; 22], which can encourage agents to fully explore more actions and avoid premature convergence of strategies to local optima. The advantage function \(Adv_{t}\) can be expressed as follows:  
\[Adv_{t}=Q_{\pi}(s_{t},a_{t})-V_{\pi}(s_{t}) \tag{18}\]  
It is hard to find the specific values of \(Q_{\pi}\) and \(V_{\pi}\) in actual training, so it is necessary to approximate \(Adv_{t}\). Firstly, TOMAC-PPO causes the agent to collect the T step trajectory \(\{s_{t},a_{t},r_{t}\}\) with the current strategy \(\pi(A|s_{t};\theta_{\text{now}})\) so that it can get a discount return \(u_{t}=r_{t}+\gamma\cdot r_{t+1}+\cdots+\gamma^{T-t}\cdot r_{T}\).