## 3 Proposed Supply Chain Inventory Management Method
### Dual Policy Model with Information Transmission
\begin{split} J(\theta^{i})=&\mathbb{E}[\mathbb{I}_ {o^{i}_{t}\leq 1}max(J^{PDO}(\theta^{i}),[PERSON]^{IDV-TEAM}(\theta^{i}))\\ +&\mathbb{I}_{o^{i}_{t}>1}min(J^{PDO}(\theta^{i}),J ^{IDV-TEAM}(\theta^{i}))\\ -&\alpha KL(\hat{\pi}^{i},\pi^{i})]\end{split}  \tag{6}\]  
As the data used for learning is obtained by individual policy \(\pi^{i}\), the importance sampling is used to update the team policy \(\hat{\pi}^{i}\):  
\[ \begin{split}\delta^{i}_{t}(\hat{\theta}^{i})=&\frac{ \hat{\pi}_{\theta^{i}}(a^{i}_{t}|o^{i}_{t})}{\pi_{\theta^{i}_{old}}(a^{i}_{t}|o^ {i}_{t})}\\ \hat{J}(\hat{\theta}^{i})=&\mathbb{E}[clip(\phi^{i}_ {t}(\hat{\theta}^{i}),1\pm\zeta)\hat{A}_{t}-\beta KL(\pi^{i},\hat{\pi}^{i})] \end{split}  \tag{7}\]  
where \(\zeta\) is a hyperparameter that limits the update step size, \(\beta\) is a hyperparameter that gradually decreases with training, indicating that the influence of individual policy on the latter stages of training decreases, and \(\hat{A}_{t}\) is the estimated advantage function obtained through the GAE algorithm. In the MAPPO algorithm, the joint policy is updated by using the local information of each agent to update the policy of each agent in the direction of each agent's individual reward. In the cooperative problem, it is necessary for the agents to have a certain level of communication and coordination mechanism in order to prevent coupling of the joint policy of the participating parties during the update. So we introduce the information transmission mechanism in our model to solve this problem. If \(\{i_{1},i_{2},...,i_{m}\}\) is a subset of the set of all agents \(N\), where \(i_{k}\) represents the \(k\)th element in the subset, and the complement of the subset is denoted by \(-i_{1:m}\), then the state-action value function of the subset is:  
\[Q_{\pi}^{i_{1:m}}(s,\mathbf{a}^{i_{1:m}})\triangleq\mathbb{E}_{\mathbf{a}^{-i_{1}:m \sim\pi^{-1}:m}}[Q_{\pi}(s,\mathbf{a}^{i_{1:m}},\mathbf{a}^{-i_{1:m}})] \tag{8}\]  
For disjoint subsets \(i_{1:m}\) and \(j_{1:k}\), the multi-agent advantage function of subset \(i_{1:m}\) can be calculated from the state-action value function as:  
\[A_{\pi}^{i_{1:m}}(s,\mathbf{a}^{i_{1:k}},\mathbf{a}^{i_{1:m}})\triangleq Q_{\pi}^{j_{ 1:k}:i_{1:m}}(s,\mathbf{a}^{j_{1:k}},\mathbf{a}^{i_{1:m}})-Q_{\pi}^{j_{1:k}}(s,\mathbf{a}^ {j_{1:k}}) \tag{9}\]  
Therefore, if we specify an update order from \(i_{1}\) to \(i_{m}\) for the agents being updated, we can obtain:  
\[A_{\pi}^{i_{1:m}}(s,\mathbf{a}^{i_{1:m}})\triangleq\sum_{j=1}^{m}A_{\pi}^{i_{j}}(s,\mathbf{a}^{i_{1:j-1}},\mathbf{a}^{j}) \tag{10}\]  
The Equation (10) does not require any additional assumptions beyond specifying an update order for the agent's update, and subsequent agents can further update based on the understanding of the influence of the previous agents' update on the joint policy, ensuring monotonic update of the joint policy in a single update. Therefore, we maintain the information transmission factors of the individual joint policy and the team joint policy based on this.The factors record the update status of the joint policy to achieve monotonic update of the joint policy. In addition, we randomly reassign the update order of the agents at each update to eliminate the influence of a specific update order on policy update.