## 3 Proposed Supply Chain Inventory Management Method
### Dual Policy Model with Information Transmission  
In the dual policy mechanism of the method, each agent \(i\) maintains a pair of individual policy \(\pi^{i}\) and team policy \(\hat{\pi}^{i}\) separately. The individual policy \(\pi^{i}\) is used for direct interaction with the environment, and the team policy \(\hat{\pi}^{i}\) is learned from the information obtained from the information obtained from the environment. According to the PPO algorithm [26], the update of the individual policy \(\pi^{i}\) is subject to update step size constraints. If \(\eta_{t}^{i}(\theta^{i})\) is the update step size, the individual policy \(\pi^{i}\) updates with \(J^{PPO}(\theta^{i})\) as the update target according to Equation (1):  
\[\eta_{t}^{i}(\theta^{i})= \frac{\pi_{\theta^{i}}(a_{t}^{i}|o_{t}^{i})}{\pi_{\theta^{i}_{ ad}}(a_{t}^{i}|o_{t}^{i})} \tag{1}\] \[J^{PPO}(\theta^{i})=\mathbb{E}[clip(\eta_{t}^{i}(\theta^{i}),1 \pm\epsilon)A_{t}^{i}]\]where \(\epsilon\) is a hyperparameter that limits the update step size, and \(A^{i}_{t}\) is the advantage function of agent \(i\) estimated by the GAE algorithm [34] as shown in Equation (2):  
\[ \begin{split} A^{i}_{t}=\sum_{i=0}^{h}(\gamma\lambda)^{i}\delta^{i }_{t+1}\\ \delta^{i}_{t}=r^{i}_{t}+\gamma V_{\phi^{i}}(s_{t+1})-V_{\phi^{i} }(s_{t})\end{split}  \tag{2}\]  
where \(\delta^{i}_{t}\) is the TD error of agent \(i\) at time \(t\) for the trajectory of length \(h\). In our model, the individual reward \(r^{i}_{t}\) and the team reward \(r_{t}\) are made use of by each agent to learns both the individual policy \(\pi^{i}\) and the team policy \(\pi^{i}\). To ensure that the individual policy \(\pi^{i}\) does not deviate too much from the goal of maximizing team profits during the exploration process, we define the similarity between the individual and team policies as \(\sigma^{i}_{t}(\theta^{i})\), at observation \(o^{i}_{t}\) and action \(a^{i}_{t}\). The policy update also subjects to the constraint \(J^{IDV-TEAM}(\theta^{i})\) as Equation (3):  
\[ \begin{split}\sigma^{i}_{t}(\theta^{i})=&\frac{ \pi_{\theta^{i}}(a^{i}_{t}|o^{i}_{t})}{\pi_{\theta^{i}}(a^{i}_{t}|o^{i}_{t})}\\ J^{IDV-TEAM}(\theta^{i})=&\mathbb{E}[clip(\sigma^{i }_{t}(\theta^{i}),1\pm\xi)A^{i}_{t}]\end{split}  \tag{3}\]  
where \(\xi\) is a hyperparameter that limits the update step size. Next, we need to combine the information got from Equations (1) and (3) for the update of individual policy. Since the individual policy \(\pi^{i}\) is mainly used to assist exploration during the early stages of training, it should still be updated with the goal of maximizing group interests. The data used for its own training does not include information on team returns \(r_{t}\), so it should be made to close to the direction of the team policy \(\hat{\pi}^{i}\) during training. We can get following conclusion from the definition of the similarity \(\sigma^{i}_{t}(\theta^{i})\) between the individual and team policies. When \(\sigma^{i}_{t}(\theta^{i})\leq 1\) and \(A^{i}_{t}>0\), it means that under the observation information \(o^{i}_{t}\), the individual policy \(\pi^{i}(a^{i}_{t}|o^{i}_{t})\) is consistent with the team policy \(\hat{\pi}^{i}(a^{i}_{t}|o^{i}_{t})\), so when \(\sigma^{i}_{t}\leq 1\), it should be ensured that:  
\[J(\theta^{i})=\mathbb{E}[\mathbb{I}_{o^{i}_{t}\leq 1}max(J^{PDO}(\theta^{i}),J ^{IDV-TEAM}(\theta^{i}))] \tag{4}\]  
Similarly, when \(\sigma^{i}_{t}>1\) it should be ensured that:  
\[J(\theta^{i})=\mathbb{E}[\mathbb{I}_{o^{i}_{t}>1}min(J^{PDO}(\theta^{i}),[PERSON]^{ IDV-TEAM}(\theta^{i}))] \tag{5}\]  
For the individual policy to learn the team policy information appropriately, we use the KL divergence with a hyperparameter \(\alpha\) that gradually increases as the training progresses as a constraint. Therefore, the complete objective function for the individual policy \(\pi^{i}\) is as follows:  
\[