## 2 POMDP Modeling
### POMDP Specification  
For the multi-agent sequential decision problem in which the agents can only perceive partial information about the environment, the POMDP is often used for formal description. It is usually formally described by the tuple \(<N,S,A,T,R,O,\gamma>\). The information represented by each symbol is as follows:  
* \(N\): represents the set of all agents involved in the problem, and \(|N|\) is the number of agents. * \(S\): represents the state space of global states in the problem. \(s_{t}\in S\) represents the global state at time \(t\). It is a vector composed of the observation information \(o_{t}^{i}\) of each agent, \(s_{t}=[o_{t}^{1},o_{t}^{2},\ldots,o_{t}^{|N|}]\). * \(A\): represents the joint action space of agents in the problem, which is composed of the action spaces of each agent, \(A=A^{1}\times A^{2}\times\cdots\times A^{|N|}\). Considering that there are different demand levels for different materials, we use the action to represent the ratio of the production capacity and inventory level of the agent when considering the actions of the agents in the model. Therefore, for the action space \(A^{i}\) of agent \(i\), assuming \(M^{i}\) is the number of types of materials involved in the inventory of the agent, the action space \(A^{i}=[0,1]^{M^{i}}\). At time \(t\), the action of agent \(i\) is \(a_{t}^{i}\in A^{i}\). It seems logical to use continuous variables directly as the actions, but using such a continuous action space in actual training is inconvenient.On the one hand, it makes the action space size too large to exploration. On the other hand, due to the legality constraint of actions, the sampling of actions over the interval will be trimmed, causing the non-differentiable spike impact on the two endpoints of the feasible action domain. Therefore, the action space is discretized as \(\{0,0.01,0.015,0.02,0.04,0.05,0.1,0.2,0.4,1\}\). * \(T\): \(S\times A\times S\mapsto[0,1]\) represents the state transition of the global state \(S\) in the problem. * \(R\): represents the reward obtained by the agents in the environment. For agents, there are two types of reward information they can receive: individual reward and team reward. The individual reward is the reward obtained by the agent based on its local observation information, representing the cost of the agent itself. The team reward is the reward obtained by all agents based on the global state, representing the total cost of the supply chain. Minimizing the total cost of the supply chain is the common optimization target of all agents. The team reward is the sum of individual reward of all agents in the supply chain \(r_{t}=\sum_{i=1}^{|N|}r_{t}^{i}\). The individual reward of agent \(i\) is \(r_{t}^{i}=r_{t}^{stock,i}+r_{t}^{lac,i}+r_{t}^{overflow,i}\) composed of the following three parts:
* Inventory cost \(r_{t}^{stock,i}\) is the cost incurred to maintain inventory, including storage costs, storage management costs, etc. At time \(t\), the inventory ratio of mate
rial \(j\) in the inventory of agent \(i\) is \(inv_{t}^{i,j}\), and the inventory cost of the agent is \(r_{t}^{stock,i}=-\frac{\sum_{j=1}^{M^{i}}inv_{t}^{i,j}}{M^{i}}\). * Shortages cost \(r_{t}^{back,i}\) is the cost incurred due to the inability to meet the demand for materials that are due to insufficient inventory. At time \(t\), the proportion of material \(j\) in the unfulfilled due orders of agent \(i\) is \(lck_{t}^{i,j}\), and the stock shortage cost of the agent is \(r_{t}^{back,i}=-\frac{\sum_{j=1}^{M^{i}}lck_{t}^{i,j}}{M^{i}}\). * Overflow cost \(r_{t}^{overflow,i}\) is the additional storage cost incurred due to excess inventory. At time \(t\), the proportion of material \(j\) that exceeds the upper limit of agent \(i\)'s inventory capacity is \(\mathit{ovf}_{t}^{i,j}\), and the overflow cost of the agent is \(r_{t}^{overflow,i}=-\frac{\sum_{j=1}^{M^{i}}\mathit{ovf}_{t}^{i,j}}{M^{i}}\).