## 1 Introduction
### Inventory Management Methods with Actor-Critic RL  
The Actor-Critic RL combines the advantages of both value-based and policy-based methods. It evaluates the policy function based on the value function and makes decisions relying on the policy function. Currently, most mainstream RL methods are based on this architecture. [PERSON] and [PERSON] (2019) used a shared policy in a supply chain collaboration environment to compare the performance of different single-agent RL algorithms, such as Deep Deterministic Policy Gradient (DDPG) (2018), Soft Actor-Critic (SAC) (2019), and Proximal Policy Optimization (PPO) (2019), and the implementation results showed that the PPO algorithm performed the best. In this study, all homogenous agents used the same policy for inventory management. However, in actual supply chains, different participants often face significantly different environmental states and cannot be simply regarded as homogeneous agents. [PERSON] et al. (2017) attempted to use RL methods to solve inventory management problems for multiple materials. However, their approach only applies to inventory management problems for a single node. It is unsuitable for the problem of multiple nodes and materials in the scale manufacturing industry. In research on inventory management in multi-node and multi-product mode supply chains, [PERSON] et al. (2018) used multi-agent reinforcement learning (MARL) to solve inventory management problems for multiple products and nodes. However, their environmental modeling is too simplistic, particularly in modeling the central warehouse. Choosing a method similar to the (r, S)-policy of completely replenishing the warehouse will result in high additional inventory costs. [PERSON] et al. (2019) used the RL-based method to solved the new product inventory management problem. [PERSON] et al. (2019) researched the waste reduction via RL in supply chain inventory management. [PERSON] et al. (2019) modeled the supply chain management environment in the aviation manufacturing industry and used RL to solve inventory management problems for multiple materials under a single node. However, their assumptions about material demand are too simplistic and do not consider the dependencies between different types of material demand in different production processes. While there have been some methods developed for managing inventory in simple supply chain environments, there are still several areas that need to be improved for large-scale manufacturing under the main manufacturer-supplier model in the field of supply chain inventory management:  
* Demand modeling is too simple and fails to reflect the correlation between materials and process flow and the dependency relationship between different process flows;
* Different participants play different roles in the supply-demand relationship and have different goals and expectations for inventory management. Single-agent RL algorithms or homogenous MARL algorithms have limited further efficiency improvement;
* As the number of supply chain participants and the types of materials increases, the state space and action space of the environment grow explosively, requiring more efficient exploration of the policy space;
* With the increase in the number of supply chain participants, the requirement for internal collaborative cooperation within the supply chain will also increase. More effective methods are needed to prevent coupling when supply chain participants update their policies. In order to solve the issues mentioned above, we proposed a supply chain inventory management method for civil aircraft manufacturing based on MARL. The main contributions of this paper can be summarized as follows:  
* We formalized the supply chain inventory management problem of main manufacturer-supplier as a partially observable Markov decision process model.