## 3 Proposed Supply Chain Inventory Management Method
### Method Overview  
There are few common MARL environments for inventory management problems based on the main manufacturer-supplier model in the supply chain. Therefore, we first formally describe the inventory management problem under this model to construct a RL environment for training. As different participants in the supply chain play different roles and require different types and quantities of materials for inventory management, some differences exist in inventory management methods and goals. Suppose a single-agent RL algorithm [18] or homogeneous MARL algorithm [19] is used to address this problem. In that case, the model's efficiency will be limited to some extent. Therefore, it is necessary to distinguish between the main manufacturer and the supplier when modeling and use heterogeneous agents for modeling to be as close as possible to the actual production environment. As shown in the Figure 3, the policy network and value network of the agent take the data obtained from the agent's interaction with the environment as input. The input data first undergoes feature extraction using a Multi-Layer Perceptron (MLP), and the extracted features are then inputted into a Recurrent Neural Network (RNN). The output of the RNN is then fed into another MLP to obtain the final actions from the policy network or the final value from the value network. The generated actions and values are then saved in the data buffer. After collecting data for the entire trajectory through interactions, the policy network and value network of the agent will be updated with the data stored in the data buffer. The specific details and configurations related to the network structure will be elaborated on in the Section 4. As mentioned earlier, to be close to the real production environment, we made certain assumptions about the process flow and types of materials based on the actual production process. Due to a large number of materials and participants in the supply chain in the environment, on the one hand, it will result in a large state and action space of the environment, which is not conducive to policy exploration. On the other hand, when  
Figure 3: The structure of the MARL method based on the dual policy model with the information transmission mechanism. different agents update their policies, there will be significant coupling effects on the joint policy of the supply chain, and the improvement of the joint policy may not be apparent. In RL problems with multiple agents cooperating in tasks such as supply chain inventory management, the agents in the environment explore and update their policies to maximize the team reward. In actual training, on the one hand, the state space and action space of the environment explode with the increase in supply chain participants and types of materials, leading to low efficiency in policy exploration. On the other hand, as each supply chain participant shares the same team reward, they lack sufficient information to judge the impact of their policy updates on the joint policy of the supply chain [32, 33]. Therefore, we introduce a dual-policy mechanism consisting of independent and team policies that maximize individual and team rewards, respectively. By comparing the two policies, supply chain participants can understand the impact of their policy updates on the joint policy of the team. In the initial training stage, independent policies are used for exploration as much as possible. After the policies are relatively stable, the agents gradually switch to relying on team policies for decision-making using hyperparameters, which can significantly improve the efficiency of the agents' exploration. Furthermore, existing methods lack information transmission between the main manufacturer and the supplier when updating their policies. It will affect the exploration of the global optimal inventory management joint policy for the supply chain. The supply chain participants only use their local information to update their policies, ignoring the influence of others participants' policies.