Multi-Agent Reinforcement Learning with Shared Resource in Inventory Management

1 INTRODUCTION . Inventory management ( IM ) problem has long been one of the most important applications in the supply-chain industry ( Nahmias & Smith , 1993 ) . Its main purpose is to maintain a balance between the supply and demand of stock keeping units ( SKUs ) in a supply chain by optimizing replenishment decisions of each SKU . Besides leading to profit increment and operational cost reduction , efficient IM can even give rise to better services to customers . However , it is quite a challenging task in practice , especially when there are lots of SKUs involved in the supply-chain . Particularly , while all SKUs should cooperate with each other to achieve high profits , they also need to compete for shared resources e.g. , warehouse spaces , budget etc . Such co-existence of cooperation and competition renders IM a complicated game that is hard to address . Traditional methods usually reduce IM problems to solving dynamic programming problems . However , these approaches often rely on some unrealistic assumptions such as i.i.d . demand , deterministic leading time , etc . Moreover , as the state space grows exponentially along with some key factors like leading time and number of SKUs ( Gijsbrechts et al. , 2019 ) , the corresponding dynamic programming problems become intractable due to the curse of dimensionality . Because of these limitations , many approaches based on approximate dynamic programming are proposed to solve IM problems in different settings ( Halman et al. , 2009 ; Fang et al. , 2013 ; Chen & Yang , 2019 ) . While these approaches perform well in certain scenarios , they heavily rely on problem-specific expertise or assumptions e.g. , the zero or one period leading time assumption in ( Halman et al. , 2009 ) , hence can hardly generalize to other settings . In contrast , reinforcement learning ( RL ) based methods , with fast inference speed , can generalize to various scenarios in a data-driven manner . However , it is usually too costly to train a global policy that can make decisions for all SKUs , since the training efficiency can be notably curtailed because of the large global state and action space ( Jiang & Agarwal , 2018 ) ) . To further address the training efficiency issue , it is natural to adopt the multi-agent reinforcement learning ( MARL ) paradigm , where each SKU is regarded as an agent whose state and action spaces are localized and only contain information relevant to itself . There are currently two popular paradigms to train MARL in the literature : independent learning ( Tan , 1993 ) and joint action learning ( Lowe et al. , 2017 ) . Despite of their success in many scenarios , these two MARL paradigms also exhibit certain weaknesses that restrain their effectiveness in solving IM problems . On one hand , if applying independent learning , policy training of one agent simply treats all other agents as parts of the stochastic environment , hence is hard to converge due to non-stationary of the environment . On the other hand , joint action learning usually learns a centralized critic conditioned on the joint action and state spaces of all agents , which can easily become intractable with increasing number of SKUs . Furthermore , it could be quite time-consuming to sample data from joint simulator for a great number of SKUs , since it usually involves much computation on many internal variables caused by complex agent interactions . To address these challenges , we took a closer look at the IM problem and find that there exists a special structure that can be leveraged to design more effective MARL paradigm . Particularly , each agent in IM only interacts with others through the shared resource , e.g. , warehouse spaces . We introduce an auxiliary variable to represent the whole inventory level , implying the available shared resource , for all SKUs , and refer to this variable as context . From the MARL perspective , one agent can be influenced by other agents only through such context . The context dynamics actually reflect the collective behaviors of all agents . And , conditioned on context dynamics , one agent ’ s state transition and reward function are independent of other agents . In this way , leveraging context as an additional input for each agent ’ s policy/value functions enables us to both avoid the non-stationary problem caused by independent learning and mitigate the intractable centralized critics learning caused by exploding state and action spaces . Moreover , introducing context dynamics inspires us to build a local simulator for each agent in order to facilitate more efficient policy learning for each agent . Based on this structure with context dynamics , we propose a shared-resource stochastic game to model the IM problem . Specifically , we introduce a surrogate objective function to optimize the return of agent i conditioned on all other agents , denoted by −i . Here , we make two approximations to get the surrogate objective function : 1 ) rearranging the sampling process by first sampling context dynamics then sampling local state/action/reward for each agent i ; 2 ) using context dynamics sampled by previous policies . Based on above surrogate objective , our algorithm consists of two iterative learning procedures : 1 ) obtaining context dynamics from joint simulator , 2 ) updating policy for each agent by data sampled from its respective local simulator conditioned on collective context dynamics . By decoupling each agent from all others with a separate training procedure , our method can greatly reduce model complexity and accelerate learning procedure , as shown by extensive experiments . It is worthwhile to mention that our method is not limited to IM , but can also be applied to many other applications with shared-resource structure , e.g. , portfolio management ( Ye et al. , 2020 ) , smart grid scheduling ( Remani et al. , 2019 ) , etc . Our contributions are summarized as follows : • We propose shared-resource stochastic game to capture the problem structure in IM , that agents only interact with each other through shared resource . • We propose a novel algorithm that leverages the shared-resource structure to solve IM problem efficiently . • Extensive experiments demonstrate that our method outperforms existing MARL algorithms on both profits and computing efficiency . 2 BACKGROUND . 2.1 STOCHASTIC GAMES . We build our work on stochastic games ( SGs ) ( Shapley , 1953 ) , since each SKU in IM problem has its own profit ( reward ) to optimize . A Stochastic Game is defined by a tuple ( N , S , { Ai } i∈N , T , { Ri } i∈N , γ ) , where N = { 1 , · · · , n } denotes the set of n > 1 agents , S denotes the state space observed by all agents , Ai denotes the action space of agent i . Let A : = A1 × · · · × An , then T : S × A → ∆ ( S ) denotes the transition probability from any state s ∈ S to any state s′ ∈ S after taking a joint action a ∈ A ; Ri : S × A × S → R is the reward function that determines the immediate reward received by agent i for a transition from ( s , a ) to s′ ; γ ∈ [ 0 , 1 ) is the discount factor . We can formulate the joint policy of other agents ’ as π−i = ∏ j∈−i π j . Each agent i optimizes its policy πi : S → ∆ ( Ai ) to maximize its own long-term reward , which is conditioned on other agents ’ behavior , defined as maxπiη i ( πi , π−i ) = E ( st , ait , a−it ) ∼T , πi , π−i [ ∞∑ t=0 γtrit ] . ( 1 ) We will illustrate the shared resource structure of IM problem in Section 2.2 , which motivates us to propose shared-resource stochastic game as a special case of stochastic game to capture such structure in Section 3.1 . 2.2 INVENTORY MANAGEMENT WITH SHARED RESOURCE . While a typical setting for IM shall involve a supply network of multi-echelon including stores , warehouses and factories , we simplify the setting to ease our presentation . In the following , we shall focus on scenarios with one store and multiple SKUs . We further assume that there is an upstream warehouse that can fulfill requirements from the store perfectly . Our objective is to learn high-quality replenishing policies for each SKU in the store , particularly when there are a large number of SKUs . As replenishing decisions for SKUs in stores should directly consider consumption behaviors of customers and competitions from other SKUs due to limited resources like spaces , budget etc. , they are more challenging to optimize comparing to SKUs in warehouses . It is worthwhile to mention that , due to the flexibility of RL algorithms , our method can also be applied to more complex settings with multi-echelon , fluctuate supply , non-deterministic leading time etc . Similar to previous work , we follow the multi-agent RL ( MARL ) framework with decentralized agents , each of which manages inventory of one SKU in the store . We assume that the store has n SKUs in sell , all of which share a common space that can store up to Imax units at the same time . Replenishing decisions of each SKU are made on discretized time steps , which are days in our paper . For each time step t and SKU i , let İit ∈ Z denote units of i that are in stock . Hence , the following constraint shall hold for all time steps : ∀t ≥ 0. n∑ i=1 İit ≤ Imax ( 2 ) At any time step t , the agent corresponding to SKU i may place a replenishment order to request Oit ∈ Z units of products from its upstream warehouse . These replenishment orders can not be fulfillment instantly , but will take several time steps , referred as leading time , before these products are delivered to the store . Let Li denote the leading time of SKU i and T it ∈ Z its total units in transit at the time step t. Meanwhile , demands from customers Dit may consume inventory of SKU i and cause an actual sale of units Sit ∈ Z . Due to the possibility of out-of-stock , Sit may be less than Dit . Formally , dynamics of these variables can be described as follows : Sit = min ( Dit , İ i t ) ( 3 ) T it+1 = T i t −Oit−Li+1 +O i t+1 ( 4 ) Îit = İ i t − Sit +Oit−Li+1 ( 5 ) ρ = { 0 if ∑n i=1 Î i t ≤ Imax∑n i=1 Î i t−Imax∑n i=1 O i t−Li+1 otherwise ( 6 ) İit+1 = İ i t − Sit + ⌊ ( 1− ρ ) Oit−Li+1⌋ ( 7 ) As we mentioned before , due to all SKUs share a common space , it may happen that the storage overflows when ordered products arrive . In this paper , we assume that the excess SKUs will be discarded proportionally according to ρ defined in Eq . 6 , which corresponds to the overflowing ratio if we accept all coming SKUs without considering the space constraint . To calculate ρ , we also introduce an extra variable Îit , which we call afterstate of İ i t . Intuitively , it denotes units of SKU i in stock at the end of the t-th time step if we omit the capacity constraint . We shall note that other manners e.g. , prioritizing all SKUs , to resolve space overflows are possible . The algorithm that we will introduce in the following section will also apply to these settings . Undoubtedly , such behaviors will cause extra operational cost which should be avoided as much as possible in our replenishing decisions . The intermediate profit Pt it of the i-th SKU is calculated according to the following equation : Pt it = piS i t − qiOit − oI [ Oit > 0 ] − hİit ( 8 ) where pi and qi are the unit sale price , unit procurement price for the i-th SKU , respectively , and o and h are the order cost and unit holding cost , respectively . I [ · ] is an indicator function which equals to one when the condition is true and zero otherwise . The order cost reflects the fixed transportation cost or the order processing cost , and yields whenever the order quantity is non-zero . For convenience , we summarize all notations in Table 2 in Appendix B , where we also give an example with 2 SKUs in Fig . 4 to further illustrate the whole procedure .

Summary: This paper considers the multi-period inventory replenishment problem for a single store with multiple SKUs. The problem is formulated as a resource constrained stochastic game, where the store capacity is the resource constraint shared by all SKUs. The paper takes a multi-agent RL approach to solve this problem, with each SKU being modeled as an agent and having its own replenishment policy. To tackle the challenge presented by the potentially large number of SKUs (agents), a context-aware decentralized PPO method is proposed, where each agent is trained independently in its local simulator given a sample path of the shared resource state (context). Benchmarking were performed on two domains with 5 SKUs and 50 SKUs each against several model-free MARL baselines. Results show that the proposed method can achieve comparable results but with much higher sample efficiency.