Multi-Agent Reinforcement Learning with Shared Resource in Inventory Management
Authors: Mingxiao Feng
Guozi Liu
Li Zhao
Lei Song
Jiang Bian
Tao Qin
Wengang Zhou
Houqiang Li
Tie-Yan Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Agent Reinforcement Learning
Inventory Management
Shared Resource
Decentralized Training Paradigm
Model-based RL

Abstract: We consider inventory management (IM) problem for a single store with a large number of SKUs (stock keeping units) in this paper, where we need to make replenishment decisions for each SKU to balance its supply and demand. Each SKU should cooperate with each other to maximize profits, as well as compete for shared resources e.g., warehouse spaces, budget etc. Co-existence of cooperation and competition behaviors makes IM a complicate game, hence IM can be naturally modelled as a multi-agent reinforcement learning (MARL) problem. In IM problem, we find that agents only interact indirectly with each other through some shared resources, e.g., warehouse spaces. To formally model MARL problems with above structure, we propose shared resource stochastic game along with an efficient algorithm to learn policies particularly for a large number of agents. By leveraging shared-resource structure, our method can greatly reduce model complexity and accelerate learning procedure compared with standard MARL algorithms, as shown by extensive experiments.

Under review as a conference paper at ICLR 2022
MULTI-AGENT REINFORCEMENT LEARNING WITH SHARED RESOURCE FOR INVENTORY MANAGEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
We consider inventory management (IM) problem for single store with a large number of SKUs (stock keeping units) this paper, where we need to make re- plenishment decisions for each SKU to balance its supply and demand. Each SKU should cooperate with each other maximize profits, as well as compete for shared resources c.g., warchouse spaces, budget etc. Co-existence of cooper- ation and competition behaviors makes IM complicate game, hence IM can be naturally modelled multi-agent reinforcement learning (MARL) problem. In IM problem, we find that agents only interact indirectly with each other through some shared resources, e.g., warehouse spaces. To formally model MARL prob- lems with above structure, we propose shared resource stochastic game along with efficient algorithm to learn policies partieularly for large number of agents. By leveraging shared-resource structure, our method can greatly reduce model complexity and accelcrate learning procedure compared with standard MARL al- gorithms, as shown by extensive experiments.
INTRODUCTION
Inventory management (IM) problem has long been one of the most important applications in the supply-chain industry (Nahmias & Smith, 1993). Its main purpose is to maintain balance between the supply and demand of stock keeping units (SKUs) in supply chain by optimizing replenishment decisions of each SKU. Besides leading to profit increment and operational cost reduction, efficient IM can even give rise to better services to customers. However, it is quite challenging task in practice, especially when there are lots of SKUs involved in the supply-chain. Particularly, while all SKUs should cooperate with each other to achieve high profits, they also need to compete for shared resources c.g., warehouse spaces, budget etc. Such co-cxistence of cooperation and competition renders IM a complicated game that is hard to address.
Traditional methods usually reduce IM problems to solving dynamic programming problems. How- ever, these approaches often rely on some unrealistic assumptions such as i.i.d. demand, deter- ministic leading time, etc. Moreover, as the state space grows exponentially along with some key factors like leading time and number of SKUs (Gijsbrechts et al., 2019), the corresponding dynamic programming problems become intractable due to the curse of dimensionality. Because of these lim- itations, many approaches based on approximate dynamic programming are proposed solve IM problems in different settings (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019). While these approaches perform well in certain scenarios, they heavily rely on problem-specific expertise or assumptions e.g., the zero or one period leading time assumption in (Halman et al., 2009), hence can hardly generalize to other settings. In contrast, reinforcement learning (RL) based methods, with fast inference speed, can generalize to various scenarios in a data-driven manner. However, it usually too costly to train a global policy that can make decisions for SKUs, since the training efficiency can be notably curtailed because of the large global state and action space (Jiang & Agar- wal, 8)).). To further address the training efficiency issue, it is natural to adopt the multi-agent reinforcement learning (MARL) paradigm, where each SKU is regarded as agent whose state and action spaces are localized and only contain information relevant to itself.
There are currently two popular paradigms to train MARL in the literature: independent learning (Tan, 1993) and joint action learning (Lowe et al., 2017). Despite of their success in many sce- narios, these two MARL paradigms also exhibit certain weaknesses that restrain their effectiveness

---

Under review as a conference paper at ICLR 2022
in solving IM problems. On hand, if applying independent learning, policy training of one agent simply treats all other agents as parts of the stochastic environment, hence is hard to converge due to non-stationary of the environment. On the other hand, joint action learning usually learns centralized critic conditioned on the joint action and state spaces of all agents, which can easily be- come intractable with increasing number of SKUs. Furthermore, it could be quite time-consuming to sample data from joint simulator for a great number of SKUs, since it usually involves much computation on many internal variables caused by complex agent interactions.
To address these challenges, we took a closer look at the IM problem and find that there exists a special structure that can be leveraged design more effective MARL paradigm. Particularly, each agent in IM only interacts with others through the shared resource, c.g., warchouse spaces. We introduce an auxiliary variable to represent the whole inventory level, implying the available shared resource, for all SKUs, and refer to this variable as context. From the MARL perspective, one agent can be influenced by other agents only through such context. The context dynamics actually reflect the collective behaviors of all agents. And, conditioned on context dynamics, one agent's state transition and reward function are independent other agents. In this way, leveraging context as an additional input for each agent's policy/value functions enables to both avoid the non-stationary problem caused by independent learning and mitigate the intractable centralized critics learning caused by exploding state and action spaces. Moreover, introducing context dynamics inspires us to build a local simulator for each agent in order to facilitate more efficient policy learning for each agent.
Based on this structure with context dynamics, we propose a shared-resource stochastic game to model the IM problem. Specifically, we introduce a surrogate objective function optimize the return of agent conditioned on all other agents, denoted by -i. Here, we make two approxima- rearranging the sampling process by first sampling
tions to get the surrogate objective function: context dynamics then sampling local state/action/reward for each agent i; 2) using context dynam- ies sampled by previous policies. Based on above surrogate objective, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics from joint simulator, 2) updating policy for each agent by data sampled from its respective local simulator conditioned on collective context dynamics. By decoupling each agent from all others with a separate training procedure, our method can greatly reduce model complexity and accelerate learning procedure, as shown by exten- sive experiments. It is worthwhile to mention that method is not limited to IM, but can also be applied to many other applications with shared-resource structure, e.g., portfolio management (Ye al., 2020), smart grid scheduling (Remani et al., 2019), etc.
Our contributions are summarized as follows:
We propose shared-resource stochastic game capture the problem structure in IM, that agents only interact with each other through shared resource.
We propose a novel algorithm that leverages the shared-resouree structure to solve IM problem efficiently.
Extensive experiments demonstrate that our method outperforms existing MARL algo- rithms both profits and computing efficiency.
2 BACKGROUND
2.1 STOCHASTIC GAMES
We build our work on stochastic games (SGs) (Shapley, 1953), since each SKU in IM prob- lem has its own profit (reward) to optimize. A Stochastic Game is defined by tuple I agents, S denotes the state space observed by all agents, A' denotes the action space of agent i. Let A := A x ... x A", then T S x A + A(S) denotes the transition probability from any state E S to any state s' S after taking joint action E A; R S A S I is the reward function that determines the immediate reward received by agent for transition from (s, a) to s':7 [0, 1) is the discount factor. We can formulate the joint policy of other agents'
(N,S, (A') "T, (Rr) :7. where N = (l,...,.. .n) denotes the set of
2

---

Under review as a conference paper at ICLR 2022
as π II,---I----- --- n². Each agent optimizes its policy n' S A (A') to maximize its own long-term reward, which is conditioned on other agents' behavior, defined as
max,in (T', x = (o,,0t, ()
(I)
t=0
We will illustrate the shared resource structure of IM problem in Section 2.2, which motivates us to propose reaoed-resource stochastic game as special case of stochastic game capture such structure in Section 3.1.
2.2 INVENTORY MANAGEMENT WITH SHARED RESOURCE
While a typical setting for IM shall involve supply network of multi-echelon including stores, warehouses and factories, we simplify the setting to ease our presentation. In the following, shall focus on scenarios with one store and multiple SKUs. We further assume that there is an upstream warchouse that can fulfill requirements from the store perfectly. Our objective is to learn high-quality replenishing policies for each SKU in the store, particularly when there are large number of SKUs. As replenishing decisions for SKUs in stores should directly consider consumption behaviors of customers and competitions from other SKUs due to limited resources like spaces, budget etc., they worthwhile mention that, due to the flexibility of RL. algorithms, our method can also be applied to more complex settings with echele, fluctuate supply, non-deterministic leading time etc.
are more challenging to optimize comparing to SKUs in warehouses.
Similar to previous work, we follow the multi-agent RL (MARL) framework with decentralized agents, each of which manages inventory of one SKU the store. We assume that the store has n SKUs in sell, all of which share common space that can store up to maz units at the same time. Replenishing decisions of each SKU are made on discretized time steps, which are days in our paper.
For each time step and SKU i, let i Z denote units of that are in stock. Hence, the following constraint shall hold for all time steps:
V 2 O. maax
At any time step I, the agent corresponding to SKU i may place replenishment order to request Of E units of products from its upstream warehouse. These replenishment orders cannot be fulfillment instantly, but will take several time steps, referred as leading time, before these products are delivered to the store. Let L, denote the leading time of SKU i and Ti E its total units in transit the time step t. Meanwhile, demands from customers Dj may consume inventory of SKU i and cause an actual sale of units S E Z. Due to the possibility of outkkf-stock, St may be less than Di. Formally, dynamics of these variables can be described follows:
S min (D:,
(3)
Ti T O + 0441
i = i; - St + +
(5)
if Ei S Imna
Th
(6)
otherwise
F/ = - 5
(7)
As we mentioned before, due SKUs share common space, may happen that the storage overflows when ordered products arrive. In this paper, we assume that the excess SKUs will be discarded proportionally according to p defined in Eq. 6, which corresponds to the overflowing ratio if we accept all coming SKUs without considering the space constraint. To calculate p. we also introduce an extra variable 1;. which we call afterstate of 17. Intuitively, it denotes units of SKU i in stock at the end of the t-th time step if we omit the capacity constraint. We shall note that other manners e.g. prioritizing all SKUs, resolve space overflows possible. The algorithm that
3

---

Under review as a conference paper at ICLR 2022
will introduce in the following section will also apply to these settings. Undoubtedly, such behaviors will cause extra operational cost which should be avoided as much as possible in our replenishing decisions.
The intermediate profit Ptş of the i-th SKU calculated according to the following equation:
pt = p; -0;; oI [0 > 0] hīg
where p; and qi are the unit sale price, unit procurement price for the i-th SKU, respectively, and and h are the order cost and unit holding cost, respectively. 1[-] is an indicator function which equals one when the condition true and zero otherwise. The order cost reflects the fixed transportation cost or the order processing cost, and yields whenever the order quantity is non-zero.
For convenience. we summarize all notations in Table in Appendix B, where we also give an example with 2 SKUs in Fig. to further illustrate the whole procedure.
3 METHODOLOGY
3.1 SHARED-RESOURCE STOCHASTIC GAME
In this section, show that IM problem introduced in Section 2.2 can be formulated as a shared-resource stochastic game, where each agent is only influenced by other agents through a shared resource pool. We define a shared-resource stochastic game (S*
tuple [18") VEN: : "eA')eNNT. (R') sseN: ?). where N = [1, .n} denotes the set of n >
agents, St denotes the state space of agent C denotes the context space observed by all agents, A' denotes the action space of agent i. Here, the context represents the occupied capacity of shared- resource. Let S = S' x .... x Sth and A = AA' x x.... A",, then T S x C x A A(S x C) denotes the transition probability, which can be decomposed as follows. The context is affected by St; ae). Since we dealing with resource, the transition function of context usually has some additive structure with respect to all agents, which we will illustrate later in Section 3.2. Given context Cc+1, the transition function of state for each agent independent with other agents, i.e., $1+1 ~ P.(- st, af c++1). Reward function is also independent with other agents given context, ri ~ Rs az c4+1). We refer P: and R' as local transition and reward functions. E [0, 1) is the discount factor. Each agent i optimizes its own policy ा² St x C + A (A') to maximize the expected long-term reward conditioned on other agents' policies π defined as
all agents, i.e., C++11 Po(. St,
max (T²', " () = B (s,, "TTTT
Given the above definition, an IM problem can be formulated as a shared-resource stochastic game by letting rf = Ptç at = Oj. and c = Σm= I for all SKU and time step t. Moreover, state of each SKU will be a concatenation of information like Tü Ps, its historical actions and demands etc. A detailed description of the state space can be found in the Appendix E.3.
3.2 SURROGATE OBJECTIVE WITH CONTEXT DYNAMICS
We now introduce how to optimize the return for agent i conditioned on other agents' behaviors. Roughly speaking, the context dynamies reflect the collective behaviors of other agents. It possible to estimate the objective function approximately for each agent only based on its local transition dynamics, local reward function and the context dynamics. First, we give a detailed deseription of the transition model of shared resource e from each agent's perspective. Given such dynamics for the shared resource, we then show how to approximate the objective function in Eq. 9 by rearranging the sampling process. Finally, we replace the policies for sampling context dynamics by old policies from the previous iteration to accelerate the decentralized training.
3.2.1 CONTEXT DYNAMICS AND LOCAL SIMULATOR
We use of to represent the amount of resource occupied by agent at time step and & the total amount of occupied resource. We further let ह- denote the amount of resource occupied by all
4

---

We use notation C to represent (2)-1,4). We refer (s", c²) as the state for agent i, and c as the context. For each agent, have the following sampling process T given the context dynamics of Under review as a conference paper at ICLR 2022
agents but i. Since the capacity has the additive structure, we have the following equations:
à = = c 4 C +
(10)
Similarly, denote the afterstate of e é.. From the perspective of agent i, it can view the context 6 as part of the environment. Hence, given s/ e, and a), P(č s), al. e)) represents how the replenishment decision influences a. Due the additive structure, we have & = é + + Then by applying a resource overflow resolution procedure as in Eq. 6, we obtain 2,4 representing state of the resource in the next step.
ct ~ Po(. o oi 1/1 ~ P.l. 1
642 (
(11)
ri (s oi (411
Given the context dynamics for e we can build local simulator for agent according to the above equations. To ease our presentation, we only consider one kind of shared resource i.e., warehouse spaces in the above formulation. However, can be extended to support multiple resources as long as these resources have the additive structure as in Eq. 10.
3.2.2 SURROGATE LOss FUNCTION
To leverage local simulators, we rearrange the sampling process for evaluating 77(5=, +) as follows. First, we follow the regular sampling process in Eq. 9, and get samples of c, Then, we re-sample (st, af, cf) based on samples of G following Eq. 11. It is worth noting that we make an approx- imation here. Given samples for ei we actually need to inference the posterior of (s af ct). However, since we consider scenarios with lots of agents, is reasonable assume that each agent i has limited influence on its context e-i, Therefore, we assume that c, has limited influence on the inference of (st, at, c)) and sample the latter directly according to Eq. 11. By rearranging the sampling process, we obtain the following surrogate objective,
maz " = Le, 'TTT E (1
(12)
t=0
In practice, quite costly to sample from the joint dynamics T, but much cheaper to sample from the local dynamics T. To leverage data samples efficiently, we propose use the samples from previous policies in our objective function, which is a common trick in reinforcement learning. Specifically, we use samples of c, collected by policies (T ग Tid) of the previous iteration, and further rewrite the surrogate objective function follows:
mf ff' oodd क Told= = Bie ")~T mass Ff
(13)
As long as current policies stay close to previous ones, we can adopt above surrogate objective function to improve sample efficiency.
3.3 ALGORITHM
In the following, we will present details about our proposed algorithm, which is referred as Context- aware Decentralized PPO (CD-PPO). We call context-awind aware as the context c, is part input to train policy and value functions of each agent. Such context-aware approach can avoid the non- stationary problem occurred independent learning methods (which does not consider dynamics of others during training), since the context reflects other agents' collective behaviors which can
5

---

Under review as a conference paper at ICLR 2022
GetContextDynamics
Sampled
Ent
(c Jiso
Enrlecat
P' 4u 1i 4.47.
Sampled Data
DecentralizedPPO
Figure 1: Our algorithm consists of iterative learning procedures: Get context dynamics {c J from the joint simulator and previous policies Tola Told" 2. Train policy æ' with data sampled from the local simulator conditioned on context dynamics {cr'T JE=D-
impact dynamics of each individual agent. In the meanwhile, method can mitigate the intractable centralized critic learning by avoiding using exploding joint state and action space, i.e., (8t, ae, Gt), as input of the critic.
As shown in Algorithm I, our algorithm consists of two iterative learning procedures: 1) obtaining context dynamics by running all agents in the joint environment (Algorithm 2): 2) updating policy of each agent using data sampled from its local simulator conditioned on the context dynamics (Al- gorithm 3). Moreover, our algorithm follows decentralized training paradigm with an acceptable communication cost of the context dynamics. We refer readers to Appendix C for detailed descrip- tion.
It is worth noting that a naive approach is to optimize policies by Eq. 9 with data sampled from the joint simulator. Nonetheless, we find that it is empirically more time-consuming to sample one step in the joint simulator than letting each of the π. local simulators sample once. One major reason lies in that agent interactions take most the simulation time. Our method, with the advantage of henceforth much cheaper to sample. We
leveraging local simulators to simplify interactions, refer readers to Appendix for more discussions on the benefit leveraging local simulators.
Algorithm Context-aware Decentralized PPO
Given the joint simulator jnvjoint and local simulators {Env) Hi Initialize policies n and value functions V for i = I....,n
for M epochs do // Collect context dynamics via running joint simulation "T= <- GelContextDynamiess/ (arumiccsEEyjonnee joint b))) (Algorithm 2)
{cr 17 . (
for = 1,2...... K do
for all agents do
// Set capacity trajectory by context dynamics
Enviocat- wwt..cc.t.aeetorrrc(c i }(=)))
// Train policy by running simulation in the corresponding local environment
n² V +- DecentralizedPPO/ (Enviocats x², Vi) (Algorithm 3)
end for end for
Evaluate policies {x J joint simulator Eny joint end for

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We evaluate the performance of CD-PPO in three IM problems, which contain 5, 50, and 100 SKUs, respectively. By changing space size of the warchouse, we further testify how CD-PPO performs comparing to series of baselines under different levels of competition. On these settings, we demonstrate that our algorithm can achieve comparable results as SOTA baselines, but is more sample-efficient.
4.1 EXPERIMENT SETUPS
Our experiment is conducted on simulator which can support both retailing and replenishment for multiple SKUs in one store. Instead of sampling demands from some hypothetical distributions, we instead directly use authentic demand data from Kaggle (Makridakis et al., 2020), which con- tains sales history of five years (2011-20) (2011-2016) for various SKUs in Walmart. We randomly choose 155 SKUs from the data and use sales of the first four years as our training set, while the oth- ers are the testing set. For all other information, e.g. price, cost, leading time etc., that are necessary to instantiate our simulator but not included in the data set, we will randomly sam- ple them from certain reasonable ranges. The evaluation metric is the total profit in dollars. For all results that present, run each of all algorithms for four times with random seeds and present the average performance with standard deviations. The source code as well as instruc- tions to reproduce our results can be found in https:/ /anonyoooouus 4open. science/r/ replen shment-ma r basel nes-- 75F4.
Note that the simulator we are using is developed for general purposes, it contains extra details like replenishing orders fulfillment scheduling, transportation management etc., hence it requires lots of computation resources simulate IM problems with many SKUs (> 100). This also hinder us to extend our experiments to cases with more than 100 SKUs on single machine. We leave it our future work further testify our algorithm in distributed environment where simulating large scale IM problems is possible.
We compare our method CD-PPO with the strong model-free on-policy MARL algorithms MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020) on 5-SKUs environment and we do not show the performance of COMA (Foerster et al., 2018) and Value Decomposition methods such as QMIX (Rashid et al., 2018) due to their bad performance (i.e. negative profit). As for 50-SKUs and 100-SKUs scenario, we can only run IPPO, since MAPPO fails quickly as it suffers from the huge joint state-action space for training a centralized critic. Besides RL approaches, we also compare with base-stock policy, which is a well-known policy from OR community and widely adopted in practice (Kapuscinski & Tayur, 1999; Hubbs al., 2020). (More details can be found in Appendix F). Our implementation is based on the EPyMARI. (Papoudakis et al., 2) framework, which contains MAPPO, IPPO and several common-used MARL algorithms.
As all agents are homogeneous in IM, we let parameters of policy network and critic network be shared amongst all agents. The two networks are all constructed by two-layer MLP with hidden size 64. In our experiment, the actor network maps agent-speciffi state to a categorical distribution over a discrete action space {0, 3: 1, 3: 2, 3. 3, 4,5, 6, 7, 12), such that the real replenish- ment quantity is obtained by multiplying the action with an agent's sales mean of the past two weeks. In addition, to encourage policies to accommodate diverse context dynamics, we propose two methods to augment the context dynamics: 1) add noise to randomly chosen items with pre- defined probability; 2) replace randomly chosen items with predicted values coming from context generator model, which is also guided by the predefined probability. Besides, we also use the data collected in Algorithm for policy training. More details about the algorithm be found in Appendix C.
4.2 MAIN RESULTS
We evaluate CD-PPO, MAPPO and IPPO on 5,50, 100000K0UU environments with different sizes of warchouse spaces. IPPO and MAPPO only individual rewards rather than team reward (sum- mation of all individual rewards) to train critics. For a fair comparison, we also train IPPO and MAPPO with information related to shared resource. The training curves for 5-SKUs scenario

---

Under review as a conference paper at ICLR 2022
Table I: Profit Comparison on Different Scenarios of 50 and 100 SKUs Environment
stocis
N50-C300
310.81 1 10.36
235009 1 01.01
250.08
58.55
16 165.01
366. 74
89
108.11
No0.L No0-C2000 2000
694.87 174. 184
680.27 # $8.01
646.90 + 450.71
1376 299 850,03
11022.97
1116.68
42.71
N100-C1900
660 28
-2106 315.38
42 109.3
-1708
1306.92
-22.5
shown in Figure 2. is straightforward to observe that CD-PPO converges to the same performance comparing with other methods. In particular, CD-PPO is more sample-efficient due to its local sim- ulations in parallel. To evaluate our algorithm on larger scenario, also run CD-PPO and IPPO (same with previous settings) on 50 and 100 SKUs and record the number of data samples when reaching the median performance of baselines. The results are summarized in Table and Figure 3, where "N" and "C" denote the number of SKUs and the maximum capacity of shared resource, respectively. More details about the implementation and hyper-panne parameters we used can be found in Appendix C.
Profit Comparison
Avaragi Cata Samplis taban reach median baselina perfarmence(104)
1.5
1200
pro4R/wb contecti
1.0
1000
0.5
800
Profit
0.0
MAPPO-IRIWAN
400
IPPO-IR
200
MAFPO-IR
PPOOPOOt
N100-C1000
Samples
Eny Snv_Scenario
Figure 2: Training curves of different algorithms in 5-SKUs environment. "IR" means the environ- ment only provides individual rewards and "w/o context" denotes the algorithm does not use infor- mation related to shared resource as parts of its inputs. The X-axis also takes in samples from local simulation for CD-PPO.
Figure Average samples needed by different algorithms in 100-SKUs en- vironment to reach the median perfor- mance of baselines. The lower the value, the higher the sample-efficinncy for the algorithm.
As demonstrated Figure 2, Figure and Table 1, CD-PPO is able to produce results comparable to other strong baselines and even continues improving as the training proceeds. Moreover, our al- gorithm can curtail non-stationary effects as in centralized training and, in the meanwhile, can scale up to scenarios with large number of agents. In contrast, traditional MARL methods with CTDE paradigm even cannot start running on the 50-SKUs and the 100-SKUS environment, since the in- successfully, but
put of its critic is too huge to fit in the memory. IPPO, the other hand, lack stability under different levels of warehouse spaces. The full results and more ablation studies about how the capacity of shared resource affects the performance of CD-PPO and the influence of augmentation for context dynamics can be found in Appendix G.
5 RELATED WORK
In this section we will introduce relevant prior work including studies of 1M problem and common-used training paradigms MARL. More detailed description of other related work(c.g. Constraned/Weally-CCoulp MDP, Model-Based MARL and Mcan-Filed MARL) are shown in Appendix A.
'Specifically, for one interaction in the global environment with N agents, we consider as N data samples. For one interaction in the local (imulator(based context trajs) with agent, we consider as data sample.
8

---

Under review as a conference paper at ICLR 2022
5.1 INVENTORY MANAGEMENT
Since the pioneer work in (Fukuda, 1964), many approaches have been proposed to solve different variants of IM problems. either using exact (Goldberg et al., 2016; Huh et al., 2009) or approxi- mate (Halman et al., 2009; Fang et al., 2013; Chen & Yang, 2019) dynamic programming. As rein- forcement learning based approaches are our main focus, we only present related work of this branch in the following. Interested readers will be referred to (Gijsbrechts et al., 2019) for an overview of traditional approaches for IM.
The attempt to apply reinforcement learning to solve inventory management problem has a long history, see for instance (Giannoccaro & Pontrandolfo, 2002; Jiang & Sheng, 2009; Kara & Dogan, 2018; Barat et al., 2019; Gijsbrechts et al., 2019; Oroojlooyjadid et al., 2017; 2020). However, as their main focus to deal with challenges like volatile customer demands, bullwhip effects etc. in IM, they restricted simplified scenarios with only single SKU. While these approaches able to outperform traditional approaches in these scenarios, they overlook system constraints and coordination of SKUs imposed by shared resources. Exceptions are two recent works (Barat al., 2019; Sultana et al., 2020), where more realistic scenarios containing multiple SKUs are considered. In (Barat et al., 2019), the main contribution is to propose framework that supports efficient de- ployment of RL algorithms in real systems. As an example, the authors also introduce a centralized algorithm for solving IM problems. In contrast, a decentralized algorithm is proposed in (Sultana et al., 2020) to solve IM problems with not only multiple SKUs but also multiple echelons. In both works, the training algorithm is the advantage actor critic (A2C) (Wu al., 2018).
5.2 TRAINING PARADIGM IN MARL
MARL algorithms generally fall between two frameworks: centralized and decentralized learn- ing. There two lines of fully decentralized training: Independent Learning methods (IL.) (Tan, 1993; de Witt et al., 2020) and decentralized training with communication (Zhang et al., 2018; Sukhbaatar et al., 2016; Peng et al., 2017). For IL, each agent learning independently to optimize its own reward and perceives the other agents as part of the environment. As for fully decentralized methods, representative methods usually build a direct communication pipe to share the message amongst agents to avoid non-stationary issue in MARL framework. One part of fully centralized approaches (Claus Boutilier, 1998) assume cooperative game and directly extend single-agent RI. algorithms by learning single policy to produce the joint actions of all agents simultaneously. Another type of centralized methods is called value dccomposition (VD), which typically represents the joint Q-function as function of agents' local Q-functions (Sunchag et al., 2017; Son et al., 2019; Rashid et al., 2018) and has been considered gold standard in MARL. In contrast to pre- vious methods, Centralised Training Decentralised Execution (CTDE) allows sharing of informa- tion during training, while policies are only conditioned on the agents' local observations enabling decentralised execution. The main category of CTDE algorithms methods in which each agent consists of a decentralised actor and a centralised critic, which is optimised based on shared information between the agents. Representative studies of CTDE are MADDPG (Lowe et al., 2017), COMA (Foerster et al., 2018) and MAPPO (Yu et al., 2021) etc.
centralised policy gradient
CONCLUSION
In this paper, we address inventory management problem for single store with a large number of SKUs. Our method is based on shared resource structure, where agents only interact indirectly with each other through shared-resource. By leveraging such structure, our method can outperform existing MARL algorithms in terms of both final performance and computation efficiency. It is worth mentioning that our method is not limited to IM, but also applicable wide range of real-world applications with shared-resoure structure.
In real-world applications, we usually need to deal with thousands of agents, which poses a challenge for existing MARL algorithms. To address such challenges, we need develop efficient algorithms which are compatible with distributed training. In this paper, we take our first step towards de- veloping efficient and practical MARL algorithms for real-world applications with shared-resoure structure, and will continue to address above challenges arisen worl-world applications in our future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Pritee Agrawal, Pradeep Varakantham, and William Yeoh. Scalable greedy algorithms for task/resource constrained multi-agent stochastic planning. (2016). In Proceedings of the 25th In- ternational Joint Conference on Artificial Intelligence IJCAI 2016: New York, July 9, volume 15, pp. 10-16, 2016. A.2
Souvik Barat, Harshad Khadilkar, Hardik Meisheri, Vinay Kulkarni, Vinita Baniwal, Prashant Ku- mar, and Monika Gajrani. Actor based simulation for closed loop control of supply chain us- ing reinforcement learning. In Proceedings of the 1 18th international conference on autonomous agents and multiagent systems, pp. 1802-1804, 2019. 5.1
Shalabh Bhatnagar and K Lakshmanan. An online actor-critic algorithm with function approxima- tion for constrained markov decision processes. Journal of Optimization Theory and Applications, 153(3):688-7000 708, 2012. A.2
Craig Boutilier and Tyler Lu. Budget allocation using weakly coupled, constrained markov decision processes. In Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence (UAI- 16). pp. 52-61, New York, NY, 2016. A.2
René Carmona, Mathieu Laurière, and Zongjun Tan. Model-free mean mean-field reinforcement learning: mean-field mdp and mean-field q-learning. arXiv preprint arXiv:1910.1280) 2019, A.1
Wenbo Chen and Huixiao Yang. A heuristic based on quadratic approximation for dual sourcing problem with general lead times and supply capacity uncertainty. IISE Transactions, 51(9):943- 956, 2019. ISSN 24725862.
doi: 10. 10.1080/24725854.200 1.337533 5.1
Yu Fan Chen, Miao Liu, Michael Everett, and Jonathan P How. Decentralized non-communicating multiagent collision avoidance with deep reinforcement learning. In 2017 IEEE international conference on roboties and automation (ICRA), pp. 285-292 IEEE, 2017. A.1
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/AAI,, 1998(746-752):2, 1998. 5.2
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr. Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 2020. 5.2
Raghuram Bharadwaj Diddigi, Sai Koti Reddy Danda, Shalabh Bhatnagar, et al. Actor-critic al- gorithms for constrained multi-agent reinforcement learning. arXiv preprint arXiv: nete:9905000900 2019. A.2
Roel Dobbe, David -ridovich-Keil, and Claire Tomlin. Fully decentralized policies for multi-agent systems: An information theoretic approach. arXiv preprint arXiv:1707.06333 2017. A.1
Dmitri A Dolgov and Edmund H Durfee. Resource allocation among agents with mdp-induced preferences. Journal of Artificial Intelligence Research, 27:505-5444 549, 2006. A.2
Jiarui Fang. Lei Zhao, Jan C. Fransoo, and Tom Van Woensel. Sourcing strategies in supply risk management: An approximate dynamic programming approach. Computers & Opera- tions Research, 40(5):1371-1382. 2013. ISSN 0305-0548. doi: http::/dbi.org//0.101//// 1016 2012.08.016. URL https:/ hwww/wwwwst www.sccienedireec... cwm/ htmm///ienne/articleee 503050548120018999 5.1
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 5.2,
Michael Fowler, Pratap Tokekar, Charles Clancy, and Ryan K Williams. Constrained-action pomdps for multi-agent intelligent knowledge distribution. In 2018 IEEE International Con- ference on Robotics and Automation (ICRA), pp. 3701-3708. IEEE, 2018 A.2
Yoichiro Fukuda. Optimal Policies for the Inventory Problem with Negotiable Leadtime. Manage- ment Science, 10(4):690-708, 1964. ISSN 0025-1909. doi: 10. 10.1287/mnsc. 10.4.690. 5.1
10

---

Under review as a conference paper at ICLR 2022
Ilaria Giannoccaro and Pierpaolo Pontrandolfo. Inventory management in supply chains: A rein- forcement learning approach. International Journal of Production Economics, 78(2):1531611, 2002. ISSN 09255273, doi:
5.1
Joren Gijsbrechts, Robert N. Boute, Jan Albert Van Mieghem, and Dennis Zhang. Can Deep Rein- forcement Learning Improve Inventory Management? Performance and Implementation of Dual Sourcing-Mode Problems. SSRN Electronic Journal, pp. 1-33, 2019. ISSN 1556-5068. doi:
10.2139/ssm.3302881. 1.
David A Goldberg, Dmitriy A Katz-Rogozhnikov, Yingdong Lu, Mayank Sharma, and Mark Squillante. Asymptotic optimality of constant-order policies for lost sales inventory models with large lead times. Mathematics of Operations Research, 41(3):898-913, 2016. 5.1
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017. A.1
Nir Halman, Diego Klabjan, Mohamed Mostagir, Jim Orlin, and David Simchi-Levi. A fully polynomial-ting approximation scheme for single-item stochastic inventory control with discrete demand. Mathematics of Operations Research, 34:674-685, 2009. 1, 5.1
Christian D. Hubbs, Hector D. Perez, Owais Sarwar, Nikolaos V. Sahinidis, Ignacio E. Grossmann, and John M. Wassick. Or-gym: A reinforcement learning library for operations research prob- lems, 2020. 4.1,
Woonghee Tim Huh, Ganesh Janakiraman, John A. Muckstadt, and Paat Rusmevichientong. Asymptotic optimality of order-up-to policies in lost sales inventory systems. Management Sci- ence, 45(3):404-420) 2009. ISSN 00251909, 15265501. URL http: htwp///wwwwww/m jstor. otr.soooorroo....oo.oooooooooo org/ stable/ e 0443945555666 5.1
Chengzhi Jiang and Zhaohan Sheng. Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system. Expert Systems with Applications, 36(3 PART 2): 6520-6526, 2009, ISSN 09574174. doi: 10. 1..10166..swa..000 2008 00..00..333 .036. 5.1
Nan Jiang and Alekh Agarwal. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pp. 3395-3398. PMLR, 2018.
Roman Kapuscinski and Sridhar Tayur. Optimal Policies and Simulation Basedd Optimization for Capacitated Production Inventory Systems, pp. 7-40. Springer US, Boston, MA, 1999. ISBN 978- 1-4615-499--9
doi: 10. 10010007978-100 (1177/978811441554499994 2.2. URL httpt::////aat /0t////////11..1111000007///
978 1--1--466660 4614--111-011 4949-99999 9 __ 4.1
Ahmet Kara and Ibrahim Dogan. Reinforcement learning approaches specifying ordering poli- cies of perishable inventory systems. Expert Systems with Applications, 91:150-158, 2018. ISSN 09574174. doi: 10.10166).swwa22017...8
5.1
Orr Krupnik, Igor Mordatch, and Aviv Tamar. Multi-agent reinforcement learning with multi-step generative models. In Conference Robot Learning, pp. 776-790. PMLR, 2020. A.3
Joel Z Leibo, Vinicius Zambaldi, Mare Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. arXiv preprint arXiv:1702.03033 2017.
A.1
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4213 4213-4220, 2019. A.1
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed copprraive-competinee environments. arXiv preprint 17X.v:1706.02275, 2017. 1.2, A.1
Xueguang Lyu, Yuchen Xiao. Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics multi ageet reinforcement learning. arXiv preprint arXiv:2102.04402, 2021. A.1
11

---

Under review as a conference paper at ICLR 2022
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. arXiv preprint arXiv:1910.07483, 2019. A.1
S. Makridakis, E. Spiliotis, and V. Assimakopoulos. The m5 accuracy competition: Results, findings and conclusions. International Journal of Forecasting. 36(1):224 227, 2020. 4.1
Andrei Marinescu, Ivana Dusparic, Adam Taylor, Vinny Cahill, and Siobh Clarke. P-marl: Prediction-based multi-agent reinforcement learning for non-stationary environments. AAMAS, pp. 1897-1898, 2015. A.3
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016. A.I, C.2
Steven Nahmias and Stephen A Smith. Mathematical models of retailer inventory systems: A review. Perspectives in operations management, pp. 249-278, 1993.
Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer, 2016. A.2
Afshin Oroojlooyjadid, M Nazari, Lawrence Snyder, and Martin Takáč. deep q-network for the beer game: A reinforcement learning algorithm to solve inventory optimization problems. arXiv preprint arXiv:1708.0599 2017. 5.1
Afshin Oroojlooyjadid, Lawrence V. Snyder, and Martin Takáč. Applying deep learning to the newsvendor problem. IISE Transactions, 52(4):444-463, 2020. ISSN 24725862. doi: 10.
211.5554.2019116325515 5.1
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Confer- ence Neural Information Processing Systems Datasets and Benchmarks Track (Round l), 2021. URL https: wwppppeeeeviiieeeeeeeeeeeeeeeeee net / forum? id=cIrXX Sn5n. 4.1, E.1
Young Joon Park, Yoon Sang Cho, and Seoung Bum Kim. Multi-agent reinforcement learning with approximate model learning for competitive games. PloS one, 14(9)::0222215, 2019. A.3
Bei Peng, Tabish Rashid, Christian A Schroeder de Witt, Pierre-Alexandre Kamienny, Philip HS Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. arXiv preprint arXiv:2003.06700 2020. A.l
Peng Peng. Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang. Haitao Long, and Jun Wang. Multiagent fitrrrctiinally-cordnnnet nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv: 1006, 2017. 5.2
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp. 2295-4304. PMLR, 2018.
5.2, A.1
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints, pp. arXiv-2006, 2020. A.1
T. Remani, E. A. Jasmin, and T. P.Imthias Ahamed. Residential Load Scheduling with Renewable Generation in the Smart Grid: A Reinforcement Learning Approach. IEEE Systems Journal, 13 (3):3283-3294, 2019. ISSN 19379234. doi: 10. S0.109///YST.221888555552
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. C.2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017, A.
12

---

Under review as a conference paper at ICLR 2022
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10): 1095-1100, 1953. 2.1
Kyunghwan Son, Daewoo Kim, Wan Ju Kang. David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019. 5.2, A.1
Sriram Srinivasan, Mare Lanctot, Vinicius Zambaldi, Julien Pérolat, Karl Tuyls, Rémi Munos, and Michael Bowling. Actor-critic policy optimization in partially observable multiagent environ- ments. arXiv preprint arXiv: 1810.000, 2018. A.1
DJ Strouse, Max Kleiman-Weiner, Josh Tenenbaum. Matt Botvinick, and David Schwab. Learning to share and hide intentions using information regularization. arXiv preprint arXiv:/808. 1808.02093 2018. A.1
Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropaga- tion. Advances in neural information processing systems, 29:2244 2252, 2016. 5.2
Nazneen Sultama, Hardik Meisheri, Vinita Baniwal, Somjit Nath, Balaraman Ravindran, and Har- shad Khadilkar. Reinforcement learning for supply chains. arXiv preprint arXiv:2006. 04037, 2020. 5.1
multi-node inventory management in
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. deloo-decompoosition networks for cooperative multi-agent learning. arXiv preprint arXiv: /706.05999, 2017. 5.2, A.1
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993. 1.2.2
Jianhong Wang, Yuan Zhang, Tac-Kyun Kim, and Yunjic Gu. Shapley q-value: a local reward approach to solve global reward games. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7285-7292 2020a. A.1
Tonghan Wang, Jianhao Wang, Chongyi Zheng. and Chongjie Zhang. Learning nearly decompos- able value functions via communication minimization. arXiv preprint arXiv: (xXiv:9900005366 2019. A.1,
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.08039, 2020b. A.1
Yuhuai Wu, Elman Mansimov, Shun Liao, Alec Radford, and John Schulman. Openai baselines: 2018. 5.1
Acktr & a2c. https:///// /ttp::::///pea..i// openai.e.....///..
Yaodong Yang. Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mcan field multi- agent reinforcement learning. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5567-5576, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. A.I
Yunan Ye, Hengzhi Pei, Boxin Wang, Pin Yu Chen, Yada Zhu, Jun Xiao, and Bo Li. Reinforcement- learning based portfolio management with augmented asset movement prediction states. AAAI 2020 34th AAAI Conference on Artificial Intelligence, pp. 1112-1119, 2020. ISSN 2159-5399. doi: 10. 20.1609/aaiivv34jpp v34201.5462.
Chao Yu, Akash Velu, Fugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955
2021. 4.1, 5.2, A.1, C.I
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi- agent reinforcement learning with networked agents. International Conference on Machine Learning. pp. 5872-5881. PMLR, 2018. 5.2
Ruohan Zhang, Yue Yu, Mahmoud El Chamie, Behçet Açikmese, and Dana H Ballard. Decision- making policies for heterogencous autonomous multi-agent systems with safety constraints. In IJCAI, pp. 546-553, 2016. A.2
13

---

Under review as a conference paper at ICLR 2022
Weinan Zhang, Xihuai Wang. Jian Shen, and Ming Zhou. Model-based multi policy optimiza- tion with adaptive opponent-wise rollouts. arXiv preprint arXiv:2105.03363 2021. A.3
14

---

Under review as a conference paper at ICLR 2022
A RELATED WORK
A.1 TRAINING PARADIGM IN MARL
In this section we will provide a more detailed description of related work for training paradigm used in MARL. From the perspective of training schemes, it can be devided into three categories: decentralized training decentralized execution (DTDE), centralized training centralized execution (CTCE) and centralized training decentralized execution (CTDE), Recent deep MARL works often use the CTDE or CTCE training pradigm.
The CTCE paradigm allows the straightforward employment of single-agent approaches such as Actor-Critic (Mnih et al., 2016) or policy gradient algorithms (Schulman et al., 2017) to multi-agent problems. The representative work of CTCE is (Gupta et al., 2017), which represented the centralized executor a set of independent sub-policies such that agents' individual action distributions are captured rather than the joint action distribution of all agents.
Value-based CTDE approaches, which are also known as Value Decomposition methods (Peng et al., 2020; Mahajan et al., 2019; Rashid et al., 2020; 2018; Son et al., 2019; Sunehag et al., 2017; Wang et al., 2020b; 2019), mianly focus on how centrally learned value functions can be reasonably decoupled into decentralized ones and have shown promising results. methods on CTDE, on the other hand, have heavily relied on centralized critics. One of the first works utilizating centralized critic was COMA (Foerster et al., 2018), framework adopting centralized critic with a counterfactual baseline. For convergence properties, COMA establishes that the overall effect on decentralized policy gradient with a centralized critic can be reduced to single-agen actor-critic approach, which ensures convergence under the similar assumptions like A2C.
Dolicy-gradient-o
Concurrently with COMA, MADDPG (Lowe et al., 2017) proposed to dedicated centralized critic for each agent in semi-competitive domains, demonstrating compelling empirical results in continuous action environments. Recently, MAPPO (Yu et al., 2021), on-policy policy gradient multi-agent reinforcement learning algorithm, achieves strong results comparable to the state-of-the-art on a variety of cooperative multi-agent challenges. Despite its on-policy nature, MAPPO is competitive to ubiquitous off-pollcy methods such as MADDPG, QMix, and RODE in terms of final performance, and in the vast majority of cases, comparable to off-policy methods in terms of sample-efficiency. In addition, many incremental research inspired by MADDPG, COMA or MAPPO also borrowed the centralized critic baselines e.g. M3DDPG (Li et al., 2019), SQD- DPG (Wang et al., 2020a), etc. Mean Filed Q-Learning (Yang et al., 2018; Carmona et al., 2019) takes a different approach from the CTDE based methods. It employs mean field approximation over the joint action space in order to address the scalability issue that exists in the prior methods.
Contrary to CTDE, in DTDE paradigm, each agent has associated policy which maps local observations to a distribution over individual actions. No information is shared between agents such that each agent learss independently. DTDE has been applied cooperative navigation task (Chen et al., 2017; Strouse et al., 2018), to partially observable domains (Dobbe et al., 2017; Srinivasan al., 2018), and to social dilemmas (Leibo et al., 2017). For more comparisons of centralized critic and decentralized critic, please see (Lyu et al., 2021).
In this paper, we design a decentralized traning paradigm avoiding the flaws of traditional training paradigms proposed in literature. The fundamental drawback of the DTDE paradigm is that the environment appears non-stationary from single agent's 's viewpoint because agents neither have access to the knowledge of others, nor do they perceive the joint action. Some studies reported that DTDE scale poorly with the number of agent due to the extra sample complexity, which is added to the learning problem (Gupta et al., 2017); An obvious flaw for CTDE/CTCE is that state-action spaces grow exponentially by the number of agents. Even though there are some attempts proposed that the joint model can be factored into individual policies for each agent To address the so-called curse of dimensionality, CTDE/CTCE methods have use at least joint states overall agents the input to approximate global value function to give guidance for centralized critics or decentralized policies. Meaning that these traditional training schemes still not have strong expansion capabilities
15

---

Under review as a conference paper at ICLR 2022
to large number of agents when system's state is combined by local state for each agent. As for our approach, we train agents independently with learned dynamics model of utilization's trend for shared resource, which will give agent enough information to learn optimal policy (we will explain it in the following sections). At the meantime, we can also, improve efficiency of data sampling since don't always use the original joint simulator containing all agents for data collection. Instead, we mainly running the light-cost local simulator by embedding the learned dynamic model, which can significantly reduce the cost of data collection process, especially when running joint simulator (such as inventory management) expensively.
A.2 MDP SCENARIOS MARL
There are limited similar variants of MDP settings which have been studied under MARL frame- work, to our knowledge. Dec-POMDP (Olichoek & Amato, 2016) is the most common setting stud- ied in MARL research, especially in fully cooperative tasks in which agents share the global team reward rather than individual rewards; In the scenarios of Constrained MDP setting (C-MDP, (Bhat- magar& & Lakshmanan, 2012; Wang et al., 2019; Diddigi al., 2019)), there some constraints in the system, such as the penalty caused by illegal actions in autonomous driving (Zhang et al., 2016; Fowler et al., 2018), the resource-allocation constraints in scheduling tasks (Agrawal et al., 2016; Dolgov & Durfee, 2006), or limited bandwidth during communicating among agents (Fowler al., 2018), etc. Similar with C-MDP, Weakly-Couples Constrained MDP (WC-C-MDP (Boutilier & Lu, 2016)) consider the problem of budget (or other resource) allocation in sequential decision problems involving a large number of concurrently running sub-processes, whose only interaction through their consumption of budget. Different from mentioned scenarios, we focus on the situa- tion where agents can only get the individual reward from the environment, so that team reward is invisible; Comparing with C-MDP and WC-C-MDP, the penalty of overstocking in which is related with stock levels of all agents in future, while in C-MDP and WC-C-MDP, the corresponding cost function is only based on historical states of the system. Leading that can not optimize only the combined objective of long-term summation of return and penalty.
A.3 MODEL-BASSE MARL
For model-based MARL, there are relatively limited works of literature. P-MARL (Marinescu et al., 2015) proposed an approach that integrates prediction and pattern change detection abilities into MARL and thus minimises the effect of non-stationarity in the environment. The environment is modelled as a time-series, with future estimates provided using prediction techniques. Learning is based on the predicted environment behaviour, with agents employing this knowledge to improve their performance in real-time. (Park et al., 2019) proposed to use a centralized auxiliary predic- tion network to model the environment dynamics to alleviate the non-stationary dynamics problem. (Krupnik et al., 2020) built a centralized multi-step generative model with a disentangled variational auto-encoder predict the environment dynamics and the opponent actions and then performed tra- jectory planning. AORPO (Zhang et al., 2021) is a Dyna-style method in which each agent builds its multi-agent environment model that consist of a dynamics model and multiple opponent models, and trains its policy with the data both generated from adaptive opponent-wise rollout and interacted from the real environment. To our knowledge, unlike previous work, our approach the first to accelerate the learning process in MARL by building only part dynamic model of the whole sys- tem. Our algorithm is also a Dyna-style method like AORPO. And it is clear that the difficulty of modeling process will be reduced significantly and the learning process is more efficient.
A.4 MEAN-FIELD MARL
B DETAILS FOR INVENTORY MANAGEMENT PROBLEM
We summarize all notations for Section 2.2 in Table 2. We also give an example with 2 SKUs in Fig. 4 to further illustrate the whole procedure for inventory dynamics.
16

---

Under review as a conference paper at ICLR 2022
08 20.4 =2
5,4, 4
Og 7,44
10,75
1 - 10- 3 12,7!
= 12-12-55 5+
[0.78 = 12
20.77
7 13,77 10
20]
step
step
DJ = 5,0j
12 11 22.
Replenishing orders
Inax 30
214+22-
Customers' demands
(7++20) 20)
Resolving storage overflow
Figure 4: A diagram to illustrate an inventory dynamics in two time steps.
Table 2: Notations.
Notation
Explanation
Units in stock of SKU i at the f-th time step
Units in stock of SKU i at the end of time step / if no discard
O
Order quantity the i-th SKU at the t-th time step
D
Demand the i-th SKU at the t-th time step Sale quantity of the i-th SKU the t-th time step
S T
Units in transit of the i-th SKU at the f-th time step
Pt
Profit generated on the i-th SKU at the t-th time step
ps
Unit sales price of i-th SKU Unit procurement cost of i-th SKU Unit order cost Unit holding cost for a time step
qi
0
h
C ALGORITHM DETAILS
Here we provide the pseudocode of our algorithm with context augmentation Algorithm 4. And details of sub-algorithms will be introduced in the following subsections. Notes that all pseudocode are assumed using RNN as networks.
C.1 GET CONTEXT DYNAMICS
Our algorithm follows the algorithmic structure of the Multi-Agent PPO (Yu et al., 2021), in which we maintain two separate networks for and Vs(s). And the first stage of our algorithm is only running policies in the origin joint environment to get episodes for context dynamics, i.c. the trajec- tories of et and cr'. This process is similar with all other traditional MARL methods. Notes that can also save the transitions of each agent for learning policy and value function network. In joint sampling(Algorithe 2), we only record the part for the context dynamics to train the context model prepared for next stage's 's training.
following pseudocode
After data collecting for context dynamics, we will use LSTM model fo to train a surrogate predic- split the collected
tor extra augmentation for collected dynamics in next stage. In details, trajectories of context into several sub-sequences of length 8 and the training objective is to mini- mize the mean-squared error of the L + day's capacity predicted given the dynamics of previous days.
c+112
(14)
17

---

Under review as a conference paper at ICLR 2022
C.2 DECENTRALIZED PPO
With the collected context dynamics of the shared resource, is easy to run the second stage: sampling the local simulators for each agent and then training the policy and critic with data. It is worth noting that the main difference between our training paradigm and traditional MARL methods under CTDE structure is that we directly sampling local observations in the extra simulators in which only one agent existed rather than the joint simulator in which all agents interacting with each other. other words, in the new local simulator, there is only one SKU in the entire store, and the trend of available capacity is completely simulated according to the given context dynamics.
In practice, we parallelly initialize new instances of the original inventory environment with the new configure settings which only contains specific SKU and a fixed trajectory of context, As for the fixed context trajectory, some noise in some items with a predefined probability:2) replace some items with predicted values comes from the trained context model also by the predefined probability. Then we the policy to interact with the local simulators to conduct episodes under the embedded context dynamics and put them into shared replay buffer since all transitions homogeneous and we shared the parameters over all policies. And decentralized training will be started by utilizing the shared replay buffer of transitions collected from local simulators.
use the subtraction results {c (c= with some augmentations: 1) add
We consider a variant of the advantage function based on decentralized learning, where each agent learns a agent-specific state based critic (s() parameterised by & using Generalized Advantage Estimation (GAE, (Schulman et al., 2016)) with discount factor 7 0.99 and 0.95. We also add an entropy regularization term to the final policy loss (Mnih et al., 2016). For each agent i, we have its advantage estimation as follows:
A
Σ I=
(15)
where 6% rt (t), + 7Vo (#1+1) Vo (st) is the TD error time step t and h is marked as steps num. And we use individual reward provided from local simulator ri(s af). So that the final policy loss for each agent becomes:
To (a) st)
L (0) = EE st st,a Tucule))
min
A
Toaa (af st)
(16)
(at
st)
clip
1 - e, +e +e)
A
[a)
As for training value function, in addition to clipping the policy updates, our method also use value clipping to restrict the update critic function for each agent to be smaller than as proposed by GAE using:
('( Egg BagnThal 9) min { (v& (st) - v/)²
(17)
(Vosc (si) + clip (Vo (si) Vonia (s)) -0, -c,,t)) -c)))) "
where Pold are old parameters before the update and V Ai + Vo (sf). The update equation restricts the update of the value function to within the trust region, and therefore helps to avoid overfitting to the most recent batch data. For each agent, the overall learning loss becomes:
C(O, () Σ (0( Aeritic L' (0) + Xentropy H (T''
(18)
obvious that all networks are trained in the decentralized way since their inputs are all local variables which stem from the light-cost local simulators. As mentioned before, at this learning
18

---

Under review as a conference paper at ICLR 2022
stage, there are interactions between any two agents. Although it seems like the way of indepen- dent learning, we need to point that we use the global context simulated from the joint environment, which is essentially different from independent learning methods since they will not consider this style glohal information which is simulated from joint simulator but be fixed in the local simulators. Our decentralized training have several advantages: firstly, the local simulator is running efficient because of its simple only-one-agent transition function; secondly, this paradigm avoid the issue for non-stationary occurred in the traditional MARL methods since there are no interaction amongst agents so that it is need to consider influences of other agents; thirdly, we can use more diverse context trajectories to make agents face various situations of the available levels of the store, which leads to improve the generalization of the networks to be trained; fourthly, it is easy for this training paradigm to be extended to large-scale distributed training by running parallel simulation whose communication cost is also acceptable for modern distributed training frameworks.
the critic and actor networks are RNNs, then the loss functions additionally sum over time, and the networks are trained via Backpropagation Through Time (BPTT). Pseudocode for local sampling with recurrent version of policy networks shown in Algorithm 3.
Algorithm GeeContextDyynmicc
INPUT policies { } and the joint simulator Env joint
(Optional) Initialize w, the parameters for context model fe data buffer D = 0
for bs = 1 to batch_size do
= [] empty list
initialize fg.,. ... hö actor RNN states
for = to 1' do
for all agents i do
pi, h, = T'' (st, Ct, h 0z)
af ~xp Pi
end for
Execute actions at, observe rt, 8t+1
I --= [St, Ct he, at, Pt, S1+1, ct+1]
end for
// Split trajectory into chunks of length L
for 10 1, T//L do
DU (e(l:++
end for end for // (Optional) Train the context model for augmentation if Need to train context model then for mini- k ., K₁ do b t random mini-bateh from D with all agent data Update capacity-context dynamics model with Eq. 14 end for Adam update w with data end if'
OUTPUT context dynamics ( ]T for i L, ....,, (Optional) fw
D LOCAL SAMPLING V.S. JOINT SAMPLING
Our algorithm follows decentralized training paradigm, which is compatible with distributed train- ing framework. We can run parallel local simulation while communication cost of context dynamics also acceptable. With distributed training, method can be much more efficient than learning from the joint simulator directly.
One may argue that urbutted/parallelizedd joint simulator. While this can indeed improve sampling efficiency in many scenarios, its usefulness is limited particularly for systems with a large number of agents. In typ-
also leam from joint simulator efficiently by implementing dis-
19

---

Under review as a conference paper at ICLR 2022
Algorithm 3 DecentralizedPPO
Generate data for agent i with corresponding context dynamic model INPUT local simulator Enviocal: policy Té and value function %
Set data buffer D = 0
Initialize , i actor RNN states
Initialize k6. "..v..... kö critic RNN states T= eeeptty list
for 1 = 1 to T do
pi, hi, n' (s) hi 11 03)
of ~pp (s Ge (i-11,i<9)
oi
Mi
Execute actions aj in Enviocats and then observe ri 441,
r+ 1s) cf oz M 1i, 1141, 4+1l
end for
Compute advantage estimate À via GAE on 7 (Eq. 15)
Compute reward-to-go R on T
// Split trajectory т into chunks of length I. in D
for = 0, 1, T//L do
D = U (
Al: 1+ L). RlI : 1+ L))
end for for mini-batch 1 1..... K₂ do
t random mini-batch from D with all agent data for each data chunk e in the mini-batch b do update RNN hidden states for n² and V from first hidden state in data chunk end for end for
Calculate the overall loss according Eq. to Eq. 18
Adam update 0: on I.' I,'(,)) and H with data b
Adam update o, on Z [xbe)) with data
OUTPUT policy nà. and value function V
Algorithm Context-aware Decentralized PPO with Context Augmentation
Given the joint simulator Envjoint and local simulators (Enviocut)] 1 Initialize policies x' and value functions V for i = 1....,
Initialize context model fw, and the augmentation probability Poug for M epochs do // Collect context dynamics via running joint simulation
1T {cr 1==0 fc, "T for t- GelContexdDynamiesseee
{ fw) (Algorithm 2)
for l = 1,2
K do
for all agents i do
// Set capacity trajectory by augmented context dynamics Enviorat-set wtt.asttett..jttorryy........... (aug( (c, "T fu.Paug)) // Train policy by running simulation in the corresponding local environment
मं V + DecentralizedPPO (Enviocate n², V (Algorithm 3)
end for end for
Evaluate policies {x H joint simulator Eny joint end for
20

---

Under review as a conference paper at ICLR 2022
ical multi-agent system, interactions occur frequently among all agents. Actually, one advantage of multi-agent systems is to model complex interactions among agents. However, such interactions will remarkably reduce the efficiency improvement brought by parallelism, as these interactions often require involved agents to synchronize which usually consumes lots of time on waiting and communicating. For instance, in IM problems, each time step, all agents should be synchronized in order to calculate in Eq. 6 before moving to the next time step.
On the other hand, in our local sampling approach we simplify such costly interactions by utilizing special structures of shared-resource stochastic games. Under our approach, there is no need for agents to be synchronized before moving to the next time step local simulators. As result, our method can be much more efficient than learning only from the joint simulator in practice.
E TRAINING DETAILS
E. THE CODEBASE
As part of this work extended the well-known EPyMARL codebase((Pppoudaki et al., ))21)) to integrated our simulator and algorithm, which already include several common-used algorithms and support more environments well as allow for more flexible tuning of the implementation details. It convenience for us to compare our algorithm with other baselines. All code for our new codebase is publicly available open-source on Anonymous GitHub under the following link: https: // anonymous 4open. sci sccienceeeecccccccc / r/repl en shmenttmaa r rarrrr-aaa basee mes--75F4
E.2 HYPERPARAMETERS DETAILS
Table 3 presents the hyperparameters used in our algorithm for 5-SKUs environment.
Hyperparameters
Value
runner
ParallelRnnee
batch size run
10
decoupled training
True
use individual envs
True
max individual envs
5
decoupled iterations
train with joint data
True
context perturbation prob
1.0
hidden dimension
64
learning rate
0.00025
reward standardisation
True
network type
FC
entropy coefficient
0.01
target update
200 (hard)
n-step
Table 3: Hyparameters used in CD-PPO
E.3 DETAILS OF STATES AND REWARDS
Table 4 shows the features of the state for our MARL agent that corresponds to the i-th SKU on the 1-th time step.
It's worthy to note that we use the profit generated on the i-th SKU at the t-th time step Pt divided by 1000000 as the individual reward of i-th agent at the t-th time step, for team reward methods, simply sum up all the individual rewards, which corresponds to the daily profit of the whole store at the 1-th time step, divided by 1000000.
21

---

Under review as a conference paper at ICLR 2022
Features
State Storage information
Storage capacity
Inventory information
Quantity of products in stock I Quantity of products in transit Ти
History information
Replenishment history the latest days [0 21:-
0
Sales history in the latest 21 days [S] 21: 21. S
Standard deviation of historical sales std (S) 21 ... (
Product information
Unit sales price pi Unit procurement cost qi
Context
Global storage utilization
Current total storage level of the store Σj=1 P
Global unloading level
Current total unloading level the store Σ == 0 b,
Global excess level
Current total excess level of the store p 2 0
t-1
Table 4: Features of the state
BASE-STOCK POLICY
Algorithm 5 Base-stock Policy
Input:
(Di) i T, IM parameters
Output:
2. (O
// Description. policy for single SKU
1/ Z::aaeesstockk level for SKU
1/(0r) :Base-stock replenishment policy for SKU i, from i3 4
(D()) :Demand series of SKU used to infer Z,, from i to l2
1/e:v 2 1, E R, Hyper-parameter to control storage utilization level
TT:T E N,. Hyper-parameter to control replenishing interval
//M parameters:incuudig leading time L, storage capacity C, etc
// Solve Mixed Integer Programming with dual simplex methods:
S = min (Di, 7;)
T7+1 T -0 0
l/ T -5: 0
Z: max 2,
Pe s.l.
t=11
Of = max (0, Zi 1 -Ti)
PO p:S a0; hT;
4 1.sss 12
// Reelenishing policy deduction:
0 max (0, 2; 13 T2) tt
Oj min (0 vC -
E4 ( +T)))
,ts St54
Oj Of1 [t mod stst
return Z,, (O
In addition to comparing with related MARL baselines, we also add well-known algorithm "Base- stock" from OR community as our non-RL baseline. The pseudo-code for base-stock policy can be found in Algorithm 5, where Z, is called the base-stock for agent and is computed by solving mixed integer programming problem. After that, Z, will be used to guide replenishment of agent i periodically. We shall note that base-stock policy not deal with complex IM variant settings
22

---

Under review as a conference paper at ICLR 2022
like coordinated replenishment (multiple SKUs with storage capacities), order cost and stochastic VLTs, etc. These complicated realistic constraints are exactly what we use to test other MARL algo- rithms. Thus may happen that Base-stock policies constantly overflow warchouse capacities when storage is tight, in which cases incoming products are abandoned proportionally as we explained in Section 2.2. This explains Base-stock's s poor performance on some of the envs.
Base-stock utilizes Linear Programming to work out the base stock levels and thus the replenishing policy. "Static" indicates that the base stock levels are based demand data from training set and then kept static while making decisions on test set. "Dynamic" updates its base stock levels on a regular time cycle. "Oracle" directly access the whole test to calculate its base stock levels, which is essentially cheating version used to show the upper limits of Base-stock policy. In practice, we conduct grid search to find proper and т. We refer readers to (Hubbs et al., 2020) for detailed description. Our implementation of Base-stock baselines is also inspired by it.
G ADDITIONAL RESULTS
G.1 THE FULL RESULTS ON ALI ENVIRONMENTS
Here we present all the experiment results in Table and display the training curves of algorithms in 50-SKUs scenarios in Figure 5 and Figure 6. We also present the results from OR methods designed to deal with IM problem, namely Base-stock, in Table 5.
Please note that, actually, VLT (i.e leading time) for each SKU N50 and N100 is stochastic during simulation. On the one hand, the specified VLTS can be modelled as exponential distributions our simulator. On the other hand, the real lead time for each procurement can be longer than specified due to lack of upstream vehicles. (Distribution capacity is also considered in the simulator, though not the focus of this paper). So that the results running on N50 and N100 are all considering stochastic VLTs.
Table 5: Profit Comparison All Environments
19007
Table 6: Average samples needed by different algorithms to reach the median performance base- lines on All Environments
Data Samples liki
Far Semmatin
IPPI-IR contear)
MARK comtect) IPPO-I
MAPRO-IE
MAPRXXwis
IPR)
MAAPP
C50
26
718.0
ESS G3
1071.1
718
1071.10
62330
TE.07
389.27
N50
2354.30
N/A
NIA
ROA
N/A
N90-C2000
2996.87
3138 JU
NIA
808.0
NIA
N/A
N/A
N130-C71000
47.14
10
N/A
NIA
N/A
100. 48
NIA
N131-550000
60.07
NIA
197.57
NIA
N/A
11517
N/A
It's worthy noting that throughout all the environments we' ve experimented on, CD-PPO enjoys higher sample efficiency shown in Table 6. Though it seems that in environment where the storage capacity is extremely tense, like N5-C50, CD-PPO performs not as well as IPPO, we reason that these tense situations favor IPPO (with team reward), which may quickly learn how adjust each agents' behavior to improve team reward. While CD-PPO, owing to its decentralized training paradigm, struggles to learn highly cooperative policy to cope with the strong resource limits while meeting stochastic customer demands. Yet we still claim that CD-PPO does outperform its IR (w/o context) baselines while producing comparable performance when compared to IR (with context) algorithms. On the other hand, IPPO(with team reward) struggles to learn good policy in 50-agents environment for the large action space compared to the single team reward signal, and fails when faced with N50-C2003 whose state space is even larger. To top it all off, CD-PPO does have its strengths in terms of sample efficiency. Though we will continue to address the above challenge in future work.
23

---

Under review as a conference paper at ICLR 2022
Proft
Proft:
10
104
163
104
107
Broft: (log SCIl
EPO-IRINO contect)
101
IPPO-R/W/ contecti
-16
FFO contect:
-10
IPPO context)
C3-990
-10
-10
-10'
-16
-104
-16-
-104
-10
-164
-10-
-107
-10'
-10f
-10
Samples
Date Sarplees
Figure 5: Training curves on N50-C500.
Figure 6: Training curves on N50-C2000
G.2 ABLATION STUDIES FOR CONTEXT AUGMENTATION
In this section, aim seek better way to augment the context dynamics in order to boost performance of our algorithm. We ponder the following two questions:
Q1: Which is better way to augment the original data, adding noise using Deep predic- tion model?
To answer this question, set out experiments environment N5-C100 with algorithm CD- PPO, using either context trajectories generated by deep LSTM prediction model, or simply adding random normal perturbation to the original dynamics trajectories, both on different seeds. As shown in Figure 7, those runs with deep prediction model generated dynamics enjoy less std and bet- ter final performance. This could result from that the diversity of deep model generated trajectories surpasses that of random perturbation.
Profit Comparison
le6
Noise
Pred
1.0
0.5
Profit
0.0
-0.5
-1.0
2
3
5
6
7
Data Samples
1e6
Figure Training curves of CD-PPO with different augmentation methods
24

---

Under review as a conference paper at ICLR 2022
1e6
Profit Comparison
1.0
0.5
Profit
0.0
-0.5
fake Dynamics prob: 0%
fake Dynamics prob: 50%
fake Dynamics prob: 100%
-1.0
2
4
5
Data Samples
1e6
Figure 8: Training curves of CD-PPO with varied ratio of augmented data
Q2: Does dynamics augmentation improve the performance of the algorithm? If so, how much should perturb the original data?
We run similar experiments on environment N5-C100 with CD-PPO, in which the local simulator ingested with mixture of original dynamics data and LSTM generated data. The ratio of per- turbed dynamics data varies from 0% to 100%. And we found that the algorithm turns out the best performance when use fully generated data, shown in Figure
25


Reviews:
[{"review_id": "wA-K20kRhpR", "reviewer": "Reviewer_XZpy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a nice paper that addresses a simplified version of a major issue faced by retailers (inventory management and capacity control) as a MARL problem. The approach identifies that in the IM problem - the shared context is only really the resource that needs to be divided between different agents. \n\nThe paper has a nice explanation of the problem - though it suffers from weak baselines (other PPO methods), rather than trying out a few different approaches (dual SGD, admm etc). The data is rich in sales/demand - but seems to be a simple simulation for other relevant parameters (VLT, price, cost etc). So that would definitely require some improvement to see if the MARL approach can generalize to those scenarios\n\nThe reward given to the agent resembles a shadow price, but has a hard truncation to respect the capacity. This isnt very realistic and could probably be improved a bit. Finally, it seems to be only for one time period? So with arbitrary stochastic VLTs, how do the authors control for the fact that order arrivals can be later then expected and then blow the capacity?\n\nFinally, why wouldn't one just treat every SKU as an instance from some joint stochastic process (with context of course) - and do a dual sgd loop at the batch level. Is the stochastic game formulation adding much more than that?\n\nOverall I weakly recommend accepting the paper as it addresses a novel way to approach the capacity control problem.", "summary_of_the_paper": "The authors propose a MARL style algorithm to solve the capacity modelling problem for the periodic review inventory management problem.", "summary_of_the_review": "The paper is a nice MARL approach to the IM Capacity Problem. I like the experiments and the modification of the public data used. The baselines are not ideal and the reward function doesn't resemble a real capacity issue - but it is a good step towards a more general IM solution", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ezC_MW_G8AT", "reviewer": "Reviewer_Uk4L", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \nA new MARL algorithm to deal with joint replenishment problem in inventory management is proposed. \n\nWeaknesses:\nThe claims of the paper about the performance of the algorithm is not well supported with numerical experiments. Also, there are some questions in the paper that need to be addressed. See below comments for more details. \n\nQ1- With the introduced shared contextual information, still, the env is non-stationary from the viewpoint of each agent. In other words, if agent $i$ is in state $s_t^i$ and takes $a_t^i$, the distribution of next state $s_{t+1}^i$ and the reward $r_t^i$ heavily depend on the actions of other agents. How do you deal with that? How can one make sure that the training is not hindered with that? In other words, can you guarantee that the training is successful for any data/demand distribution, any number of SKUs, and other characteristics of the problem? \nQ1-1 Same issue exists when you do sampling $c^{-i}^t$ and $(s^i_t; a^i_t; c^i_t)$ from the old policy. \n\n\nQ2- Since in theory, the algorithms with independent learners may not work in practice, do you have any intuition why does CD-PPO work on the presented examples? Does the data have any given feature/characteristic? \nQ2-1 In a broader question, can you categorize the type of problems that CD-PPO work well for them? \n\n\nMinor comments:\nSome of the references do not have all the required information to find the paper, like the journal, doi, etc, like:\n\nYaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multiagent reinforcement learning, 2020. A.1\n\nGeorgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks, 2021. 4.1, E.1\n\nAfshin Oroojlooyjadid, MohammadReza Nazari, Lawrence Snyder, and Martin Tak´aˇc. A deep qnetwork for the beer game: A deep reinforcement learning algorithm to solve inventory optimization problems, 2020a. 5.1\n\nChristian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?, 2020. 4.1, 5.2\n\nPlease take check all the references and update them with the latest version. \n\n\n[1] Khouja, M., & Goyal, S. (2008). A review of the joint replenishment problem literature: 1989–2005. European Journal of Operational Research, 186(1), 1-16.\n[2] Otero-Palencia, C., Amaya–Mier, R., & Yie-Pinedo, R. (2019). A stochastic joint replenishment problem considering transportation and warehouse constraints with gainsharing by Shapley Value allocation. International Journal of Production Research, 57(10), 3036-3059.\n[3] Ongkunaruk, P., Wahab, M. I. M., & Chen, Y. (2016). A genetic algorithm for a joint replenishment problem with resource and shipment constraints and defective items. International Journal of Production Economics, 175, 142-152.\n[4] Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020). Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9), 3826-3839.\n[5] OroojlooyJadid, A., & Hajinezhad, D. (2019). A review of cooperative multi-agent deep reinforcement learning. arXiv preprint arXiv:1908.03963.\n[6] Zhang, K., Yang, Z., & Başar, T. (2021). Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, 321-384.\n[7] Minner, S., & Silver, E. A. (2005). Multi-product batch replenishment strategies under stochastic demand and a joint capacity constraint. IIE Transactions, 37(5), 469-479.\n[8] Hoque, M. A. (2006). An optimal solution technique for the joint replenishment problem with storage and transport capacities and budget constraints. European journal of operational research, 175(2), 1033-1042.", "summary_of_the_paper": "A multi-agent RL algorithm is proposed to deal with the inventory management problem where there are several SKUs. Since items should compete for a limited budget and warehouse space, the replenishment problem is modeled as a multi-agent game and a multi-agent reinforcement learning algorithm is proposed to solve that. The agents use a shared piece of information which is the utilized warehouse space and other shared resources (if any) and rather than that there is no other shared information among them. So, each agent uses that shared information along with its local observation, and local reward and train decentralized agents in an independent learner scheme. The PPO algorithm is used to train the independent policies. The agents share the parameters of the actor and critic to mitigate the computation time and power. The action is selecting one of 15 possible values to multiply to the sum of sales over the last two weeks. The algorithm is evaluated on instances with 5 and 50 SKUs and compared with some benchmarks. \n", "summary_of_the_review": "Same as above \nI'll update my recommendation based on the answers of the authors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dXDWfv9nAhb", "reviewer": "Reviewer_xoVm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper tackles an important problem with impact in the real world.\n- The formulation as a stochastic game where the SKUs are the players is novel and potentially useful.\n- The approximations to make simulations more efficient are sensible. \n- Simulations show that the proposed approach is more applicable than the considered baselines.\n- The paper is well written and easy to follow.\n\nWeaknesses:\n- Although the paper tackles a real world problem, the scenarios used for the experiments are too simple; and although the authors claim that the approach could be used for more complex settings, the fact is that the authors acknowledge scalability issues. Hence, it is difficult to know if this approach will actually work in the real world. I would have expected to see the algorithm working with more realistic settings, including at least hundreds of SKUs, a multi-echelon network and ideally stochastic lead times. The study of the response under disruptions would have been especially appreciated. \n- Moreover, the baselines chosen for the experiments aren't very relevant. First, they are not well suited for the problem formulation where there are many SKUs, but they could still work for other formulations of the same inventory management problem. Second, I would have expected comparison with well known operations research baselines.\n- The authors also claim their approach could work for non-homogeneous SKUs, but it is not clear how. Would they consider the agent's ids (or features) as inputs to the neural networks?\n- Another point for improvement is the comparison with related work. For example, I think the current approach is very related to potential games and to mean-field games. Comparison with the former could bring better theoretical understanding and if the current game is potential (as I suspect), the authors could present their approach as a distributed method and compare with the potential-based centralised approach. Comparison with the latter would be very interesting as they potentially could scale the current formulation to arbitrarily large number of SKUs by relying on a mean-field formulation. \n\nMinor comments\n- Some simulation results are not clearly displayed. For example, what is the rationale for the bold font in Table 1? IPPO-IR (w/o context) performs almost as well as CD-PPO and with less variance in the N50-C2000 scenario. Also, why comparing with the worst baseline and not the median or the second best profit in Figure 7?\n- Ref to A2C is not Tsitsiklis & Konda, 2000, but an post in the OpenAI blog that suggest a synchronous version of A3C.", "summary_of_the_paper": "The authors formulate an inventory optimisation problem as a stochastic game, where each SKU is a player. In order to reduce computational complexity, the game is formulated such that the only interactions among players are through the available storage space they have to share. The paper also proposes an algorithm that relies on local simulators (as opposed to joint simulation) and surrogate losses to improve efficiency. Simulations show that the proposed algorithm is able to outperform other MARL baselines for up to 50 SKUs.", "summary_of_the_review": "The introduces a potentially useful approach for an important real-world problem. But the benefit of tackling a real-world problem vanishes because the proposed approach has been evaluated in very simple environments and with baselines that are not very relevant for this problem. The paper would have higher impact with some of these improvements:\n- The algorithm is evaluated in a realistic environment.\n- The algorithm is compared with relevant IM baselines (operation research approaches actually used in the real life) even in this toy scenarios.\nAlso, the paper misses two important lines of related work, namely potential and mean-field games, which could offer new insights to the  proposed approach.\n\n\nMeta-review\n=========\nI think the paper tackles an important real world problem with an interesting approach and, although the experiments were a bit lacking, the authors have included a sensible baseline and a more realistic scenario.\n\nI still see some scalability issues and I would appreciate a deeper and insightful discussion with related approaches, like potential and mean-field games. But I think the paper is in better shape than before and it might be useful for other researchers in the field.\n\nSo I am happy to increase my score to 6 if the other reviews don't see major flaws.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A5jScIYo45D", "reviewer": "Reviewer_yMmz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper studies a very interesting and promising industrial application of RL. Improvement in inventory replenishment has profound real-world impact.\n- A model-based MARL algorithm is proposed to tackle the challenge of large number of agents by exploiting the structure of the multi-SKU replenishment problem, that is, the replenishment of the SKUs are associated only through the global capacity constraint. \n- Empirical results appear to support that the proposed method achieves greater learning efficiency than the model-free baselines.\n\nWeaknesses\n- It's not clear how exploration is done in the proposed method. In order for the individual SKUs to train and update their policies independently, the global simulator has to run to obtain a sample path of the shared resource state (context) in each epoch. This basically allocates the capacity for each SKU using the previous policy, and each agent only learns within its allocated capacity trajectory. The inner loop of training is very restrictive in this way given that capacity constraint is a core element in inventory replenishment. How does the joint policy explore different capacity allocations?\n- The objective function does not incorporate any penalty for inventory overflow, which is a serious issue in practice.\n- The surrogate loss function is based on the assumption that each SKU has limited impact to its context (shared resource). When the number of SKUs is small, this would not hold. In the experiments, did you observe any difference in performance over the 5-SKU and 50-SKU cases? \n- The paper should add at least one classical non-RL replenishment algorithm as baseline. Otherwise, it is hard to judge the significance of the contribution.\nMoreover, mean-field MARL [1] is designed for large number of agents. The idea of the context in this method is similar in principle to MF-MARL. \n[1] Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W. and Wang, J., 2018, July. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 5571-5580). PMLR.\n- The results compare the number of environment interactions. I assume this is the interaction with the global environment. In the proposed method, each agent further interacts with its local environment for a number of steps, while the model-free methods do not. Hence, the comparison may not be entirely fair.", "summary_of_the_paper": "This paper considers the multi-period inventory replenishment problem for a single store with multiple SKUs. The problem is formulated as a resource constrained stochastic game, where the store capacity is the resource constraint shared by all SKUs. The paper takes a multi-agent RL approach to solve this problem, with each SKU being modeled as an agent and having its own replenishment policy. To tackle the challenge presented by the potentially large number of SKUs (agents), a context-aware decentralized PPO method is proposed, where each agent is trained independently in its local simulator given a sample path of the shared resource state (context). Benchmarking were performed on two domains with 5 SKUs and 50 SKUs each against several model-free MARL baselines. Results show that the proposed method can achieve comparable results but with much higher sample efficiency.", "summary_of_the_review": "This paper presents some novelty in addressing the issue of large number of SKUs (agents) in applying MARL to solve the multi-SKU inventory replenishment problem, by exploiting the structure of the problem and devising a model-based approach. The empirical results show that the proposed method achieves better sample efficiency in the global environment than the model-free baselines. However, there are several important algorithmic issues not investigated in this paper, and the important related method in the literature is not compared with.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a pretty nice paper, but it suffers a bit from being in an 'uncanny valley' between application and research.  The approach clearly has been made, and derives from, the application under consideration.  However, the application is not a real application, and rather is a simplified simulation. That's okay, but it means that the application here is not the real goal.  So, our attention should go to the solution technique.  Unfortunately, this seems rather specific, exploiting known structure for the specific problem at hand, and lacking other reasonable baselines one could imagine.

So, this is not really an application paper, as the application in the paper is a proxy.  But this is also not really an algorithm paper, as the algorithm is not clearly shown to be generalisable to other settings.  And this is also not a theory paper that tells us something general and meaningful.

These are just observations - this is not criticism per se.  But it means I struggle a little to find something meaningful to learn from this paper, that could be applied elsewhere.

This, in addition to the overall recommendations by the reviewers, unfortunately lead me to reject the paper in its current form.  I want to thank the authors for engaging with the discussion, and hope they have found it interesting and rewarding, despite the outcome this time around.
Decision: Reject