Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management
Authors: Yuandong Ding
Mingxiao Feng
Guozi Liu
Wei Jiang
Chuheng Zhang
Li Zhao
Lei Song
Houqiang Li
Yan Jin
Jiang Bian
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Agent Reinforcement Learning
Inventory Mangement

Abstract: In this paper, we consider the inventory management (IM) problem where we need to make replenishment decisions for a large number of stock keeping units (SKUs) to balance their supply and demand. In our setting, the constraint on the shared resources (such as the inventory capacity) couples the otherwise independent control for each SKU. We formulate the problem with this structure as Shared-Resource Stochastic Game (SRSG) and propose an efficient algorithm called Context-aware Decentralized PPO (CD-PPO). Through extensive experiments, we demonstrate that CD-PPO can accelerate the learning procedure compared with standard MARL algorithms.

MULTI-AGENT REINFORCEMENT LEARNING WITH SHARED RESOURCES FOR INVENTORY MANAGEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we consider the inventory management (IM) problem where we need make replenishment decisions for a large number of stock keeping units (SKUs) to balance their supply and demand. In our setting, the constraint on the shared resources (such as the inventory capacity) couples the otherwise in- dependent control for each SKU. We formulate the problem with this structure as Shared-Resoune Stochastic Game (SRSG) and propose an efficient algorithm called Context-aware Decentralized PPO (CD-PPO). Through extensive experi- ments, we demonstrate that CD-PPO can accelerate the learning procedure and achieve better performance compared with standard MARL algorithms.
INTRODUCTION
The inventory management (IM) problem has long been one of the most important application sce- narios in the supply-chain industry (Nahmias & Smith 1993). Its main purpose is to maintain a balance between the supply and demand of stock keeping units (SKUs) in supply chain by op- timizing the replenishment decisions for each SKU. An efficient inventory management strategy cannot only increase the profit and reduce the operational cost but also give rise better services to help maintain customer satisfaction (Eckert (007). Nevertheless, this task is quite challenging in practice due the fact that the replenishment decisions for different SKUs compete for shared re- sources (e.g., the inventory capacity or the procurement budget) as well as cooperate with each other to achieve a high total profit. This becomes more challenging when the number SKUs involved in the supply-chain becomes larger. Such co-existence of cooperation and competition renders IM complicated and challenging problem.
Traditional methods usually reduce the IM problem to dynamic programming (DP). However, these approaches often rely on unrealistic assumptions such as iid customer demands and deterministic leading time (Kaplan 1970; Ehrhardt 1984). Moreover, when the state space grows rapidly along with the scaling-up of some key factors such as the leading time and the number of SKUs, the problem becomes intractable by DP due to the curse of dimension (Gijsbrechts et al.) 2019). Due to these limitations, many approaches based on approximate dynamic programming are proposed to solve IM problems in different settings (Halman et al. 2009 Fang al. 2013 Chen & Yang 2019). While these approaches perform well certain scenarios, they rely heavily on problem-specific expertise assumptions, e.g., the zero one period leading time assumption in (Halman et al., 2009), and thus can hardly generalize to other settings. In contrast, reinforcement learning (RL) based methods, with short inference time, can be generalized into various scenarios in data-driven manner. However, it is hard to train a global policy that makes decisions for all SKUs due to the large global state and action space (Jiagg Agarwal, 2018). To further address the training efficiency issue, it is natural to adopt the multi-agent reinforcement learning (MARL) paradigm, where each SKU is controlled by an individual agent whose state and action spaces are localized and only contain information relevant to itself.
There are currently two popular paradigms to train MARL in the literature: independent learning Tan 1993) and joint action learning (Lowe et al. 2017). Despite of their success in many scenarios, these two MARL paradigms also exhibit certain weaknesses that restrain their effectiveness in solv- ing the IM problem. For independent learning, the policy training of one agent treats all other agents as a part of the stochastic environment. may largely increase the hardness training convergence due to the non-stationarity of the environment. For joint action learning, centralized critic is

---

ally trained to predict the value based on the global state (of all SKUs) and the joint action, which can easily become intractable with a growing number of SKUs. Furthermore, it is time- consuming to sample data from joint simulator for large number of SKUs due to the high computational cost to calculate the internal variables that model the complex agent interactions.
To address these challenges, we leverage the structure in the IM problem design more effective MARL paradigm. Particularly, each agent in system interacts with the others only through the competence of shared resources such as the inventory capacity. We introduce an auxiliary variable called context to represent the shared resources (e.g., the available inventory level for all SKUs). From the MARL perspective, the dynamics of the context actually reflect the collective behaviors of all the agents. Conditioned on the context dynamics, we assume the transition dynamics and the reward function of the agents are independent. input for the policy or the value network of each agent enables us to not only prevent the non- stationarity in independent learning but also feed global information to the critic without leading to an intractable global critic.
this way, leveraging the context as an additional
Based on the context, we propose Resred-- Stochastic Game (SRSG) to model the IM prob- lem. Since the context and the policies of each agent depend on each other, it is hard to solve for them simultaneously. Accordingly, we make two assumptions to circumvent this issue: 1) rearranging the sampling process by first sampling the contexts and then sampling local the state/action/rewaar for each agent; 2) using context dynamics sampled by previous policies. With these assumptions, we design efficient algorithm called Context-aware Decentralized PPO (CD-PPO) that consists of two iterative learning procedures: I) obtaining the context samples from joint simulator, and 2) updating the policy for each agent by the data sampled from its corresponding local simulator condi- tioned on the collective context dynamics. By decoupling each agent from the others with a separate local simulation, our method can greatly reduce the model complexity and accelerate the learning procedure. At last, we conduct extensive experiments and the results validate the effectiveness of our method. Besides, not limited to the IM problem considered in this paper, our method may be applied to other applications with shared resources such portfolio management (Ye et al. 2020) where different stocks share the same capital pool and smart grid scheduling (Remani et al. 2019) where different nodes share total budget.
Our contributions are summarized as follows:
We propose Shared-Resouue Stochastic Game to capture the problem structure in the IM problem where agents interact with each other through competing for shared resources.
We propose a novel algorithm called Context-aware Decentralized PPO that leverages the shared-resource structure to solve the IM problem efficiently.
We conduct extensive experiments to demonstrate that our method can achieve the perfor- mance on par with state-of-the-art MARI. algorithms while being more computationally and sample efficient.
2 BACKGROUND
2.1 STOCHASTIC GAMES
(also known as Markov game). A stochastic game is defined as tuple (N, A, T R, 7) where N [n]] denotes the set of n agents, S is the state space, A := A x x x A" the action space composed of the action spaces of individual agents, T S S x A + (SS) is the transition dynamics, R = n Rt is the total reward which is the summation of individual rewards R' : S x A' x S - IR and E [o. 1) is the discount factor. For the i-th agent, we denote its policy as T² S + A (A*) and the joint policy of the other agents as ग IIje[n X.. Each agent optimizes its policy conditioned on the policies of the others, i.e.,
We build our work on the formulation of the stochastic game (SG) (Shapley
(953)
max ma , -ー)- E
(1)
0=0
2

---

where rf is the individual reward obtained by following the joint policy composed of I' and n- We will later propose Resouure Stochastic Game (SRSG) which is a special case of stochastic game.
2.2 INVENTORY MANAGEMENT WITH SHARED RESOURCE
In this paper, we consider simplified setting with one store and multiple SKUs to better focus on the shared-resource structure of the problem. We further assume that there is an upstream warchouse that fulfill all the requirements from the store. Our objective is to train a high-quality replen- ishing policy for each SKU in the store, particularly when there are a large number of SKUs. It is worthwhile to mention the following two points: First, this setting is more challenging than manag- ing the SKUs in warehouses due to the direct impact of the changing customer demands. Second, due the flexibility of RL algorithms, our method can also be applied to more complex settings, e.g., with multi-echelon structure or fluctuate supply.
Similar previous work, follow the multi-agent RL (MARL) framework with decentralized agents, each of which manages the inventory of one SKU in the store. We assume that the store has n SKUs in sell, all of which share a common inventory capacity that can store up max units at the same time.
Replenishing decisions of each SKU are made on discretized time steps (which are days in this paper). For the f-th time step and the i-th SKU, we denote the in-stock quantity by I E This quantity is constrained by the limit of the inventory capacity (i.c., the shared resources). Accord- ingly, the following constraint holds:
Vt 2 O, Lmax.
The dynamics of the system is as follows: For the 1-th time step and the i-th SKU, the correspond- ing agent can place a replenishment order to request Oj E Z units of products from its upstream warchouse. It takes several time steps (referred to as the leading time Lj) before these products are delivered to the store. We denote the total units in transit at the t-th time step by Ti E Z and the total units arrived at the 1-th time step by A& Z. Meanwhile, the customer demand is Dj and leads to an actual sale of S E Z units. Due to the possibility of out-of-stock, S may be smaller than D; We consider the setting where the leading time L; and the demand Dj are stochastic.
Since all the SKUs share a common inventory space, the storage may overflow when the previously ordered products arrive. In this paper, we assume that the excess SKUs will be discarded proportion- ally according to coefficient p which is the overflowing ratio if we accept all the arriving products. To calculate p. we introduce an extra variable I; which is the afterstate of i; Intuitively, it denotes how many units of the i-th SKU in stock at the end of the t-th time step if we omit the capacity constraint. Note that, although other ways to deal with overflow (e.g., prioritizing some SKUs) are possible, our algorithm will also apply to these settings and currently we only consider this naive way. Undoubtedly, overflow will lead to extra operational cost which should be avoided much as possible in the replenishing strategy.
For clearness, we summarize the dynamics these variables follows:
S = min (Di,
4
Σ o
T:T+
T = Tς Af+1 + O²11
f
1 St 1
P: max Σ 7
I
max; 0
/
i
(3)
f 1 S
+
[11
(4)
The immediate profit of the i-th SKU is calculated according to the following equation:
F p g of [0 0] -hig
(5)
3

---

where p; and 9; the unit sale price and the unit procurement price for the i-th SKU respectively, o and h are the order cost and the unit holding cost respectively, and [[]] is an indicator function which equals to one when the condition is true and zero otherwise. The order cost reflects the fixed transportation cost or the order processing cost, and yields whenever the order quantity is non-zero. See the further description for the problem setting in Appendix A
3 METHODOLOGY
In this section, we will first introduce the which decouples the control of individual agents in the multi-agent problem with shared resources by introducing context. Next, we introduce the dynamics of the context and how to build an efficient local simulator that receives the context trajectories as the input for the IM problem considered in our paper. To accelerate the sampling process in estimating and optimizing the objective function, we further introduce two approximations which leads practical algorithm called Context-aware Decentralized PPO (CD-PPO).
Stochastic Game (SRSG) formulation
3.1 RESOURE STOCHASTIC GAME
In this section, we formulate the multi-agent problem with shared resources as Shared- Resource Stochastic Game (SRSG) where each agent is influenced by the others only through a shared resource limit. We define the shared-resource stochastic game as tuple (N, TEE (N,( C,A,T,
where the definitions N, and y follow the definitions in the
stochastic game introduced Section 2.1 We use S to denote the local state space of the i-th agent, and C to denote the context space observed by all agents. For example, the context can be used to represent the occupied capacity of shared resources (e.g., the inventory space). Denote S = x...xxxx Sth and A == AA x...xx x A". The transition dynamics T 8 x C A ASS C) can be decomposed as the context transition dynamics c++1 ~ Po(- Ct, ax) and the local state transition dynamics s|+1
P. st al, Ct+1) where Ct E s St, St S, of A', and ag A. The reward function rt El= ri is additive and r ~ R (s), c, az). The objective for the ((4") to maximize the objective function that shares I
i-th agent is to learn a policy n² S x the same form as that defined in equation]
Given the above definition, IM problem can be formulated as SRSG by letting rî = F2g of Of and e Σ I for all E [n] and time step Moreover, the state of the i-th SKU is concatenation of information such as the in-transit quantity Tý, the sale price ps, its historical actions and the customer demands (see details in Appendix C
3.2 CONTEXT DYNAMICS AND LOCAL SIMULATOR.
We use è to represent the amount of resources occupied by the i-th agent at the (-th time step and & to represent the total amount of occupied resources. We further use o7 to denote the amount of resource occupied by all agents except i. Assuming that the resources are additive, we have the following equations:
④
& a
(6)
Similarly, we denote the afterstate of éi as éj. i.e., the amount of occupied resources at the end of the t-th time step. From the perspective of the i-th agent, the context é can be regarded as a part of the environment. Hence, P(č) s at, 2)) represents how the replenishment decision influences éj. Similarly, have & é: + à-t Then, by applying resource overflow resolution procedure defined in equation 3 and equation we can obtain é.. (the resources in the next step).
We use the notation e to represent (ê,-1,22). 22). We refer (s',c²) as the state for the i-th agent, and c² as the context. For each agent and given the context trajectory (c7))20 the sampling process
4

---

GetContextDynamic
Sampled
ynt
method can (c Jiso
Enrlecaa
P' tu 1i 4.47.
Sampled Data
DecentralizedPPO
Figure 1: Our algorithm consists of two iterative learning procedures: 1) Collect the context trajec-
tory {c,')T It=o from the joint simulator with previous policies Tha- Told for each [n]. 2) Train each policy I' with samples collected from the local simulator conditioned context trajectory (c
(denoted as T') is as follows:
441 Pol. s of G C.),
1/4 Pil. s Cyy o (411
r Rt (s), az, c241 G71))
Accordingly, we can build a local simulator for each agent according to the above equations. Note that the local simulator is computationally more efficient than the joint simulator that simulates the presentation, we only consider one kind of
process of SKUs simultaneously. To simplify shared resource (i.e., the inventory space) in the above formulation. However, can be extended to support multiple resources as long as these resources are additive.
For the tractability and the computational efficiency of the sampling process for evaluating the ob- jective function defined in equation T we make two approximations to the sampling process.
First, assume that each agent has infinitesimal influence the context c This is reasonable for the Shared-resourge setting where the resource consumption of one agent does not significantly influence that of the others. Second, assume the policy changes slowly during the learning pro- cess. Therefore, instead of repeatedly using the current policies to collect the context trajectories, we allow use the context trajectories collected by the policies in the previous iterations. Accord- ingly, we can run the joint simulator with the current policies infrequently to collect trajectories {c, (c,')=20 3720 for different agents indexed by Then, we can evaluate the local rewards with the local simulator under different possible local policies for each agent. In this way, we can greatly reduce the frequency to simulate using the joint simulator which time-consuming.
3.3 ALGORITHM
In the subsection, we will present details of our proposed algorithm called Context-aware Decen- tralized PPO (CD-PPO). We call it context-aware because we feed the context c, as the input of the policy and the value function of each agent. Such a context-aware approach can avoid the non- stationary problem in independent learning methods (which do not consider the policy change of the other agents during training) since the context reflects the collective behaviors of the other agents which can in turn impact the dynamics of each individual agent. In the meanwhile, mitigate the centralized critic (i.c., the critic that receives (st, az, Ct) as the input) in many centralized training and decentralized execution (CTDE) methods which becomes intractable when the number of agents increases and the joint state and action space explodes.
We show algorithm in Algorithm and Figure [ which consists of two iterative learning proce- dures: 1) obtaining the context trajectories by running all agents in the joint environment; 2) updating the policy of each agent using the data sampled from its local simulator conditioned the context

---

Algorithm Context-aware Decentralized PPO
Given the joint simulator Envjoint
Given the local simulators {Enviceal ii= for each agent
Initialize the policy n¹ and the value function V for each agent L,....
for m 1, AM epochs do
// Collect context trajectories from joint simulation
{cr 11 0: fc, o GelContextDynamiiess Envioint: { (T7)) 1im1)
for k = 1, 2. K do
for all 1,2, ......... agents do // Set the context trajectory
Enviogal Sinv set_context____eeeeeeeeeees___e_ wtw.coonttttajeeetii context_traeectory((( trajectory(fc ((
// Train policy by simulating in the corresponding local environment n², V*)
r.. V f DecentralizeePPP(EEng
end for end for end for
1c6
Profit Comparison
Samplex
perfarmensee100)
1.5
P40 18
1.0
pmoww content)
1000
800
Proft
800
IPPO IR/wlo context!
-1.0
MAFRO
IPPO IR
200
-1.5
0.0
0.2
0.4
0.6
0.8
1.2
1.4
N100-C1000
N100-C4000
Data Samples
ln7
Env_Scenard
Figure 2: Training curves of different algo- rithms in the setting with 5 SKUs. "IR" indi- cates that we train the agents using only in- dividual rewards and "w/o context" indicates that we do not feed the global information related the shared resources to the agent. The x-axis takes the samples from all local simulators into account for CD-PPO².
Figure 3: The average number of samples needed by different algorithms to reach the median performance across all baselines the setting with 100 SKUs. The lower the value, the higher the sample-efficiency for the algorithm. "N" and "C" denote the num- ber of SKUs and the maximum capacity of shared resource respectively.
trajectories. Moreover, our algorithm follows decentralized training paradigm with infrequent communication through the context trajectories. Due to space limit, we refer readers to Appendix B for detailed description of the algorithm (including the description of the subroutines).
is worth noting that a naive approach is to optimize policies based on the samples collected from the joint simulator. Nonetheless, one step in the joint simulator than simulating for one step in each of the n local simulators due to the complex agent interaction. Such phenomena become more significant when there are more types of shared resources or more complex constraints to consider (e.g., when we consider smarter discarding strategy to deal the overflow instead of discarding excess units for each SKU according to P:). In our experiments, the simulator are using is developed for the general purpose. It contains extra details such as replenishing orders fulfillment scheduling and transportation management, and thus requires large computational costs to simulate the IM problem with more SKUs. Our method, with the advantage of leveraging local simulators to accelerate simulation, is more computationally efficient.
find that takes more computational costs to simulate for

---

4 EXPERIMENTS
We evaluate the performance of CD-PPO in several inventory management problem instances with different scales (i.e., with 5, 50, 100, and 1000 SKUs) and different resources limits (by changing the limit of the inventory space). We demonstrate that, our algorithm can achieve better results and is significantly more sample-efficient than the previous SOTA baselines.
Setups. Our experiment is conducted on a simulator that can simulate the replenishment procedure for multiple SKUs in one store. Instead of sampling demands from some hypothetical distributions, we replay the demand series from two authentic datasets: a dataset from Kaggle (Makridakis et al. 2020) that contains the sales history of five years (2011-2015 with 155 randomly selected SKUs and a dataset from a retail partner that contains the sales history of two years (2018-2019) with 1000 SKUs. The former dataset is used to support the experiment settings with 5, 50, and 100 SKUs and the latter dataset is used for the setting with 1000 SKUs. We use the sales of the last one year (the last 100 days) as the testing set for the former dataset (for the latter dataset) and the remaining data as the training set. For other necessary information (such as the prices, the costs, and the leading time) to instantiate our simulator but not included in the dataset, we first select a reasonable range for each of the parameters and then sample from the range. We use the total profit in dollars as the evaluation metric. We run all the algorithms with four different random seeds and present the average performance with standard deviations.
Baselines. The baseline algorithms used in our experiment is based on the EPYMARL (Papoudakis) et al. 2021) framework where commonly used MARI. algorithms are implemented. In our prelim- inary experiments, we find that two model-free on on-policy MARL algorithms IPPO (de Witt et al.) 2020) and MAPPO (Yu et al., (021) outperform the other algorithms including COMA (Foerster et al., 2018), QMIX (Rashid et al.) 2018), etc. MAPPO learns a centralized critic and therefore becomes intractable due to the large state-action space when the number of SKUs is greater equal to 50. Due to the bad performance of the other algorithms (i.e., mostly resulting in negative profits), only show the experiment results of different variants of TPPO (and MAPPO only for the setting with 5 SKUs). We also compare with two operation research (OR) methods: the base-stock policy (Kaplan, 1970) and the (s, S)-policy (Ehrhardt 1984) which are two well-known OR strategies and are widely adopted in practice (see e.g.. Kapuscinikk & Tayur 1999; Hubbs et al. (2020). Previous studies make great efforts to determine the parameters in these strategies. We will present their performance with the optimal parameters for each SKU obtained based on the future information, which can be regarded as the skyline. (See details in Appendix D)
Implementation. We share the parameters of the policy network and the value network across dif- ferent agents/SKUs. The policy/value network consists of a two-layer MI.P with 64 neurons on each hidden layer. In our experiment, the poligy network maps the local state to categorical distribution 4
5 3, 4, 5, 6, 7, 9, 12] where the real replenishment quantity is obtained by multiplying these numbers with the average daily sales of the SKU over the past two weeks. By default, the RL. algorithms use the individual reward rather than the team reward (e.g., the summation of the individual rewards from all the agents) the objective of each agent. We also feed global information related to the shared resources to the agents. In CD-PPO, to encourage the policy adapt for stochastic context dynamics in practice, we adopt the following techniques in the training phase to augment the collected context trajectories: 1) adding noise to randomly cho- sen items from context trajectories, or 2) replacing randomly chosen items with predicted values generated by context generating model. (See details on the implementation in Appendix C
over a discrete action space {0, - 2. 1, 3' 3. 2, 2:
Results. For the scenario with 5 SKUs, we show the training curves in Figure 2 We can observe that while CD-PPO converges to the same performance level as the other algorithmss in this simple setting, it is more sample-efficient due to its efficient learning with local simulators.
To evaluate our algorithm on larger scenarios, we run CD-PPO and IPPO on the settings with 50, 100, and 1000 SKUs. To evaluate sample efficiency, we also record the number of samples when different algorithms reach the median performance level across all baselines. We summarize the results in Table I and Figure The experiment results indicate that CD-PPO outperforms other
SSpecifically, for one interaction in the global environment with N agents, we consider it as N samples. For one interaction in the local simulator (based on context trajectories) with one agent, we consider it 1 data sample. We empirically observe that the running time is roughly proportional to this statistics.
7

---

Table 1: Comparison of the profits from different algorithms on the settings with different number of SKUs (indicated by "N") and different limits of the inventory space (indicated by "C"). "IR" indicates training with individual reward and "NC" indicates feeding the global information related to the shared resources to the agent, Base-stock and (s, --policy are the skylines.
NS0C500
N50C2k
N100CIK
N100C4K
N1000C30K
N1000C200K
IPPO-IR-NC
235+611
689::49
-2107+3155
-2223+2536
88#87
571=102
IPPO-IR
250+558
546+:460
-1126=+400
148+1017
120=71
366+213
IPPO-NC
164.1143
-1373.1870
1064
-6501-16234
11966111871
-2495.1.8874
IPPO
367-1900
1110311116
1396
-6019119056
-12
-7:199
CD-PPO
311-176
695
660-1:100
1298 125
233-1:19 197
741-:157
Basc-stock
688
919
1189
1887
73
322
(s.S)-policy
704
941
1227
1981
621
1210
RL baselines on 5 out of the 6 settings and even beats the performance of the base-stock policy skyline. (Recall that search the best parameters for each SKU using the future information in these skylines.) The superior performance of CD-PPO may due to the avoidance of non-stationarity by introducing the context. In contrast, traditional MARL methods with the CTDE (centralized training and decentralized execution) paradigm cannot scale up to the environment with even 50- 100 SKUs due the fact that the inputs of the centralized critic is too large to be loaded in to the memory. IPPO, on the other hand, can run successfully but underperforms CD-PPO. More importantly, we observe that CD-PPO is more sample efficient than IPPO. Notice that, since there is no need for the agents to be synchronized before the next time running the joint simulation, the training for individual agents can be parallelized to further reduce the wall time. The full results and more ablation studies on how the capacity of shared resource affects the performance of CD-PPO and the influence of the context augmentation technique can be found in Appendix
RELATED WORK
In this section, we will introduce the related work on the inventory management problem and the training paradigms in MARL.
Inventory management.
The inventory management problem has a long history and the classi-
cal methods for single-cchelon systems include the economic order quantity (EOQ) model (Harris) 1934). the (R, Q)-policy (Chen) 2000), the base-stock policy (Clark & Scarf 1960; Kaplan 1970). the (s, -)-pollcyyy (Ehrhardi) 1984), and other continuous/periodie review methods (cf. Axsäter 2015). The base-stock policy is a special case of the (s, ()policy and we provide a detailed intoduction for these two methods in Appendix D The (s. -ppolicy and the (R. Q)-policy are proved to be optimal under different assumptions (see)Chen) 2000). However, these assumptions do not hold in many practical scenarios and researchers start to solve the inventory management problem using the approximate form (Halman et al. 2009 Fang et al. 2013 Chen & Yang 2019) of dynamic programming (DP) instead of the exact form (Huh et al.) [200);
1913 Wilson
et al. 2016). Us-
ing the DP solution based posterior information as the label to train an end-to-end model achieves superior performance in real industrial scenarios (Qi et al. 2020).
To better exploit the patterns behind big data, reinforcement learning becomes a popular choice for solving the inventory management problem in data-driven manner (see e.g., Giannoccaro & Pontrandolfo 2002. Jiang & Sheng, 2009 Kara & Dogan 2018 Barat al., 2019 Gijsbrechts et 2019 Droojlooyjadio et al.) 2017 2020). However, as their main focus is to deal with chal- lenges such as volatile customer demands and bullwhip effects in IM, they are restricted to sim- plified scenarios with only a single SKU. While these approaches are able to outperform classical methods these scenarios, they overlook the system constraints and the coordination between the SKUs constrained by shared resources. Exceptions are two recent papers (Barat et al.) [019:] Sultana ct al. 2020) where more realistic scenarios containing multiple SKUs are considered. In Barat et al. 22019). the main contribution is to propose framework to support efficient deployment of RL algo- rithms in real systems. As an example, the authors introduce a centralized algorithm for solving the
IM problem. In contrast, dccentralized algorithm is proposed in Sultana et al. (2020) to solve the IM problem with multiple SKUs in a multi-echelon setting. These two papers use the off-the-shelf RL. algorithm (i.e., A2C/Wu et al. 2018) to solve for the policy.
8

---

Training paradigms in MARL. From the perspective of the training paradigms in MARL, the algorithms can be divided into three categories: centralized training methods, centralized training and decentralized execution (CTDE) methods, and independent learning methods. The centralized training methods adopt single agent RL. algorithms directly to the multi-agent setting (see e.g. Gupta et al., 2017). However, these methods can only learn centralized policy which is not suitable for scenarios that call for policies to be executed in a decentralized way. Accordingly, the CTDE methods are more preferable and they can be further divided into 1) value-based CTDE methods (aka value decomposition methods) that centrally learn a value function that can be reasonably decoupled into decentralized ones such as VDN (Sunehag et al.) 2017), QMIX (Rashid et al., 2018). Weighted QMIX (Rashid et al.) 2020). and QTRAN (Son et al. 2019: and 2) policy-based CTDE methods that also relies on a centralized critic such as COMA (Foerster et al. 2018), MADDPG (Lowe et al., 2017), and MAPPO (Yu et al. 2021). However, for the inventory management problem considered in our paper, there are a large number of agents in the system in which the centralized critic becomes intractable due to the large joint state space. Therefore, only independent learning methods are applicable to our setting. A representative independent learning MARL algorithm is IPPO (de Witt et al. 2020) where each agent learns critic and policy (with parameters shared with others) based on the local observations to predict values and make individual decisions respectively. However, these methods easily suffer from the non-stationarity problem since each agent has to learn in an environment that changes across the training process without the information on how the policies of the others change. In this paper, we propose Context-aware Decentralized PPO (CD-PPO) which follows the independent learning paradigm (i.e., without the need to resort to the joint state/action space) but avoids the non- -tationttnn problem by feeding a context variable to the agent.
Weakly-coupled MDP is first proposed by (Meuleau et al. 1998) to model the scenario where the constraints of the shared global or local resources couple the otherwise independent sub-processes. However, their solution is closely related to the air campaign planning problem and can hardly generalize to other applications. Our setting is similar to weakly-coupled MDP with shared local resources where the instant (instead of the cumulative) inventory level cannot exceed certain limit, but they still differ since the shared local resources directly result in a feasible action set in their setting which is easier solve. The subsequent work reakle-coupled MDP solves the problem with classical optimization methods such as approximate linear program or Lagrangian relaxation (see e.g.. Adelman & Mersereau 2008 Ye et al.) 2017 Brown & Zhang 2022), whereas our algo- rithm is a practical deep reinforcement learning based method for the multi- setting with limits
on shared resources. Mean-field MARL (Yang et al. 2018, Carmona et al. 2019 Chen et al.) 2021
is derived from the seminal work on mean-field field games (Lasry & Lions 2007 Guéant et al.
2011)
to model the scenario where there exists large number of
agents
and the effect of each agent on the system is infinitesimal. This setting is different from ours where the optimal policy for each agent different and each agent learns based on local context variable cr" instead of a global mean-field variable.
6 CONCLUSION
In this paper, we consider the inventory management problem for a single store with a large number of SKUs. In our setting, the control for different SKUs interact indirectly with each other through a constraint on the limit of the shared resources. We first modeled the problem as Stochastic Game (SRSG) to leverage the shared-resource structure of the problem. Starting from SRSG, we proposed Context-aware Decentralized PPO (CD-PPO) where each agent can be trained independently in a local simulator conditioned on the context trajectories that screen the interac- tions from other agents. Our experiment results indicated that CD-PPO outperforms state-of-the --tth-e-tt MARI algorithms while being more computationally sample efficient. It is worth mentioning that our method not only limited to the inventory management problem but also has the potential to apply to other real-worl, multi-agent applications with similar shared-resouree structure.
In real-world applications, we usually need to deal with thousands of agents, which poses a challenge for existing MARL algorithms. To address the challenge, we need to develop efficient algorithms that enables effective (e.g., being stable during the training) and efficient (e.g., with a distributed training paradigm) training. In this paper, took first step towards developing efficient and practical MARL algorithms for real-world applications with the thared-resouree structure, and we will continue address the challenges arisen in real-world applications in our future work.

---

REFERENCES
Daniel Adelman and Adam J Mersereau, Relaxations of weakly coupled stochastic dynamic pro- grams. Operations Research, 56(3):712-727, 2008.
|
Sven Axsäter. Inventory control, volume 225. Springer, 2015. 5
Souvik Barat, Harshad Khadilkar, Hardik Meisheri, Vinay Kulkarni, Vinita Baniwal, Prashant Ku- mar, and Monika Gajrani. Actor based simulation for closed loop control of supply chain ing reinforcement learning. In Proceedings of the 18th international conference on autonomous agents and multiagent systems, pp. 1804,1804, 2019.
David Brown and Jingwei Zhang. On the strength of relaxations of weakly coupled stochastic dynamic programs. Operations Research, 2022.
René Carmona, Mathieu Laurière, and Zongjun Tan. Model-free mean- field reinforcement learning: mean-field mdp and mean-field q g-learning arXiv preprint arXiv: 1280, 2019. [
Fangruo Chen. Optimal policies for multi-echelon inventory problems with batch ordering. Opera- fions research, 48(3):376 2000. 5
Minshuo Chen, Yan Li, Ethan Wang, Zhuoran Yang. Zhaoran Wang, and Tuo Zhao Pessimism meets invariance: Provably efficient offline mean-field multi-agent rl. Advances in Neural Infor- mation Processing Systems, 34:17913 1911999999996. 2021.
Wenbo Chen and Huixiao Yang. A heuristic based on quadratic approximation for dual sourcing problem with general lead times and supply capacity uncertainty. IISE Transactions, 51(9):943- 956, 2019. ISSN 24725862. doi: 00.10000247258542211888 1537533.
Andrew J Clark and Herbert Scarf. Optimal policies for a multi-echelon inventory problem. Man- agement science, 6(4):475 1960. 5
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020.
Scott Grant Eckert. Inventory management and its effects on customer satisfaction. Journal of Business and public Policy, 1(3):1-13, 2007.
Richard Ehrhardt. (s. s) policies for dynamic inventory model with stochastic lead times. Opera- tions Research, 32(1):121-13,, 1984.
Jiarui Fang, Lei Zhao, Jan C. Fransoo, and Tom Van Woensel. Sourcing strategies in supply risk management: An approximate dynamic programming approach. Computers Opera- tions Research, (0(5):1371-138) 2013. ISSN 0305-0548. doi: https///oo.org//0.0016///
2012.08.016. URL https: wttt://wwwceeeciirretttcctc htt.:///ww...oeccceeeeeeeetaaata
00305005812000 89X 1
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
llrria Giannoccaro and Pierpaolo Pontrandolfo. Inventory management in supply chains: A rein- forcement learning approach. International Journal of Production Economics, 7832)113--61, 2002. ISSN 09255273, doi: 00.1016000022-52230000000555
5
Joren Gijsbrechts, Robert N. Boute, Jan Albert Van Mieghem, and Dennis Zhang. Can Deep Rein- forcement Learning Improve Inventory Management? Performance and Implementation of Dual Soureing-Mode Problems. SSRN Electronic Journal, pp. 1-33, 2019. ISSN 1556-5068. doi:
10.21199ssmm.3322881
David A Goldberg, Dmitriy A Katz-Rogoziae Gatz-RRogozhnikov, Yingdong Lu, Mayank Sharma, and Mark Squillante. Asymptotic optimality of constant-ordes policies for lost sales inventory models with large lead times. Mathematics of Operations Research, (1(3):898-911 2016.
10

---

Olivier Guéant, Jean-Michel Lasry, and Fierre-Louis Lions. Mean field games and applications. In Paris-Princetor lectures on mathematical finance 2010, pp. 205-266060000 Springer, 2011. 1.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using
deep reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems, pp. 66-83. Springer, 2017. 5
Nir Halman, Diego Klabjan, Mohamed Mostagir, Jim Orlin, and David Simchi-Levi. fully polynomial-time approximation scheme for single-item stochastic inventory control with discrete demand. Mathematics of Operations Research, 34:674-685, 2009. 5
Ford W Harris. How many parts make at once. The magazine of management. 1913.
Christian D. Hubbs, Hector D. Perez, Owais Sarwar, Nikolaos V. Sahinidis, Ignacio E. Grossmann, and John M. Wassick. Or-gym: A reinforcement learning library for operations research prob- lems, 2020.
Woonghee Tim Huh, Ganesh Janakiraman, John A. Muckstadt, and Paat Rusmevichientong. Asymptotic optimality of order-up-to policies in lost sales inventory systems. Management Sci- ence, 55(3):404-420, 2009. ISSN 00251909, 15265501. URL http: jstor org/
sstale/40539155
5
Chengzhi Jiang and Zhaohan Sheng. Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system. Expert Systems with Applications, 36(3 PART 2): 6520-6526, 2009, ISSN 09574174. doi: 10. 10.10166.eswa..208..7777
5
Nan Jiang and Alekh Agarwal. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pp. 3395-3398. PMLR, 2018.
Robert S Kaplan. A dynamic inventory model with stochastic lead times. Management Science, 16 (7)449-5507, 1970.
Roman Kapuscinski and Sridhar Tayur. Optimal Policies and Eimulation-Basee Optimization for Capacitated Production Inventory Systems, pp. 7-40. Springer US, Boston, MA, 1999, ISBN 978- 1-4615-4949-9, doi: 10.1007/978- 000100797788---1---4999959
1-4615-4949-9 URL
778-1-4615-1 4959--9999999999
Ahmet Kara and Ibrahim Dogan. Reinforcement learning approaches for specifying ordering poli- cies of perishable inventory systems. Expert Systems with Applications, 91:150-158, 2018. ISSN 09574174. doi: (0.10166f.esaa2211....0000 5
Jean-Michel Lasry and Pierre-Louis Lions. Mean field games. Japanese journal of mathematics, (1):229-260, 2007.5
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed pnnperatvve-oopptttiyy environments. arXiv preprint arXiv:1706.02227, 2017.
5
Makridakis, E. Spiliotis, and V. Assimakopoulos. The m5 accuracy competition: Results, findings and conclusions. International Journal of Forecasting, 36(1):224-221 2020.
Nicolas Meuleau, Milos Hauskrecht, Kee-Eung Kim, Leonid Peshkin, Leslie Pack Kaelbling, Thomas L Dean, and Craig Boutilier. Solving very large weakly coupled markov decision pro-
cesses. In AAAI/IAAI, pp. 165-172, 1998. 5
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuogiu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1933 PMLR, 2016. B.2
Steven Nahmias and Stephen A Smith. Mathematical models of retailer inventory systems: A review.
Perspectives in operations management, pp. 249-278, 1993. 1
Afshin Oroojlooyjadid, M Nazari, Lawrence Snyder, and Martin Takáë. deep nentwwrrk for the beer game: A reinforcement learning algorithm to solve inventory optimization problems. arXiv preprint arXiv: 7008 05924,, 2017. 5

---

Afshin Oroojlooyjadid, Lawrence V. Snyder, and Martin Takáč. Applying deep learning to the newsvendor problem. IISE Transactions, 52(4):444 463, 2020. ISSN 24725862. doi: 10.1080/
24725854.2019.1632522. 5
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1), 2021.
URL httpt:::/// hettts://oppnreeee http://pppeeviiiiiee net www./orrmttormemte//am forum? d
Meng Qi, Yuanyuan Shi, Yongzhi Qi, Chenxin Ma, Rong Yuan, Di Wu, and Zuo-Jun Max Shen. A practical end-to-end inventory management model with deep learning. Available at SSRN 3737780, 2020. 5
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp.
PMLR, 2018.
Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation. arXiv e-prints, pp. arXiv-2006, 2020. 5
T. Remani, E. A. Jasmin, and T. P.Imthias Ahamed. Residential Load Scheduling with Renewable Generation in the Smart Grid: A Reinforcement Learning Approach. IEEE Systems Journal, 13 (3):3283-3294, 2019. ISSN 19379234. doi: 10. (0.1109//SYST220888885555
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. B.2
Lloyd S Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10):
1095-1100, 1953. 2.1
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019. 5
Nazneen N Sultana, Hardik Meisheri, Vinita Baniwal, Somjit Nath, Balaraman Ravindran, and Har- shad Khadilkar. Reinforcement learning for multi-product multi-node inventory management in supply chains. arXiv preprint arXiv:2006.0400 04037, 2020. 5
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint /706.v770.002222 2017.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Wilson, A scientific routine for stock control. Harvard Business Review XIII, 1934.
Yuhuai Wu, Elman Mansimov, Shun Liao, Alec Radford, and John Schulman. Openai baselines: 2018.
Acktr & a2c. https:// wttp.ccccccbcccccccecc wwwaaassennnnsaat-aaaaaaaaa
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In International conference on machine learning, pp. 5571-5580. PMLR, 2018. 5
Fan Ye, Helin Zhu, and Enlu Zhou. Weakly coupled dynamic program: Information and lagrangian relaxations. IEEE Transactions on Automatic Control, 63(3):698-713, 2017. 5
Yunan Ye, Hengzhi Pei, Boxin Wang, Pin Yu Chen, Yada Zhu, Jun Xiao, and Bo Li. Reinforcement- learning based portfolio management with augmented asset movement prediction states. AAAI 2020 34th AAAI Conference on Artificial Intelligence, pp. 1112-1119, 2020. ISSN 2359-5399. doi: 10. 10.9///aa 24400.5562.. 1
12

---

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXi::1003.0155,
2021. # S B.1
13

---

In the first stage of our algorithm, we run the policies in the origin joint environment to get context trajectories, i.e., {c This process is similar to other popular MARL methods. Note that also save the transitions of each agent for learning the policy and the value network. We A DETAILS FOR THE INVENTORY MANAGEMENT PROBLEM
In this appendix, we summarize the notations introduced in Section 2.2 Table We also give an example with two SKUs in Figure 4 to further illustrate the dynamics of the inventory management problem.
B DETAILS FOR THE ALGORITHM
In this section, will introduce the details of the designs in CD-PPO including the details for the subroutines Algorithm [ Additionally, to augment the context trajectories collected by the joint simulator, design context augmentation technique which leads variant of CD-PPO with context augmentation (cf. Algorithm 4). We will provide experiment results on this variant in Appendix
B.1 GET CONTEXT DYNAMICS
Our algorithm follows MAPPO Yu et al. 2021) and maintains two separate networks for Th and Va.
provide the the pseudocode of this joint sampling process in Algorithm where we record the context trajectories for the training in the next state.
Additionally, we design a context augmentation technique in this stage to augment the collected context trajectories. After collecting the context trajectories, use an LSTM model fe to train a surrogate predictor as an extra augmentation for the collected context trajectories in next stage. In details, we split the collected context trajectories into several sub-sequences of length 8 and the training objective is to minimize the mean-squared error of the I, + 1 day's 's capacity predicted given the dynamics of previous L days.
min(fele-Z......e; c++1)² This results in CD-PPO with context augmentation which presented in Algorithm
B.2 DECENTRALIZED PPO
With the collected context dynamics of the shared resource, it is easy run the second stage: sam- pling on the local simulators for each agent and then training the policy and critic with data. It is worth noting that the main difference between our training paradigm and traditional MARL meth- ods under CTDE structure is that we directly sampling local observations in the extra simulators in
Table 2: Notations for the inventory management problem.
Notation
Explanation
Units in stock of SKU at the end of the t-th time step
Units in stock of SKU at the t-th time step before discarding the excess Lead time for the order placed at the t-th time step Order quantity the i-th SKU at the t-th time step Arrived quantity of the i-th SKU the f-th time step Demand of the i-th SKU at the t-th time step Sale quantity of the i-th SKU at the f-th time step Units transit of the i-th SKU at the [-th time step Profit generated the i-th SKU the 1-th time step
L
O
A
D
S
T
Pt
Unit sales price of i-th SKU
Ø
Unit procurement cost of i-th SKU
0
Unit order cost
Unit overnight holding cost
14

---

which only one agent existed rather than the joint simulator in which all agents interacting with each other. In other words, in the new local simulator, there is only one SKU in the entire store, and the trend of available capacity is completely simulated according to the given context dynamics.
In practice, we parallelly initialize new instances of the original inventory environment with the new configure settings which only contains specific SKU and a fixed trajectory of context. As for the fixed context trajectory, we use the subtraction results { {e-") with some augmentations: 1) add some noise in some items with predefined probabílity:2) replace some items with predicted values comes from the trained context model also by the predefined probability. Then we run the policy to interact with the local simulators conduct episodes under the embedded context dynamics and put them into a shared replay buffer since all transitions are homogeneous and we shared the parameters over all policies. And decentralized training will be started by utilizing the shared replay buffer of transitions collected from local simulators.
We consider a variant of the advantage function based on decentralized learning, where each agent learns a agent-specific state based critic Vş(st) parameterized by ø using Generalized Advantage Estimation (GAE, (Schulman et al. 2016)) with discount factor 7 0.99 and = 0.95. We also add entropy regularization term to the final policy loss (Mnih al. 2016). For each agent i, we have its advantage estimation as follows:
A
Σ (72)
where 8/ as) + VVo (2(+2) Vo (st) is the TD error time step t and h is marked as steps num. And we use individual reward provided from local simulator ri(s of). So that the final policy loss for each agent: becomes:
me (at st) Taua (at st) Ai
L' (0) = E B) atenl-Trall
min
(10)
To (a st
clip
1 - c, +e ) A
Toold
(a)
As for training value function, in addition to clipping the policy updates, our method also value clipping to restrict the update of critic function for each agent to be smaller than as proposed by GAE using:
min {va (sf) - -v:)² v:
('( = Taa (179)
(11)
(Vons (si) + clip (Vo (si) Vouta (si), - +c -c,))) 13 "1l
where Gold are old parameters before the update and 1/F = A + Vo (st). The update equation restricts the update of the value function within the trust region, and therefore helps to avoid
O = 20,4,
5,5, =1
11 = 10,73 0
3 12
12 [0.78 = 20j
20,
2f 0
20 13,7/
13
time
time step
time step
3.0/ =
DJ 5,0 =11
1
Roplenishing orders
11
Imax
(14+22-31)
0.22
Customers' demands
(7+
Resolving storage overflow
Figure 4: A diagram to illustrate an inventory dynamics in two time steps.
15

---

overfitting to the most recent batch of data. For each agent, the overall learning loss becomes:
6) E (0) + Aeritic + Aentropy H. (T²*)
(12)
It is obvious that all networks are trained in the decentralized way since their inputs are all local variables which stem from the light-cost local simulators. As mentioned before, this learning stage, there are no interactions between any two agents. Although it seems like the way of indepen- dent learning, we need point that we use the global context simulated from the joint environment, which is essentially different from independent learning methods since they will not consider this style global information which is simulated from joint simulator but be fixed the local simulators. Our decentralized training have several advantages: firstly, the local simulator is rumning efficient because of its simple only-one-agent transition function; secondly, this paradigm avoid the issue for non-stationary occurred in the traditional MARL methods since there no interaction amongst agents so that it is no need to consider influences of other agents; thirdly, we can use more diverse context trajectories to make agents face various situations of the available levels of the store, which leads to improve the generalization of the networks to be trained; fourthly, it is easy for this training paradigm to be extended large-scale distributed training by running parallel simulation whose communication cost is also acceptable for modern distributed training frameworks.
If the critic and actor networks are RNNs, then the loss functions additionally sum over time, and the networks are trained via Backpropagation Through Time (BPTT). Pseudocode for local sampling with recurrent version of policy networks shown Algorithm
Algorithm 2 GelContextDynamic
INPUT policies {Th] ) and the joint simulator Envjooint (Optional) Initialize the parameters for context model fo set data buffer D = 0 for bs to batch..size do
T = [ empty list initialize h ... hi, actor RNN states
for 1= to do
for all agents do
Pi, bi. = T' C, M M 0,)
of ~
end for
Execute actions a,, observe re, St+11 end for
// Split trajectory into chunks of length L
for = T//L do
D = (el:
end for end for // (Optional) Train the context model for augmentation if Need train context model then for mini-batch & ....., K1 do
b < random mini-batch from D with all agent data Update capacity-context dynamics model with Eq. end for
Adam update with data b
end
OUTPUT context dynamics (T- [T=1 for 1, 1,... n; (Optional) fa
16

---

Algorithm: DecentralizedPPO
Generate data for agent with corresponding context dynamic model INPUT local simulator Enviocal: policy Tè, and value function d. Set data buffer D = 0
Initialize h hü. actor RNN states
Initialize hob v.. hö.v critic RNN states T empty list
for t- 1 to 7 do
pú. hi,a - n ( 1 cf
aj ~pp of M (i),,, ct h 1
Execute actions of in EnVlocal: locals and then observe ri. 441 8
T+ T+ [si c, oi, hi, h v. s1+ dt11l
end for
Compute advantage estimate À via GAE (Eq. Compute reward-to-go R t // Split trajectory т into chunks of length L in D for 0, 1, T//L do T, Al 1+ L}. R1 : 1+ LJ)
D D DUDUUUT end for for mini-batch k- = 11 1,..., K₂ do
f random mini-batch from D with all agent data for each data chunk in the mini-batch b do
update RNN hidden states for x' and V from first hidden state in data chunk end for end for
Calculate the overall loss according Eq. equation 10 equationn 12
Adam update 0, on L' L'(0,) and gH with data Adam update ob on L (ov)) with data
OUTPUT policy T.. and value function
Algorithm 4 Context-aware Decentralized PPO with Context Augmentation
Given the joint simulator Enyjaint and local simulators (Envi In Initialize policies म' and value functions V for i = 1,...,
/==1
Initialize context model fa and the augmentation probability Paug for M epochs do
// Collect context dynamics via running joint simulation
{c 11 .. for 1 fa +- GelComeetDyyannicsDEByno { 1 (Algorithm) for = 1,2 K do
for all agents do
// Set capacity trajectory by augmented context dynamics
stt..cttsttt...ttttttttrro (aug() {c; "T==- fwo Paug))
// Train policy by running simulation in the corresponding local environment
v² Vi + DecentralizedPPO) (Enviocat> n², Vii (Algorithm
end for
end for
Evaluate policies { H on joint simulator Envjoiat end for
17

---

C DETAILS FOR THE EXPERIMENTS
C.I THE CODEBASE
As part of this work we extended the well-known EPyMARL codebase (adleaase(Papoudakki et al. 202T)) to
integrated our simulator and algorithm, which already include several common uueddn algorithms and support more environments as well as allow for more flexible tuning of the implementation details. It convenience for us compare our algorithm with other baselines. All code for our new codebase is publicly available open-source on Anonymous GitHub under the following link: https: // anonymous Aopen. wtteaeeeecccccceeeeeeeeeeeeeeeee
coamment-marl-- searr-baseeinee -
C.2 HYPERPARAMETERS DETAILS
We present the hyperparameters used in our algorithm for environment with 5 SKUs in Table
Table 3: Hyparameters used in CD-PPO
Hyperparameters
Value
runner
ParallelRunner
batch size run
10
decoupled training use individual envs
True
True
max individual envs decoupled iterations
5
train with joint data context perturbation prob
True
1.0
hidden dimension
64
learning rate
0.00025
reward standardisation
True
network type
FC
entropy coefficient
0.01
target update
200 (hard)
n-step
5
C.3 DETAILS OF STATES AND REWARDS
Table shows the features of the state for our MARL agent that corresponds to the i-th SKU on the I-th time step.
worthy to note that we use the profit generated on the i-th SKU the I-th time step Fi divided by 1 x 10 as the individual reward of i-th agent at the 1-th time step, for team reward methods, we simply sum up all the individual rewards, which corresponds to the daily profit the whole store at the t-th time step, divided by 1 x 10
D BASE STOCK POLICY AND (s, S)-POLICY
In this paper, we choose two well-known classical methods, the base-stock policy and the Ss policy, from the OR community as our non-RL baselines. In the (s, S)-policyy, the two parameters and S control the replenishment strategy where is the reorder point and S is the maximum level. On each time step (or review period), when the inventory position (including the units in stock and the units in transit) declines to or below the reorder point we order up to the maximum level S. The base-stock policy is a special case of the (s, -polllccy with = S, i.c., on each time step we place order to fill the inventory position up to S. Previous studies proposes different algorithms to determine these two parameters (see e.g., |Axsäter 2015). In our experiments, we search for the posteriori optimal parameters for each of the SKUs in control. Therefore, the performance of these two policies presented Table can be regarded as the best performance that can achieved by such classical methods.
18

---

Table 4: Features of the state
Features
State
Storage information
Storage capacity C
Inventory information
Quantity of products in stock I Quantity of products in transit T
History information
Replenishment history in the latest 21 days [0 ... 0%-1l
Sales history in the latest 21 days [S] 21; St
Standard deviation historical sales std (Si (5/-21 21" ... 1
Product information
Unit sales price ps Unit procurement cost 9i
Context
Global storage utilization
Current total storage level of the store Σ P
Global unloading level
Current total unloading level the store 2 1- 0 1- 1,+1
Global
excess
level
Current total excess level of the store px E,= 0 0-11,
E ADDITIONAL EXPERIMENT RESULTS
E. THE COMPLETE RESULTS
Here we present the complete experiment results in Table and display the training curves of algo- rithms 50-SKUs scenarios in Figure and Figure We also present the results from OR methods designed to deal with IM problem, namely the base-stock policy, in Table 5
Please note that, actually VLT (i.c., leading time) for each SKU in N50 and N100 stochastic during simulation. On the one hand, the specified VLTs can be modelled as exponential distributions in our simulator. On the other hand, the real lead time for each procurement can be longer than specified due to lack of upstream vehicles. (Distribution capacity is also considered in the simulator, though not the focus of this paper). So that the results running on N50 and N100 are all considering stochastic VLTs.
worthy noting that throughout the environments we have experimented on, CD-PPO enjoys higher sample efficiency shown in Table 6 Though it seems that in environment where the storage capacity is extremely tense, like N5-C50, CD-PPO performs not as well as IPPO, we reason that these tense situations favor IPPO (with team reward), which may quickly learn how adjust each agents' behavior to improve team reward. While CD-PPO, owing to its decentralized training paradigm, struggles to learn highly cooperative policy to cope with the strong resource limits while meeting stochastic customer demands. Yet we still claim that CD-PPO does outperform its IR (w/o context) baselines while producing comparable performance when compared IR (with context) algorithms. On the other hand. IPPO(with team reward) struggles to learn good policy in 50-agents environment for the large action space compared the single team reward signal, and fails when faced with N50-C2000 whose state space is even larger. To top it all off, CD-PPO does have its strengths in terms of sample efficiency. Though we will continue to address the above challenge in our future work.
E.2 ABLATION STUDIES FOR CONTEXT AUGMENTATION
In this section, aim to seek better way to augment the context dynamics in order to boost performance of our algorithm. We ponder the following two questions:
Q1: Which is better way to augment the original data, adding noise using Deep predic- tion model?
To answer this question, set out experiments environment N5-C100 with algorithm CD- PPO, using either context trajectories generated by a deep LSTM prediction model, or simply adding random normal perturbation to the original dynamics trajectories, both on different seeds. As
19

---

Prott
Proft:
10
104
162
104
107
Sroff: (log SCIl
EPO-IRINO contect)
101
IPPO-R/W/ contecti
-16
FFO contect:
-10
IPPO context)
C3.990
-10
-10
-10'
-16
-104
-16-
-104
-10
-164
-10-
-107
-10-
-10f
-10
Datua Samples
Dabe Sarplees
Figure 5: Training curves on N50-C500.
Figure 6: Training curves on N50-C2000
shown in Figure those runs with deep prediction model generated dynamics enjoy less std and bet- ter final performance. This could result from that the diversity of deep model generated trajectories surpasses that of random perturbation.
1e6
Profit Comparison
Noise
Pred
1.0
Profit
-0.5
Data Samples
1e6
Figure 7: Training curves of CD-PPO with different augmentation methods
1e6
Profit Comparison
1.0
Profit
-0.5
fake Dynamics prob: 0%
take Dynamics prob: 50%
Dynamics prob: 100%
Data Samples
1e6
Figure 8: Training curves of CD-PPO with varied ratio of augmented data.
20

---

Q2: Does dynamics augmentation improve the performance of the algorithm? If how much should we perturb the original data?
We run similar experiments on environment N5-C100 with CD-PPO, in which the local simulator is ingested with a mixture of original dynamics data and LSTM generated data. The ratio of per- turbed dynamics data varies from 0% to 100%. And we found that the algorithm turns the best performance when we use fully generated data, as shown in Figure g
21

---

Table Average number of samples needed by different algorithms to reach the median performance of the baselines on different environments
-6501 4214-222444400 06
6619.288 1.955.4999 .9
-2223. -2223.11 11-11122236000
1297.751 1
148.00=1017.47
N100-C4000
410.59
493.32
755.47
N/A
N/A
N/A
N/A
N100-C400X
-2106.98 1-15.33
1126.42-449.88
- 198811991066611
-669 83 1. 1395.92 929
660.28 149.94
N/A 1151.57
127.57
60.07
N100-C1001
N/A
N/A
N/A
DC
-17.339
-22.55
91.17
N/A
N/A
N/A
N/A
N100-C1000
Table 5: The complete experiment results of all the variants.
N/A 539.26 26
191. 88
47.14
694.87 174.184
N/A -1773.9-287.03
-1002971115.66
N/A
N/A
N/A
0O
0O
N/A 48.92
545.86
689.27--48.8 077-48.99
N50-C2000
2573
22000
1155911
42,7092
N50-C200
2996.87
3138.49 49
8483.04 3.04
N/A
N/A
N/A
N/A
N/A
N/A
DC
DC
310.81 16.7.44 76.46
N/A 16-666066 61
250.03-158.33
N/A 33
164.33+443.01
366.74 89.58
N50-C500
N/A 12802.89 82
N50-C500
235.09+-60.
33000810
408. 1434
5484.49
N/A 9195.23
-397.85 831
-430.08
N/A
N/A
0
00
N/A
N/A
N5-C100
522.80
711.97
987.27
806.56
1298.40 .00
1.99
92.41+2.78
94.70-188884
91.38
14.41
97. 97.89=6.65
1..51=1.551
13.36
71.57:3.114
N5-C100
92. 7.990113.30
48 .949
60.8602
97.7010
00
0C
00
N5- N5-C50
708. 10
588.63
1671.10
708.
1671.10
00
00
66.02
40.37 40.37+4..89
39.32+15.55
43.33
54.87-9926
74.11+1.55
49.2411.32
63.22-11.755
48 1..49
NS-C50
17.4834
33.9469
38.6207
CD-PPO (ours)
HAPPO-IR-NC
Env Scenario
IPPO-IR-NC -C-
MAPPO-NC
MAPPO-IR
IPPO-IR
IPPO-NC
MAPPO
IPPO
MAPPO- IR(w/o context)
IPPO-IR (1T0-IR(WM (w/o context)
MAPPO (w/o context)
IPPO(W/o context)
Basestock (Statie).
Desestock(Dynantt
Basestock (Oracle)
CD-PPO (ours)
Env Scenario
MAPPO-IB IR
IPPO-IR
MPPPO
IPPO
22


Reviews:
[{"review_id": "oJQurN2uXu", "reviewer": "Reviewer_cRM3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors have considered the problem of managing inventory on the replenishments for a number of SKUs to handle the supply and demand. This paper proposes Context-aware Decentralized PPO (CD-PPO) by formulating the problem as a Shared-Resource Stochastic Game (SRSG). The authors have conducted rigorous experiments that CD-PPO learns quickly and achieve a good performance with MARL algorithms.", "summary_of_the_review": "Overall a good research article but needs a lot of improvement to be considered for publication in ICLR 2023. ", "strength_and_weaknesses": "\nStrengths:\n1. Interesting to address inventory management problem through a Shared-resource Stochastic Game (SRSG) \n2. Introduction of context awareness to avoid non-stationary problem and acceleration of sampling process in estimating and optimizing objective function.\n3. Experimentation set up and baselining methods used in the study\n\nWeakness:\n1. Although the authors claim that the CD-PPO algorithm has outperformed MARL but it was indicated that it is due to the avoidance of non-stationarity by introducing the context but there are other methods like IPPO-IR with context showed a poor performance. This needs to be explored in detail.\n2. The problem space considered was too small to be accepted in a premier conference like ICLR.\n3. Clear standing and providing relevant real-world applications with shared-resource structure will enhance the readability of this article.\n4. Rationale for the selection of MARL algorithms needs to be explored in detail.\n5. There are several areas where the usage of language needs to be improved considerably. Some of the example are as follows:\na. In our setting, the constraint on the shared resources (such as the inventory capacity) couples the otherwise independent control for each SKU.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "V2Y1QTCULM5", "reviewer": "Reviewer_cFPA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, multi-agent reinforcement learning was used over dynamic programming to optimise inventory management. In this study, they proposed using Shared-Resource Stochastic Game to capture the problem structure in the inventory management where different agents interact with each other through competing for shared resources - an interesting approach for this type of problem. In addition to this, they propose a novel algorithm called Context-aware Decentralized PPO that leverages the shared-resource structure to solve the inventory management problem efficiently. and finally they performed extensive experiments to demonstrate that their method can achieve the performance on par with state-of-the-art MARL algorithms. In their study, they lack the founding conclusion but based on the intext questions, what was set out was achieved within the scope of the study", "summary_of_the_review": "I recommend that the paper be accepted, but that a concluding section is added to the paper. As it currently stands, there is a lot of concluding thoughts missing from the paper - however impressive the algorithms and hyperparameters were.", "strength_and_weaknesses": "This paper has an impressive and detailed account for the different algorithms used as well as the hyperparameters to make them function. What was missing was wrapping up the paper within the concluding remarks so that the reader can at a glance summate the totality of the research paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KoYH1zTiBO", "reviewer": "Reviewer_Yg5s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A multi-agent RL algorithm is proposed to solve the multi-SKU inventory optimization in a single-echelon problem. The goal is to obtain the order quantity for each SKU while the overall capacity constraint is held and the total profit is maximized. The profit is calculated by getting the sell amount, purchase cost, holding cost, and fixed order cost. To define the MDP, the state definition for each SKU at each time involves in-transit quantity, the sale price, its historical actions, and its customer demands.\nTo manage the capacity constraint that all the SKUs try to get as much as possible, they propose to use a context variable which is consisted of all the capacity usage of all SKUs at the current time and some of the previous steps.\nTo build the simulator, it is assumed that the effect of SKU on the overall capacity is negligible, so they mostly run the simulator for each agent locally. \n\nThe weights of the policy and critic networks are shared among all agents. The action is one of {0, 1/3, 2/3, 1, 4/3 , 5/3 , 2, 5/2 , 3, 4,5,6,7,9,12} times the average of daily sales over the last two weeks. The reward is the local cost of the agent, and the state definition is as mentioned earlier.\n", "summary_of_the_review": "Overall it is a good paper and studies an important problem. Although, there are some gray areas in the paper and the claims so that I do not recommend acceptance with the current version. Once the author address my concerns/questions, I raise my vote.", "strength_and_weaknesses": "Strength: \n- A new RL algorithm to consider the capacity constraint in single stage inventory optimization problem.\n- The proposed algorithm works better than some existing classical and RL-based approaches in problems with up to 1000 items. \n\nWeakness: \n- Lack of appropriate benchmarks \n- Weak presentation on some sections \n\nHere are the details of my questions:\n\nQ1- At equation (4), you add a constraint on the arrival inventory if that results in passing the inventory capacity. The question is where do you assume the extra inventory stays? Do not you need to consider the holding cost, or some type of penalty cost for that? \nQ1-1 This issue becomes worst if you consider a multi-echelon inventory optimization problem, in which each stage receives inventory from its predecessors, and early arrival is usually penalized with high costs. \n\nQ2- At the end of page 4, the last line, do you mean $c^{-i}_t$? It is now $c^{-i}$.\n\nQ3- I am not sure if the assumption of using an independent simulator for each SKU is a good idea. Specifically, it is mentioned that the multi-SKU simulator includes \"replenishing orders fulfillment scheduling and transportation management\" which I am not sure if they are necessary for your purpose. If you do not have those details in the single SKU simulator, you do not need to have them in the multi-SKU simulator too. Definitely computation-wise it is helpful, but assume a store that chooses to purchase bulky items that can sell them very quickly, like the garden items for the early spring. Those items heavily affect the overall used capacity and the simulator cannot ignore them. \n\n\nQ4- \"MAPPO learns a centralized critic and therefore becomes intractable due to the large state-action space when the number of SKUs is greater or equal to 50\". I would not use this baseline since it cannot scale as you mentioned. [1] uses an attention method to create the state input for the critic method to address this issue.\n\nQ5- The details of the chosen \"values\" like the prices, the costs, and the leading time were not available in the actual dataset. Can you please add those in an appendix in the paper? Without those, it is not possible to replicate your result.\n\nQ6- I did not understand why you need equation 8 and the LSTM model to learn the transition function for the next step for capacity. I assume you can easily obtain that by inventory updating rules. Why do you need a predictor there? Can you please clarify?\n\nQ7- The IPPO algorithm uses only $s_t$. I am not sure why it cannot use $c_t$. In reality, it is a piece of available information for all agents. I am not sure why you have not added that as part of the state.\n\nQ8- How do you monitor/penalize if a solution results in a violation of capacity constraint? \n\nQ9- I think the mean-field approach could be a good benchmark for your work since it has been tested many times on many-agent problems. See [2,3] for more details.  \n\nQ10- The current algorithm cannot be directly applied to a multi-echelon inventory optimization problem. It is not clear how the excess inventory (compared to the capacity of successor/predecessors) can be handled. Also, the performance of that is not known. Given that, I would emphasize that in each piece of the paper that this algorithm is for a single-echelon inventory optimization problem. \n\nQ11- How do you consider the service level in your model? In other words, how do you enforce the attainment of a given service level for each item?\n\nQ12- A viable benchmark for your work would be [4]. Although it does not consider fixed-order cost, you should be able to set that as a benchmark for your model when the fixed-order cost is zero and check your optimality gap. This comparison could determine the true value of your algorithm.\n\n\nMinor comments: \n\nM1- provide the the pseudocode -> provide the pseudocode\nM2- There are several cited arxiv papers which some (that I know) are published at some venue. I would suggest you go over all the papers and update them all.\n\n\n[1] Iqbal, Shariq, and Fei Sha. \"Actor-attention-critic for multi-agent reinforcement learning.\" International conference on machine learning. PMLR, 2019.\n\n[2]] Yang Y, Luo R, Li M, \"Mean field multi-agent reinforcement learning\". PMLR, 2018: 5571-5580.\n\n[3] Caines, Huang and Malhame, \"Mean field games\", 2017. DOI 10.1007/978-3-319-27335-8_7-1", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bh1v6YqDfUt", "reviewer": "Reviewer_626R", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the context of multi-agent reinforcement learning, this work addresses the problem of inventory management, where a large number of stock keeping units need to learn how to optimally make replenishment decisions, under a shared inventory capacity. The work contributes the formulation of the shared-resource stochastic game (SRSG) to capture this problem setting and proposes to decouple the agents for a more efficient training procedure, by separately capturing and modelling the shared dynamics (i.e., the state of the shared inventory capacity). This allows the use of a variant of PPO, denoted as Context-aware decentralised PPO (CD-PPO), to obtain a sample efficient MARL solution for this problem.\n", "summary_of_the_review": "The work addresses an important issue regarding bridging the gap between real-world large multi-agent systems and MARL approaches. However, due to a lack of technical details, it is difficult to judge the novelty and significance of the presented contributions. I cannot currently recommend the acceptance of this work. ", "strength_and_weaknesses": "Strengths: \n- This work addresses inventory management, an important sub-problem of supply-chain management \n- It tries to bridge the gap between MARL methods and real-world applications with a large number of agents, addressing issues such as sample efficiency and computational costs for large simulators\n- It demonstrates a manner of learning effective policies, by modelling and predicting separately the shared dynamics and using these predictions in the training procedure of the agents\n\nWeaknesses:\n- Insufficient details are presented for the creation of the local simulator and the context dynamics modelling\n- There are also some clarity issues regarding CD-PPO and how exactly it is different from IPPO or MAPPO, how many networks it uses (more details below)\n- The clarity issues obfuscate also the ability to judge the experimental evaluation, and the significance of the results\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject