Whittle Index with Multiple Actions and State Constraint for Inventory Management
Authors: Chuheng Zhang
Xiangsen Wang
Wei Jiang
Xianliang Yang
Siwei Wang
Lei Song
Jiang Bian
Venue: ICLR 2024 poster (2024)
Keywords: MARL
Inventory Management
Whittle Index
Primary area: reinforcement learning
TL;DR: We propose a novel form of Whittle index that extend to a MARL setting suitable for practical inventory management.

Abstract: Whittle index is a heuristic tool that leads to good performance for the restless bandits problem. In this paper, we extend Whittle index to a new multi-agent reinforcement learning (MARL) setting with multiple discrete actions and a possibly changing constraint on the state space, resulting in WIMS (Whittle Index with Multiple actions and State constraint). This setting is common for inventory management where each agent chooses a replenishing quantity level for the corresponding stock-keeping-unit (SKU) such that the total profit is maximized while the total inventory does not exceed a certain limit. Accordingly, we propose a deep MARL algorithm based on WIMS for inventory management. Empirically, our algorithm is evaluated on real large-scale inventory management problems with up to 2307 SKUs and outperforms operation-research-based methods and baseline MARL algorithms.

Published as a conference paper at ICLR 2024
WHITTLE INDEX WITH MULTIPLE ACTIONS AND STATE CONSTRAINT FOR INVENTORY MANAGEMENT
Chuheng Zhang', Xiangsen Wang", Wei Jiang', Xianliang Yang', Siwei ; Lei Song', Jiang Bian'
'Microsoft Research Asia "Beijing Jiaotong University University of Illinois Urbana-Champaign chuhengzhang emroooofff com
vangxiangsengbjtu.co edu. wei weij40illino edu
{xi (xianliang . yang, siweiwang, dri.song, jiang. biann @mi awwiiirooo.ccommm com
ABSTRACT
Whittle index is a heuristic tool that leads to good performance for the restless bandits problem. In this paper, we extend Whittle index to a new multi-agent reinforcement learning (MARL) setting with multiple discrete actions and a pos- sibly changing constraint on the state space, resulting in WIMS (Whittle Index with Multiple actions and State constraint). This setting is common for inventory management problems, where each agent chooses a replenishing quantity level for the corresponding stock-keeping-uni (SKU) such that the total profit is max- imized while the total inventory does not exceed certain limit. Accordingly, we propose a deep MARL algorithm based on WIMS for inventory management, and evaluate our algorithm empirically real large- scale inventory management problems with up to 2307 SKUs. The results show that our algorithm outperforms operation-research-baase methods and baseline MARL algorithms.
INTRODUCTION
The inventory management problem is one of the most important problems in operation research (OR) with a history dating back to Fukuda (964).. The objective of the problem is to balance the supply and demand of different boock-eeeiing-unee and vendor lead time (VLT). A good strategy for inventory management can not only reduce the operational cost and increase the profit but also improve customer satisfaction (Eckert 2007).
(SKUs) the face of stochastic customer demands
Traditional methods from the operation research community solve the problem using dynamic programming (DP). However, these methods often rely on strong assumptions such as i.i.d. customer demands and deterministic lead time (Kaplan 19700 EErbardi 1984) which make them hard to apply for a general setting. While approximate DP methods (Halman et al. 2009: Fang et al. 2013 Chen & Yang 2019) can solve for more problem formulations by relaxing the assumptions, they still rely heavily on cooblem-specifie expertise or assumptions, e.g., the zero or one period leading time assumption (Halman et al. 22099.. In contrast, reinforcement learning (RL) based methods 1) do not rely on strong assumptions and are applicable to general scenarios; 2) can exploit the underlying pattern from real data (instead of solving for fixed distributions of customer demands and VL.T); and 3) enable efficient inference instead of, e.g., planning based on the prediction in the inference time as in (Qi et al. 2020). However, it is quite inefficient for RL with single agent to learn global policy to make decisions for all SKUs due to the exponentially large state and action space (Jiang & Agarwal 2018). Therefore,
natural to adopt the multi-agent reinforcement learning (MARL) paradigm, where each SKU is controlled by one agent whose state and action spaces are localized.
In this paper, we consider a constrained MARL setting for inventory management where a possibly changing constraint on the fotal inventory capacity couples the otherwise independent dynamics of a large number of SKUs. We refer to such a structure as weak coupling hereinafter. Existing MARL algorithms face some challenges in this setting. The first challenge is the trade-off between efficiency and performance. On the one hand, existing centralized centralized-training -tai-trrnnnnnnneeeee based MARL algorithms (c.g., MADDPG (Lowe et al. 2017), MAPPO YYu et al. 202T), and COMA (Foerster et al. 2018)
"Corresponding Author

---

Published as a conference paper at ICLR 2024
need to estimate centralized critic that receives the global state (i.e., the concatenation of local states of all SKUs) as the imput. However, in a typical industrial scenario, we have to manage up to thousands of SKUs, which leads to computational inefficiency due to the exponentially large MARL algorithms (e.g.. IQL. (Tan 1993) and IPPO (de Witt et al.] 2020)) are efficient but may result in poor performance due to the ignorance of the global constraint. The second challenge is that the limit of the total inventory capacity can change across different seasons (due promotion, holidays, etc. for inventory management), Accordingly, previous MARL algorithms need to re- reairain the model on new inventory limits or call for the meta RL formulation (see e.g., FFinn et al. 2017 Nichol & Schulman 2018 which complicates the algorithm/model and results in higher time complexity.
global state space. On the other hand, existing sndependent-leaming-ean
To deal with the above challenges, we resort to Whittle index (Whittle 1988) proposed as a heuristic but effective tool to solve the restless bandits problem which presents similar structure. In restless bandits, the dynamies of different arms are independent MDPs (cf. the dynamics of different SKUs are independent in our setting) but only M out of the N arms are allowed to be activated on each time step (cf. the global inventory capacity constraint in our setting), and an algorithm needs to learn how to activate arms to earn reward as much possible. The basic idea of Whittle index is to estimate the maximum affordable cost for activating each arm in its local state, and then activate the M arms with highest maximum affordable cost. By this way, decouples the global control into several local learning procedures. However, Whittle index cannot be adopted directly to our setting due to the following two gaps between our setting and restless bandits: 1) The constraint is imposed on the global state (e.g.. on the total inventory capacity) in inventory management whereas the constraint imposed on the global action (i.e., the number of activated arms) in restless bandits; 2) There are multiple actions for each SKU (e.g., multiple candidate replenishment quantities) in inventory management whereas the action for each arm binary (i.e., activate or not) restless bandits.
In this paper, we extend Whittle index to bridge the two gaps by defining new form of Whittle index called WIMS (Whittle Index with Multiple actions and State constraint). WIMS measures the cost of unit budget consumption instead of the cost per arm activation (since our constraint is on the total inventory capacity but not the total replenishment quantity) to bridge the first gap, and generates the critical points of maximum affordable costs for different actions (i.e., vector) on given local state instead of only a scalar to bridge the second gap. Then, we propose practical MARL algorithm called WIMSN that combines WIMS with the neural network to solve real inventory management problems in a computationally efficient way. We evaluate our algorithm based on real data of up to 2307 SKUs, and the experimental results indicate that our algorithm can outperform not only operation-research- based (OR-based) methods but also previous MARL algorithms.
The contributions of our paper summarized as follows:
We propose WIMS (Whittle Index with Multiple actions and State constraint) that extends Whittle index to new constrained MARL setting where a possibly changing constraint is imposed on the global state and there are multiple discrete actions for each agent.
We propose WIMSN that combines WIMS with neural networks, resulting in a practical algorithm that solves real inventory management problems efficiently.
We conduct extensive experiments demonstrate that WIMSN can outperform the OR-based methods as well as previous MARL algorithms.
2 RELATED WORK
Inventory Management. In the OR community, representative strategies for inventory management include the base-stock policy (Clark & Scarf 1960 Kaplan 1970). the (s, -ppolicy (Ehrhardt 1984]. and the (R, -)-policy Chen (200)) which are proved to be optimal under specific assumptions (see
Chen 2000 However, these assumptions are too strong to hold in practice and researchers start to use the approximate form of dynamic programming (Halman et al. 2209 Fang et al. 2013 Chen &
Yang 2019) instead of the exact form (Goldberg et al. 2016 Huh et al. 2009).
To better exploit the patterns from big data, reinforcement learning methods have been adopted to solve the inventory management problem (see e.g., Giannoccaro & Pontrandolfo 2002 Diang &
Sheng 2009
Kara & Dogan 2018; [arrtt et al.) 2019: Gijsbrechts et al. 2019) Oroojlooyjadi et al.
2017
2020).
While these methods able outperform traditional approaches in different scenarios
2

---

Published as a conference paper at ICLR 2024
such the multi- setting and the setting with volatile customer demands and bullwhip effects, they can hardly scale up to thousands of SKUs efficiently due to the ignorance of the weak coupling structure in the problem. Ding et al. (2022) considers similar setting to ours and can scale up by decoupling the learning of different SKUs conditioned on the fixed context sequence where the context indicates the allowed capacity for each SKU. However, the model is trained under given constraint level, and they need to retrain the model when the constraint level changes.
Whittle Index. Whittle index (Whittle 1988) is heuristic tool that is effective in solving the restless bandits problem (a PSPACE-hard problem) (Papadimitriou & Tsitsiklis 1994). Restless bandits problem generalizes many real-world problems including HVAC control (Wei et al. 2017). wireless sensor scheduling in cyber-physical systems (Leong et al. (2020). and dynamic multi-channel access in wireless networks (Wang et al. 2018). (Waag et al. 2020). and Whittle index is shown to be effective in these real-world applications (see e.g.. Le Ny et al., 2008 [i et al. (022). Some existing researches also try to extend Whittle index to restless bandits with multiple actions (Killian et al. 202 Xiong al.) 2022 Hodge & Glazebrook| 2015 Killian et al. 2021b), but they are still dealing with constraints on the action space and cannol be applied to our setting (where the constraint is on the state space). In this paper, we further extend Whittle index to new MARL setting where a constraint on the state space couples the otherwise independent agents (i.e., the weak coupling structure), which could be applied in wide range of applications, e-g., inventory management (Ding et al. (022), portfolio management in quantitative finance (Jiang et al.] (0117), queue scheduling (Chen al. 202I). recommending systems (Liao et al. 2022). and the air campaign planning task (Meuleau et al. 1998).
3 PRELIMINARY
Notations. 5g is used to represent Dirac delta distribution centered at a. [N] is a shorthand for 11,2....,NN I[cond] is the identity function which equals 1 if cond = True and equals 0 otherwise. Given a vector x, we use [z]: to represent the i-th component of this vector.
Restless Bandits. The restless bandits problem is defined as follows: in each round f, control policy observes the state of each of the N arms (denoted by st for all i E [N]) and activates M arms (denoted by of E {0, 1} where 1 represents activation). The joint state space is the Cartesian products of individual spaces, i.e., S : S.V. and the joint action space is defined as A = {a E A EIENN ai M) (here A = {0, 1)). Then, the policy receives reward rf ~ (s) at) on every arm i, with R (sss 0)
60 (i.e., if we do not activate arm i at this time step, its reward must be 0), and the state of each arm transits to the next state t141 P'((-sf af).
The objective is to find a policy म. S + A(A) that maximizes E EN [ where y is the discount factor.
Whittle Index Policy, The Whittle index policy first decouples the joint optimization problem of N arms into individual solutions for each arm, and then use a simple way combine these solutions.
Therefore, we can first only consider an arbitrary arm i E [N]. We suppose the agent has to pay a cost X every time it activates this arm, and the objective is to find policy æ IT S (A) (i.e., determine whether to activate this arm after observing the current state s() to maximize E E (ri
)at) r]. Accordingly, we can define the Q function and the optimal Q function as (x(,,a) E Ei= 7 (rž >aa) si 8, af a, - and (s, a) max, Q7 respectively. The corresponding optimal policy can be defined n E arg max, (s,,a))
Intuitively, the higher the cost X, the less likely the optimal activation policy T, would activate the arm in given state. This intuition can be formulated as the following indexability property:
Definition 3.1 (Indexability). Let B(X) be the set of the states where the optimal policy n; activates this arm under activation cost X. Then an arm is indexable S(X) decreases monotonically from S to 0 when X increases from oc to +00.
The Whittle index fs) for given state E S is defined as the maximum activation cost À the optimal policy TŹ is willing to pay to activate the arm on state 8, i.e.,
Definition 3.2 (Whittle Index). If an arm is indexable, then its Whittle index of each state s is defined as f(s) sup (A & S())).
3

---

Published as a conference paper at ICLR 2024
The Whittle index can be determined by estimating Q values owing to the following property:
Proposition 3.3 (Learnability). an arm is indexable, then: 1) when À < f(s), (x 1) (x(s0))
2) when [()), (x < (s, 0): 3) when À = [(s), (x 1) = (x(,
Given a global state 8 (81, $2, 8N), the Whittle index policy will first find Xg such that EE I{f(s;) 2 >g) = .M and then activate the arms with fs,) 2 Ag-
4 INVENTORY MANAGEMENT AND WIMS
In this section, we will first introduce the multi-agent reinforcement learning (MARL) formulation of the inventory management problem. Then, we will adapt the Whittle index policy to our setting by bridging the gaps between restless bandits and the MARL formulation. Specifically, we formulate the definition Whittle Index for Multiple actions and State constraint (WIMS), and obtain some of its properties. At last, calculate WIMS and the corresponding policy on simplified inventory management task for illustration.
4.1 PROBLEM FORMULATION
In this paper; we formulate the inventory management problem considered as follows: There are totally N agents, and each agent represents a single SKU. At the i-th time step, the policy receives the local states of N agents {s]] while local state of contains the current features of the i-th SKU, e.g., the resource occupation, the procurement/sellnng prices, the historical demands, the inventory/in-traat
levels, etc. Then, the policy needs to decide the local actions {aih for all the N SKUs, i.e., the indices of list of candidate quantities for replenishment orders. We denote the local state and action space of each agent as S and A == [A] respectively. and denote the joint state & (ot.... x S and the joint action as âr (at
at)) Á AN Then, the environment returns a reward r, ~ R(5,, (z)) and transits to the next state 8t+1 ~ P(11s,, az).
The N agents are coupled weakly only through a global constraint Σ I(s{) < Lr, where ((s)) the resource occupation of the i-th agent (or the inventory of the i-th SKU) at the 1-th time step, and L, is the budget level at the t-th time step. In other words, without this constraint, we can decompose the reward and the transition dynamics as R(sr, ae) = Σ Idde ot) and P(.|Se: 0x) = I (dls at) for some decomposed reward Fddee and transition dynamics pdse.
4.2 WHITTLE INDEX WITH MULTIPLE ACTIONS AND STATE CONSTRAINT (WIMS)
In this subsection, we only consider one of the agents in the aulti-agent system and omit the superscript i [N] for simplicity.
We suppose that there cost À for the occupation of each unit of shared resource (in comparison, the Whittle index policy uses À to represent the activation cost of each arm). The Q function for given policy æ and the optimal Q function are defined as
7/r (ls.)) s, on a (ss = E Σ
(s. max (s, (I) x
for all s S, a E A where the expectation is taken over all possible trajectories collected by rolling out the policy ग. According to Puterraan 1990), there exists a deterministic optimal policy for any given MDP, and we denote this deterministic policy as कं
A. i.e., (s) E arg max, x QI a).
Similarly, we can define the indexability condition for our setting.
Definition 4.1 (Indexability). For all a E [A-
- 1). let S. (A) be the set of the states where the optimal under unit inventory cost X. Then an agent in the multi-agent system 4.1 is indexable for any a, (A( decreases monotonically from S to 0 when increases from 0o +00.
policy chooses action
formulated in Section
Lemma 4.2 (Sufficient condition of indexability). An agent in the multi- agent system is indexable if it satisfies the following action monotonicity condition: for any local state E and two costs A1 X2, the optimal policies n; Xx,- nx₂ of this agent satisfy Tx, (s) S m)g()).
4

---

Published as a conference paper at ICLR 2024
-
0
0
moadd
0/94 -2-
(107))
0297.a-2
02002
WHMS MaY
Optimal Daseatodk
Apree
Figure 1: (a) Illustration how WIMS is calculated based the values learned in the simplified inventory management task. We show the case on state s¹ = (i. 0, 8) and the Q values are normalized by subtracting Q (s's', = 2) from (s')',) where
1, 3 indicates ordering 0, 1, units for replenishment respectively and À represents the unit inventory cost. (b) Illustration on how to select actions based on WIMS and given Xg. The bars indicate the WIMS of two agents on different local states. The red bars indicate the values of []()]]2 and the green bars indicate the values of [](s]] (c) The WIMS policy combined with different >g results in near-optimal policies under different budget levels. Each triangle represents the performance of the WIMS policies with different Xg. The black and blue dotted lines represent the best attainable reward under different inventory levels and the performance of the best base-stock policies.
Lemma 4.2 indicates that the above action monotomicity condition can lead to indexability naturally, and its proof is deferred to Appendix B.I
We provide intuition on why the condition is natural for inventory management in Appendix B.2 Roughly speaking, the condition requires that an optimal
policy should not reduce the replenishment quantity when the inventory cost À decreases.
Since we have A candidate actions, we define the Whittle index as vector, i.e. f(s) (A1,..., Aa-1), 1). where the a-th element represents the maximum inventory cost the optimal policy willing to pay to choose an action from {ala > a} C Its formal definition is given below.
Definition 4.3 (WIMS). If an agent is indexable, then the WIMS of each state is a vector f(() such that its a-th term is [f)]] :- supx(A= E (a))), Va E 1|.
By observing S. (A) _ S.d (A) af a, we can obtain the following proposition: Proposition 4.4. [f 2 [f())]2 2 ... A- any state s.
The WIMS values can be determined by comparing the following two types of aggregated Q values: (s j) max (x),,)) aS}
Q?(s, k) max (*)
a>k
Lemma 4.5 (Learnability of WIMS). If an agent in the multi-agent system is indexable, for any
E S and a E [A 11. we have that: 1) when X < [f()] (2 s, a) 2 (7, S a): 2) when
A [s()) (s a) < (7 S a): 3) when
[f()] Q7 a) (s, a).
This lemma leads to the update rule that we later use to determine WIMS Section S
Given global state 5; (8), 88
NN and global cost Ag. the WIMS policy takes the action
Of E 1[f(s))) N₂ +1
for each agent indexed by i, i.e., selecting the maximum action whose corresponding affordable cost ([f(s')) exceeds Ag. The global cost Ag is determined by the constraint level L, and Xg can be updated dynamically in an online manner: we decrease >g when the constraint is not violate and otherwise increase More details will be introduced in Section
Lemma 4.6 (Asymptotic optimality of the WIMS policy). When Le + oo, the WIMS policy with the global cost >g = 0 is optimal.
The proofs of Lemma 4.5 and 4.6 are deferred to Appendix B.3 and B.4 respectively.
4.3 WIMS FOR INVENTORY MANAGEMENT: TOY EXAMPLE
To justify the formulation of WIMS, we design tabular inventory management task to illustrate how the policies based on WIMS can used to solve the inventory management problem and indicate the optimality gap of these policies for inventory management.
5

---

Published as a conference paper at ICLR 2024
We construct a minimum tabular example .N = SKUs with stochastic demand and stochastic VLT as follows: The state of the i-th SKU s' = (i) consists of the index and two integers representing the in-stock quantity and the in- quantity respectively. We incorporate the index into the state features to allow for the reward/dynamics of the two SKUs to be different. For compact state representation, we clip the integers to [0, 9) and thus there are S = 200 possible local states. The action space A = [3] with the list of candidate replenishing quantities to be {0, 1, 2}. We set the unit profit, the order cost, the unit backlog cost to p (10, 20), o = (10, 10), and 6 (3,3) respectively where the two numbers stand for the values for two SKUs. We consider a constraint on the total inventory capacity, i.e., Ei= 1(x*) S I. with I(s*) = / and I. to be constant. The demand on each day is sampled uniformly from {0, 2, and each unit of in in-transit product arrives w.p. 1/3 on each day. We defer the details of the inventory management task in Appendix [
We solve this simplified task by first learn Q; for different discretized As using Q-learning and calculate the WIMS indices following Definition 4.3 Then, by using different Ag. we can obtain set of WIMS policies which will be shown to be near optimal for different budget levels.
How to calculate WIMS? To illustrate this, we fix local state see how WIMS is calculated based on the learned Q values. We fix the local state s¹ = (i, 0, 8) for the two SKUs and show the Q values under different Äs in Figure I a). Owing the action monotonicity condition, the optimal action on a given state decreases monotonically as À increases. Therefore, when À changes from -00 to +oc, the maximum value changes from the Q value of the largest action (cf. the green lines) to that of the smallest action (cf., the red lines) monotonically. This ensures that there are at most A - 1 critical points which are defined as WIMS. Note that it is possible that some action is never optimal for any A. For example, for the first SKU, the optimal action is a = 3 (green solid) when the unit inventory cost 0.39: whereas the optimal action becomes a (red solid) when the unit inventory cost A >> 0.39. In this case, the two dimensions of WIMS on this state for the first SKU are both 0.39 (this indicates that a = 2 never optimal on this state). For the second SKU, when the unit inventory cost A increases, the optimal action transits = 3 + 2 + 1 and [f = 0.56, [f(s))] the two critical points for the transition. Intuitively, WIMS can be interpreted as the priority (or the maximum affordable cost) assigned to each local state for ordering at larger quantity for that SKU. Since the second SKU is more profitable than the first SKU, it is reasonable to see that the WIMS for the second SKU is larger than that of the first SKU.
1.46 are
How to implement the WIMS policy? When a global cost X is given, agent i selects action Eoes (())) 2 +1 which is the same as the largest action a ++1 as long as the corresponding WIMS [f('')]a on the local state s (i.e.. the maximum affordable cost) is larger than the given Ag- By using different X, the policy can trade off between the cumulative reward and the constraint level. We illustrate this process in Figure [b)) where we consider two states for each of the two agents and three actions. The three actions can be selected based on relationship between the WIMS vector of the local state and the given Ag. For example, when [f() ]]2 < [f(s) < Ag we choose a 0 1 = 1; and when [f(((,))le < >g < [f(( )1 we choose a² -1+1=3
The optimality of the WIMS policy. To estimate the optimality gap of the WIMS policy, we compare it with the optimal policies and the best base-stock policies (Clark & Scari, 1960). The optimal policies under different inventory levels are obtained by searching using the genetic algorithm from pymoo Blank & Deb 2020). Note that, we search for the global optimal policy for each budget level, whereas our WIMS policy only needs to learn for local agents and obtains the indices that work for different inventory cost levels. For comparison, we also evaluate the performance of base-stock policies which is a popular method from operation research. Given a base-stock level b' for each agent, the policy orders max(b' I' T .) units for the agent in state s². We iterate over all possible base-stock levels (b2, (2) and record the best performance of these base-stock policies under different budget levels (i.e., the Pareto frontier). We present the results in Figure [ee. We can see that, while having the advantages of computational efficiency, our algorithm outperforms the best base-stock policies and performs on par with the globally optimal policies.
One advantage of the WIMS policy over other MARL algorithms is that it only needs to learn the indices once for each SKU to obtain good policies over different constraint levels. In contrast, to adopt popular MARL algorithms (such as IQL (Tan 1993)). we first need to transform the hard constraint into penalty terms in the reward function and re-train the model when the constraint
Discussion.

---

Published as a conference paper at ICLR 2024
Algorithm 1 WIMSN-C: Learn the WIMS policy with neural network on continuous À
1: Initialize: The Q network Qo : S X - R and the Whittle index network ʃa SS - RA-1
2: Phase 1: Train agents independently without any constraint
3: Initialize the replay buffer with warming-up samples collected by the random policy
4: for each iteration do
5: Select 8 ~ 'D and a E [A - 1] uniformly at random and obtain À t [fa(s)]
6: Roll out c-greedy policy of (x(()) == arg max, Qo(:, A) and store new transitions to D
7: Sample a batch B := {ss, a, r, s')} from D
8: Update the Q-network and the Whittle index network with (4) and (5) respectively B
9: end for
Phase 2: Execute agents jointly with possibly changing global constraint
11: Initialize Åg with reasonably good prior value
12: for each step do
13: Take a² = Ea [[[fa(s)]a 2 }aA + 1 for each agent i E [N]
14: Update Xg following (6)
15:
end for
level changes. For this reason, we do not compare WIMS with MARL baselines in this simplified task and defer the comparison to the experiments on real inventory management tasks in Section 6
5 PRACTICAL ALGORITHM
In this section, we will introduce practical algorithm called WIMSN (as shown in Algorithm I that learns the WIMS policy using neural networks. In the training phase, we train two networks: the Q network Qo S x A x IR + E that estimates the Q value given local state s, action and unit inventory cost X, and the Whittle index network fo : S - R A-1 that outputs the Whittle index vector for each local state (With slight abuse of notation, we move the subscript the input variable of Q in this section.) In the execution phase, we maintain a dynamically updated global cost Ag. which is used for making decisions combined with fa-
Training phase. In the training phase, each agent can learn independently in an environment without the global constraint. This makes our algorithm scalable to large number of agents. In our case, we learn the Q network and the Whittle index network whose parameters are shared across different agents. The network is trained through iterations, each of which consists of steps: the sample collection step (Line 5-6) and the network update step (Line 7-8).
In the sample collection step, we first select a random cost À as one of the Whittle indices of random state from the replay buffer. Then, we collect samples by taking the action that yields the maximum Q value conditioned on X, and store the transitions the replay buffer D.
In the network update step, we sample batch of transitions B from the replay buffer ' and update the networks based on B3. The Q network is updated with the following gradient update rule:
+- 0 31 V DE (s..s)))-- [o0(s a, A) (r AI(ss) + + max Qo(s' Q )
(4)
where 31 is the learning rate, and the expectation Ex is over different > = [f))] for random s E 'D and E [A- 1]. The Whittle index network is updated with the following gradient update rule:
A-1
s - Φ + 32 (3an)))))
VolJo()))- (Qo(s, a, (T((s))))) Qo(s. s (o(+)))))) (5)
where Qa a, a) = max.2a Qa(s, a, A), and Qo(s, S a, () maxaça Qa(8, A). B, is the learning rate and we use 32 < 31 as updating fo relies on well-estimated Qo (Robledo et al. (022).
Remark. The Q network Qo and the Whittle index network fa are coupled through simultancous training: On one hand, Qo is trained on the Às sampled from fo to gradually estimate more accurate Q values around these Às. On the other hand, with more accurate estimation of the Q values around the current Whittle indices, fe can be updated more precisely.
Execution phase. In the execution phase. execute the policy determined by global cost Ay- Specifically, given the global cost Ag. we select the action for each agent based on Eq. (3). We also refer the readers Figure (b) for an illustration this process.
7

---

Published as a conference paper at ICLR 2024
The global cost X is updated dynamically. The update rule is as follows:
1 100 + 8a( Lo), with 0 == max t-TerSt
E(())
(6)
where 21 is the prior value, (,(0) is the maximum global resource occupation over window 7, and Gy is the learning rate. When the budget is not fully utilized (i.e., [(() L, < 0), we decrease the global cost such that the agents are more willing to take aggressive actions (e.g., order more quantities for replenishment in inventory management). Otherwise, if the constraint is violated, we will increase the global cost so that the agents will not order too many quantities for replenishment afterwards.
6 EXPERIMENTS
In this section, we design experiments to 1) compare the performance of WIMSN with other baseline MARL algorithms and OR methods with fixed budget levels; and 2) evaluate how WIMSN adapts when the budget level changes. We provide the implementation details in Appendix C
Settings. We evaluate WIMSN in several inventory management tasks with different scales (i.e., different number of SKUs) and different budget levels (by changing the inventory capacity limit). Our experiment is conducted on a simulator that can simulate the replenishment procedure for multiple SKUs in one store. Instead of sampling demands from some hypothetical distributions, we replay the demand series from authentic dataset from a retail partner that contains the sales history of four years (2018-20211 with 2307 SKUs. We hold out the data of the last 200 days from the training set and split these data into evaluation set (for parameter tuning) and testing set (for reporting the performance) respectively. In the test procedure, when the total in-stock quantity exceeds the inventory capacity, we reduce equal amount of in-stock quantities of different SKUs such that the total in-stock quantity equals the capacity limit. The dropped quantities do not contribute any reward (neither income nor penalty), just as they have never been replenished. See Appendix [ for details.
OR-Based Methods. We compare our WIMSN policy with two popular operation research (OR) methods: the base-stock policy (Clark & Scart 1960, Kaplan 1970) and the (s. S)-policy (Ehrhardi) 1984). In these methods, we need to specify few parameters to control inventory replenishment. In the baseline version, we find the optimal parameters for each SKU on the training set. In the skyline version, we use those parameters that are optimal ad-hoc on the lesting set. See Appendix C.I for the notion of optimality and implementation details.
Baseline MARL Algorithms. To apply off-the-shelff MARL algorithms for our constrained MARL setting, we add penalty to reward when the constraint is violated during the training (see also Appendix C.2 We choose IPPO de Witt et al. 2020) and independent Q-learning (Tan 1993) with neural networks (Mnih et al. 2013) (IDQN) as the baselines. We do not compare with other MARL algorithms (e.g.. MAPPO Yu et al. 2021) that require to learn a joint critic for all agents since they are not computationally efficient in our environment with thousands of agents. In addition, we compare with Chen et al. (202T) which updates a global inventory cost for the constrained MARL problem for each budget level. Note that all these MARL baselines need to re-train the model when the budget level or the set of SKUs changes, but WIMS uses the same model for all tasks.
Versions of WIMSN. WIMSN-c intertwines the learning of the two networks (i.e., the Q network is trained on different values of continuous À sampled from the output of the Whittle index network). It is also possible to decouple them by only training the two networks on grid of discretized As, resulting in WIMSN-d with details introduced in Appendix C.4
Experiment Results. We evaluate the above methods on inventory management tasks with 100, 1000, and 2307 SKUs and different capacity limits, and present the results Table Here are some of our observations. First, IPPO and IDQN tend to violate the constraint limit when the budget is tigh(" In contrast, WIMSN incurs less constraint violation (< 1% for all tasks) owing to our constrained optimization formulation. Second, WIMSN earns more profit with less constraint violation than previous methods, especially with a large number of SKUs. This indicates the effectiveness of WIMSN in coordinating a large number of agents. Third, those baselines with a higher profit than
Indeed, we can further increase the constraint violation penalty in the reward function to reduce constraint violation for these MARI algorithms, but this causes instability during the training.
8

---

Published as a conference paper at ICLR 2024
OR Bared Metlods
RL. Busel Meiiods Primal-Deal
Reseline: ISS
Roucine:
SS
Skylin
RS
Styline 5S
IPPO
IDQN
WIMSN-
WIMSN
puo0i
VIO
peots
V10
prodi
V00
poofs
V10
pooli
V1O
protz
V10
pood:
V10
prodi
V30
peofi:
VIO
N100-C2.5k
7.52
20.88%
8.93
16.66%
10.25
6.03%
10.29
7.10%
5.44
38.21%
3.94
34.79%
5.85
6.75%
9.10
0.21%
5.84
0.765
NJ00 C3
11.38
6.45%
12.91
4.78%
14.16
0.00%
13:91
0.005
13.53
25-43%
12.67
24,07%
11.00
0.52%
12.81
0.085
12.38
0L0ESS
3.96%
13:22
1.30%%
16.39
0.10%
16.90
0.0%
16.52%
16-23
16.91%
14:84
16.91
0.00%
12:13
0.08
17.81
0.006
16.95
3.42%
18.01
0.003
0,25
NTL-C25
C55
13:55
1.21%
16.10
14.37
0005
8.97%
11.37
9.31%
10:91
13.27
01005
NIL-CIOL
$4.58
0.00%
15663
0.00%
15.00
0.00%
16.01
0.00%
16.05
4.50%
12.30
5.32%
16.00
0.00%
16.94
0.00%
1704
000%
15.90
0101%
16.45
0.015
15.00
0.01%
16.51
0006
16.55
0.88
16.34
00005
15:87
0.01%
16:41
16.36
0008
N2L-C5R
8.95
5.48%
9.70
4.65%
9.43
0.0%%
9.57
0.00%
5.45
5.9%
6:82
7.27%
8.86
0.00%
9.13
0.00%
9.50
0.05
11:40
0.87%
11:85
0.16/
14
0101%
12:19
000%
9.6
0.88
5:18
0.095
11:11
0.41%
01001
12.20
NZL C231L
12.99
0.09%
1341
0.006
12.35
0.00%
12.67
12.55
0.025
N36-1611
Table 1: The daily profit per SKU in x10 USD and the ratio of violation on the capacity limit of different algorithms in different inventory management tasks on held-out testing data. For the tasks. "N" indicates the number of SKUs to control and "C" indicates the capacity limit on the shared inventory (i.e., the budget level L). We compare WIMSN with RR-based methods and RL-based methods: BS (base-stock policy), SS ((s, -poolicy, IPPO (de Witt et al. 2020), IDQN (Tan 1993]. Primal-Dual (Chen et al. 2021). WIMSN-d (with discrete A8), and WIMSN-C (with continuous As). See details in Section O WIMSN models use the same model for all tasks with the different "N" and "C" while others train separate models for different tasks. We present the average over seeds. We underline Parato optimal solutions and color Parato optimal solutions with constraint violation < 1% (excluding skyline BS and skyline SS since they are ad-hoc solutions).
WIMSN always incur higher constraint violation, which mcans they may behave worse in reality. (In our experiment, constraint violation does not incur any loss, but in reality, higher constraint violation may lead to additional loss.) Besides, unlike the other methods, WIMSN does not need to re-train the model for different tasks, therefore greatly reducing the computational costs (due to space limit, the computational costs of different methods and the variance across different seeds are provided in Appendix D)
Online Adaptation for Capacity Limit. To eval- uate the adaptability of WIMSN to the changing budget level. we select 100-day period to con-
Accumulated Prom: (-100 Dola3)
1200
IPPNO ICTUUR)
mAK 2BR max VIO-IK VIO-IK
IPPO (C50K)
trol 1000 SKUs and change the capacity from 50k
1900
WINSM-C
Adaptive Ay MINN
to 100k on the 50th day. We present the cumula- tive profit per SKU and the maximum constraint
violation over this period in Figure 2 pare WIMSN-c with two IPPO models trained under the capacity limit 100k and 50k respectively (i.e.. C100k/C50k). We observe that the IPPO (C100k) can overflow in the first 50 days and per- form poorly due to the inconsistency between ca- pacity limits used in training (100k) and testing (50k), whereas the IPPO (C50k) does not perform good enough for the last 50 days due to overly conservative replenishment strategy. In contract, WIMSN-c performs well across all the time pe- riod by adjusting Xy adaptively according the current capacity limit (cf. the purple curves in Figure 2) When the capacity tight, A fluctuates around 4.2; and when the capacity doubles, X adapts quickly and decreases to values around 3.6 to allow agents to take more aggressive actions.
We com-
mmmm Zapactty-100 10nk
Dapachty=50k
Testing Days
On the task where the capacity limit changes from 50k to 100k on the 50th day, WIMSN-c earns 22.7% and 18.4% more cumula- tive profit than IPPO (C100k) and IPPO (C50k) respectively while incurring significantly less constraint violation than IPPO (C100k).
Figure 2:
7 CONCLUSION
We proposed WIMS (Whittle Index with Multiple actions and State constraint). a novel form of Whittle Index that extends to the multi-agent reinforcement learning (MARL) setting with multiple discrete actions and a global constraint on the state space. This MARL setting is suitable for many industrial tasks such as inventory management where each agent chooses a replenishing quantity level for the corresponding nock-keeping-unii (SKU) such that the total profit is maximized while the total inventory does not exceed a certain limit. Based on WIMS, we propose a deep MARL algorithm called WIMSN for inventory management. Empirically, WIMSN is computationally efficient (without the need to re-train the model when the constraint limit the combinition of SKUs changes), scalable (to thousands of agents), and high- segh-eerfoning (surpassing previous OR and MARL baselines especially hard tasks with tight constraints).

---

Published as a conference paper at ICLR 2024
REFERENCES
Souvik Barat, Harshad Khadilkar, Hardik Meisheri, Vinay Kulkarni, Vinita Baniwal, Prashant Kumar, and Monika Gajrani. Actor based simulation for closed loop control of supply chain using reinforcement learning. In Proceedings of the 18th international conference on autonomous agents and multiagent systems, pp. 1802-1804, 2019.
Blank and K. Deb. pymoo: Multi-objective optimization in python. IEEE Access, 8:89497-89509, 2020.
Fangruo Chen. Optimal policies for fions research, 48(3):376-389 2000.
inventory problems with batch ordering. Opera-
Wenbo Chen and Huixiao Yang. A heuristic based on quadratic approximation for dual sourcing problem with general lead times and supply capacity uncertainty. IISE Transactions, 51(9): 943-956, 2019. ISSN 24725862. doi: 10.108002472555420188.....
Yi Chen, Jing Dong, and Zhaoran Wang. A primal-dual approach to constrained markov decision processes, arXiv preprint arXiv:2101.10895, 2021.
Andrew J Clark and Herbert Scarf. Optimal policies for multi-echelon inventory problem. Manage- ment science, 6(4):475 490 1960.
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:20/ 1.09533, 2020.
Yuandong Ding, Mingxiao Feng. Guozi Liu, Wei Jiang, Chuheng Zhang, Li 7hao, Lei Song, Houqiang Li, Yan Jin, and Jiang Bian. Multi-agent reinforcement learning with shared resources for inventory management. In Advances in Neural Information Processing RLARealLife Workshop, 2022.
Scott Grant Eckert. Inventory management and its effects on customer satisfaction. Journal of Business and public Policy, 133:1-113, 2007.
Richard Ehrhardt. (s, s) policies for a dynamic inventory model with stochastic lead times. Operations Research, (2(1):121111) 1984.
Jiarui Fang, Lei Zhao, Jan C. Fransoo, and Tom Van Woensel. Sourcing strategies in supply risk management: An approximate dynamic programming approach. Computers Operations Research, 40(5):1371-1382, 2013. ISSN 0305-0548. doi: http::/00i.ngg111..01//....0.222 016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Yoichiro Fukuda. Optimal Policies for the Inventory Problem with Negotiable Leadtime. Management Science, 708, 1964. ISSN 0025-1909. doi: 10. 10.1288/mmsc.10
larii Giannoccaro and Pierpaolo Pontrandolfo. Inventory management in supply chains: reinforce- ment learning approach. International Journal of Production Economics, 78(2):153-1611 2002.
ISSN 09255273. doi: 10.1111660092--573300000000
Joren Gijsbrechts, Robert N. Boute, Jan Albert Van Mieghem, and Dennis Zhang. Can Deep Reinforcement Learning Improve Inventory Management? Performance and Implementation of Dual Sourcing- Problems. SSRN Electronic Journal, pp. 1-33, 2019. ISSN 1556-5068. doi: (0.21399ssm.33328811.
David A Goldberg, Dmitriy A Ratozhoogzz,,.ov. Yingdong Lu, Mayank Sharma, and Mark Squillante. Asymptotic optimality of constant-orde policies for lost sales inventory models with large lead times. Mathematics of Operations Research, 41(3):898 (11388988911, 2016.
10

---

Published as a conference paper at ICLR 2024
Nir Halman, Diego Klabjan, Mohamed Mostagir, Jim Orlin, and David Simchi-Levi. A fully polynomial-time approximation scheme for single-item stochastic inventory control with discrete demand. Mathematics of Operations Research, 34:674-685, 2009.
David Hodge and Kevin D Glazebrook. On the asymptotic optimality of greedy index heuristics for multi-action restless bandits. Advances in Applied Probability, 47(3):652-667, 2015.
Schuyler W Huck, Tracey I. Cross, and Sheldon Clark. Overcoming misconceptions about 7-scores. Teaching Statistics, 8(2):38-4) 40, 1986.
Woonghee Tim Hub, Ganesh Janakiraman, John A. Muckstadt, and Paat Rusmevichientong, Asymp- totic optimality of order-up-to policies in lost sales inventory systems. Management Science, 55 (3):404433) 2009. ISSN 00251909, 15265501.
Chengzhi Jiang and Zhaohan Sheng. Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system. Expent Systems with Applications, 36(3 PART 2): 6520 6526, 2009. ISSN 09574174. doi: 10. 10.10166jessaa22088.00
Nan Jiang and Alekh Agarwal. Open problem: The dependence of sample complexity lower bounds on planning horizon. In Conference On Learning Theory, pp. 3395-3398. PMLR, 2018.
Zhengyao Jiang, Dixing Xu, and Jinjun Liang. A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint urXiv:/700 10059, 2017.
Robert S Kaplan. A dynamic inventory model with stochastic lead times. Management Science, 16 (7):491-507, 1970.
Ahmet Kara and Ibrahim Dogan. Reinforcement learning approaches for specifying ordering policies of perishable inventory systems. Expert Systems with Applications, 91:150-158, 2018. ISSN 09574174. doi: 10.10166j.ewwa.217...l 0.64..
Jackson A Killian, Arpita Biswas, Sanket Shah, and Milind Tambe. Q-learning lagrange policies for multi-action restless bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 871-881, 2021a.
Jackson A Killian, Andrew Perrault, and Milind Tambe. Beyond" to act not act": Fast lagrangian approaches to general multi-action restless bandits. Proceedings of the 20th International Conference Aulonomous Agents and MultiAgent Systems, pp. 710-718, 2021b.
Jerome Le Ny, Munther Dahleh, and Eric Feron. Multi-uav dynamic routing with partial observations using restless bandit allocation indices. In 2008 American Control Conference, pp. 4220 4225. IEEE, 2008.
Alex S Leong, Arunselvan Ramaswamy, Daniel E Quevedo, Holger Karl, and Ling Shi. Deep reinforcement learning for wireless sensor scheduling syber-physice. systems. Automatica, 113: 108759, 2020.
Ran Li, Chuan Huang, Xiaoqi Qin, Shengpei Jiang, Nan Ma, and Shuguang Cui. Coexistence between task-and data-oriented communications: A whittle's index guided multi-agent reinforcement learning approach. arXiv preprint arXiv:2205.09:: 2022.
Guogang Liao, Ze Wang, Xiaoxu Wu, Xiaowen Shi, Chuheng Zhang, Yongkang Wang, Xingxing Wang, and Dong Wang. Cross dqn: Cross deep q network for ads allocation in feed. In Proceedings of the ACM Web Conference 2022, pp. 401-409, 2022.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-commptttivee environments. arXiv preprint arXiv:1706.0222: 2017.
Nicolas Meuleau, Milos Hauskrecht, Kee-Fung Kim, Leonid Peshkin, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Solving very large weakly coupled markov decision processes. In AAAI/AAI, pp. 165-172, 1998.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.5023, 2013.
11

---

Published as a conference paper at ICLR 2024
Alex Nichol and John Schulman. Reptile: scalable metalearning algorithm. arXiv preprint arXiv:1803.02999, 2(3):4, 2018.
Afshin Oroojlooyjadid, M Nazari, Lawrence Snyder, and Martin Takáč. A deep q-network for the beer game: A reinforcement learning algorithm solve inventory optimization problems. arXiv preprint arXiv: 2017.
Afshin Oroojlooyjadid, Lawrence V. Snyder, and Martin Takáč. Applying deep learning to the newsvendor problem. IISE Transactions, 52(4):444 463, 2020. ISSN 24725862. doi: 10.1080/
24775854.201991632502
Christos H Papadimitriou and John N Tsitsiklis. The complexity of optimal queueing network control. In Proceedings of IEEE 9th Annual Conference on Structure in Complexity T'heory, pp. 318-322. IEEE, 1994.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Meng Qi, Yuanyuan Shi, Yongzhi Qi, Chenxin Ma, Rong Yuan, Di Wu, and Zuo-Jun Max Shen. A practical end-to-end inventory management model with deep learning. Available at SSRN 3737780, 2020.
Francisco Robledo, Vivek S Borkar, Urtzi Ayesta, and Konstantin Avrachenkov. Tabular and deep learning of whittle index. In EWRL 2022-15th European Workshop of Reinforcement Learning, 2022.
Ming Tan. Multi auett reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning. pp. 330-337, 1993.
Shangxing Wang, Hanpeng Liu, Pedro Henrique Gomes, and Bhaskar Krishnamachari. Deep reinforcement learning for dynamic multichannel access wireless networks. IEEE Transactions Cognitive Communications and Networking, ((2):257-2655 2018.
Siwei Wang, Longbo Huang, and John Lui. Restless-ucb, efficient and low-complexity algorithm for online restless bandits. Advances in Neural Information Processing Systems, 33:1 1878-11889, 2020.
Tianshu Wei, Yanzhi Wang, and Qi Zhu. Deep reinforcement learning for building hvac control, In Proceedings of the 54th annual design automation conference 2017, pp. 1-6, 2017.
Peter Whittle. Restless bandits: Activity allocation in a changing world. Journal of applied probability, 25(A)2287-298, 1988.
Guojun Xiong, Shufan Wang, and Jian Li. Learning infinite-horizon average-reward restless multi- action bandits via index awareness. Advances in Neural Information Processing Systems, 35: 17911-17925, 2022.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo cooperative, multi-agent games. arXiv preprint arXiv:2 103.020033./95,, 2021.
12

---

Published as a conference paper at ICLR 2024
A LNVENTORY MANAGEMENT
We consider a simplified setting with one store and multiple SKUs to better focus on the constrained MARL setting considered in this paper. We further assume that there is an upstream warehouse that can fulfill all the requirements from the store. Our objective is to train a high-quality replenishing policy for each SKU in the store, particularly when there are a large number of SKUs and when the constraint on the inventory capacity is tight. Moreover, due to the flexibility of RL algorithms. our method can also be applied more complex settings, e.g., with multi-echelon structure or fluctuate supply.
Similar to previous work, we follow the multi-agent RL (MARL) framework with decentralized agents, each of which manages the inventory of one SKU in the store. We assume that the store has N SKUs in sell, all of which share common inventory capacity that can store up to L, units the same time.
Replenishing decisions of each SKU are made on discretized time steps (which are days in this paper). For the 1-th time step and the i-th SKU, we denote the in-stock quantity by 7; E Z. This quantity is constrained by the limit of the inventory capacity (i.c., the shared resources). Accordingly, the following constraint holds:
n 2 O, Σ S...
(7)
The dynamics of the system is as follows: For the (-th time step and the i-th SKU, the corresponding agent can place a replenishment order to request O; E Z units of products from its upstream warehouse. It takes several time steps (referred to the leading time LT)) before these products are delivered to the store. We denote the total units in transit at the 1-th time step by Ti E Z and the total units arrived at the f-th time step by A Z. Meanwhile, the customer demand is D and leads an actual sale of S E Z units. Due to the possibility of out-of-stock, S may be smaller than DŁ, We consider the setting where the leading time LTý and the demand Dj are stochastic.
For clearness, we summarize the dynamics these variables as follows:
S min (Di, 13)
A
Σ
0
T Tí 1141 Of41 The immediate profit of the i-th SKU is calculated according to the following equation:
1/ 4 A/+1
= g: - -,, [O] > o] - S
(8)
where Pt. qs oj, b; are the unit sale price, the unit procurement price, the order cost, and the unit backlog cost for the i-th SKU respectively, and T[-] is an indicator function which equals to one when the condition true and zero otherwise. The order cost reflects the fixed transportation cost or the order processing cost, and yields whenever the order quantity is non-zero.
The objective of this task is to find policy x that generates replenishment orders Ož for each SKU to solve the following constrained optimization problem:
N
N
max E
R
s.t.,
I < Le, Vt.
I=0
1=1
When the total in-stock quantity Σi 1i exceeds the inventory capacity L,, we reduce equal amount of in-stock quantities of different SKUs such that the total in-stock quantity equals the capacity limit. In the train procedure of baseline algorithms. we calculate the reward penalty for the dropped quantities before dropping them, and use the with-penalty reward to train the model. In the test procedure, the dropped quantities do not contribute any reward (neither income nor penalty), just as they have never been replenished. In addition, we will also record the quantity of constraint violation vio,- MM I L, and report the maximum constraint violation VIO = max, vio,-
Remark. We formulate the inventory management problem as an optimization problem with hard constraints. However, from practical perspective, it is unlikely strictly prevent constraint violation
13

---

Published as a conference paper at ICLR 2024
Notation
Explanation
N
The number of SKUs Inventory of the i-th SKU at the t-th time step
T
O
Order quantity of the i-th SKU at the t-th time step
D
Demand of the i-th SKU at the t-th time step
LT
Lead time of the order placed at the 1-th time step for the i-th SKU Sale quantity of the i-th SKU at the 1-th time step Quantity in transit of the i-th SKU at the time step Proftt generated on the i-th SKU at the 1-th time step Unit sales price of the i-th SKU Unit procurement cost of the i-th SKU Unit order cost of the i-th SKU Unit backlog cost of the i-th SKU Storage capacity of store
St
T
R
Pi
q
Oj
L
Table 2: Notation for the inventory management problem considered in this paper.
(exceeding the capacity limit) w/o reducing the profitability. Specifically, there are many practical workarounds when the inventory overflows, such as clearance sale, renting additional storage space, returning products, and these workarounds low-proitaability strategy. Therefore, our objective under this mathematical formulation is to obtain a policy such that this constraint strictly holds in most of the cases while achieving as high profit as possible.
preferable than using overly conservative and
For convenience, summarize the notations used in this paper in Table
B PROOF
B. PROOF OF LEMMA 4.2
Lemma 4.2 (action monotonicity to indexability) can be proved by contradiction as follows: We assume that the action monotonicity condition satisfied but there exists an such that (A) does not decrease monotonically (i.e., the agent is not indexable). This indicates that there exists a state E and λ1 < X2 such that s & Sa but E Sa (A2). other words, when À λ1. the optimal policy selects action on the same state. Therefore, we have कx, (8) S xx (8) which contradicts the assumption that x,
on state when À A2, the optimal policy selects an action a >
x (8).
B.2 FEASIBILITY OF ACTION MONOTONICITY CONDITION IN LEMMA
In this section, we provide an intuition why this assumption is natural for inventory management. By defining ((T) = E E [=](s)][=] where the expectation is over the initial state distribution and the randomness of the environment/policy, we can decompose Qx()) J(x) AI(T).
We can prove [(T [(T),) if A > Ag. This indicates that we would better to maintain higher inventory level when decreases, which in turn requires take larger actions order more quantities for replenishment.
Lemma B.1. I(TT,) S i( of
Proof. If (TTx)) > T(TT,), then
(A1 - -2) 1 = -()())))))~)(()))))))) )1d(TT))) x2)( 0,
i.e.
A( >xT(T)x)) - -) > -
14

---

Published as a conference paper at ICLR 2024
Thus,
2xa (Tx,) J(T )
( (I(((T)) (A1 (2x
2 J (1(((T)))) (A1 -
( (1
-22
J(xX,) - )²T(xT)))
= Qx (Ti2))
which contradicts the fact that Tx2 the optimal policy for X2.
B.3 PROOF OF LEMMA 4.5
The following proof goes for any S and [A- - 1] and arbitrary set of optimal policies कτ for À E R that define S,, and f(s).
1) If A < [f(s)]a == supx {)
(A)), we have x [f(s)]a for some 0. By Definition 4.1 when > 0, s E (A) and therefore the optimal policy т satisfies ( > a, i.e., Qx > a) (s. a)).
2) Similarly,
> [f()]aa we have s & (A) and therefore the optimal policy σ satisfies (+) S a, i.e. QT > a) S (s aa).
3) If O (s. > a) and (s, is continuous w.r.t A, it is not hard to see Qx > a) 2x a) when À [f())]a by combining the previous arguments.
Next, we prove the continuity of (1 a) which easily leads to the continuity of (i > a) and (s S a) since the maximum over several continuous functions is continuous. Suppose m and T2 are the optimal policies when À = X and À = X2 respectively. Due to optimality, we have 0% (s,a) 2 Q% (s,a) and (s, a) 2 (s.a) for any as) By the definition of (T(s,a) in D. (s, continuous w.r.t. for any (s, a), i.e. for given e 0, there exists 5 > O such that [} A2 S 8 (x, (s. - (s, S <aand 10 (s, a) (s, S c. Therefore, we have (s, a) (T) S -(
a) < < Q as) QQ (s. 2 which imply (8, - QN S for any (s, a) and completes
2% a) (s,a) the proof.
B.4 PROOF OF LEMMA 4.6
When L, co, want show that the WIMS policy with the global cost X 0 is optimal. This policy can be written as
a
Σ [[f(s)) 2 20 0] +1
Σ (s a) 2 08(s, +)
(10)
a<A
arg max (s)
where the first equation follows the definition of the WIMS policy (Eq. 3). the second equation follows Lemma 4.5 the third equation can be derived following the definition Eq. Notice that when the capacity limit is infinite, the optimal policy should always take the action arg max (8,
C IMPLEMENTATION
In addition to the WIMSN-C (that learns the WIMS policy with neural network on continuous A) presented in Algorithm] | We also design another version called WIMSN-d (that learns the WIMS policy with neural network on K discretized A1. >x) and present it in Algorithm 2 With prescribed grids of As, W decouples the inter-dependency for learning the Q network and learning the Whittle index network, and therefore is more stable. However, the performance of WIMSN-d may be limited by the discretization of À and this explains why WIMSN-c outperforms
15

---

Published as a conference paper at ICLR 2024
Algorithm 2 WIMSN-d: Learn the WIMS policy with neural network on discretized Äs
1: Given A := ..))
2: Initialize:
3: The Q network Qo S RAAK with the (a, k)-th element representing Qo a, An.)
4: The Whittle index network fo S R RA-1
5: Phase I: Train agents independently without any constraint
6: Initialize the replay buffer 'D with warming-up up samples collected by the random policy
7: for each iteration do
8: Select À t 2 for some l sampled from Unif([K])
9: Roll out the e-greedy version of the policy (x(+) arg maxa Qa(: a, 2)
10: Store new transitions to the replay buffer D
11: Sample a batch B := {(s, a, r, }')} from D
12: Update the Q Q-network following
13: Generate Whittle index labels for the batch f(s
Update the Whittle index network by SGD on ming [s) F[[?]
15: end for
16: & Step 2: Train joint agents
17: The same Algorithm [
WIMSN-d on tasks with over 1000 SKUs (cf. Table T. We provide the source code in https
//github. com/ zhangchubeng122 23WWMM
For OR-based methods and two previous MARL-based methods (IPPO and IDQN), the optimization objective in the train procedure is the cumulative profit plus reward penalty equal to 50% of the procurement cost of dropped products, which defines a consistent notion of optimality in these baselines.
C.I IMPLEMENTATION OF OR-BASED METHODS
We select two popular OR algorithms, the base-stock policy and the (s, S)-policy, our operational- research-based benchmarks.
For each SKU, the (s, S)-policy has two parameters: Spuram is the re-order point and Spurmm is the order up-to level. On each day, the policy places an order to replenish the inventory level up to the order up-to level Sparam if the quantity of the order exceeds the re-order point 5param- Specifically, the policy places an order with the quantity
S Sparm (0
I
S Sparam -
2 S param
(11)
otherwise
where O is the quantity of the replenishment order, / is the inventory level, T' is the quantity in transit. The base stock policy is special case of the (s, -)--olicy with Sparam 0.
For a task with N SKUs, the (s. S)-policy and the base stock policy need to specify 2N and N parameters respectively. Consequently, it is trivial to obtain the best of parameters on a given task, and we use genetic algorithm (GA) to search for these best parameters. First, we search for the best of shared parameters (i.e., using the same parameters for all the SKUs) and use it as the initial solutions in GA. Second, we search using GA with the population size 100 and the 1000. Third. of iterations is reached as well as the solution is a local optimum. To verify whether the solution is local optimum, we evaluate the performance of all the element-wist perturbations this solution and treat the solution local optimum only if the solution performs the best among these perturbed candidates.
minimum number of iterations
terminate the search only if the minimum number
The evaluation metric (in the train procedure) of these OR-based methods is the same as the baseline MARL algorithms for consistency, i.e., the cumulative profit plus penalty equal to 50% of the procurement cost of overflowed products. Indeed, profit by adjusting for the coefficient. However, we find that using different coefficients does not influence the dominant advantage of our algorithm.
trade off the constraint violation with the
16

---

Published as a conference paper at ICLR 2024
Hyperparameters
Values
The number of training iterations
5 10
The number of episodes collected in each iteration The number of time steps each episode The capacity (in #episodes) of the replay buffer 6 annealing scheme for the e-greedy exploration
8
100
5000
Linear annealing
annealing span for the e-greedy exploration c values for the c-greedy exploration
4 x 10th iterations
From 1.0 to 0.05
The number of layers in the Q network
4
The number of neurons in each hidden layer of the Q network
128
The number of layers in the Whittle index network
4
The number of neurons in each hidden layer of the Whittle index network
128
Discount rate y
0.985
Optimizer
PMSProp 5 x 10
Learning rate for the Q network Learning rate for the Whittle index network
1 x 10-4
Gradient clip
10
Table 3: The hyperparameters of WIMSN
C.2 IMPLEMENTATION OF BASELINE MARL ALGORITHMS
For these MARL algorithms that do not follow the constrained MDP formulation, we penalize the violation of the constraint into the immediate profit to help the algorithms account for the constraint. Specifically, if Σ I > L for some time step t, we penalize each agent (indexed by j) with an additional cost which equals to 0.5 x N (Ei") T - I.) x 4 where d is the unit procurement price of the j-th SKU. We train the agents using individual rewards (IR) which provides penalty-regularized reward to each agent, and we let all the agents amortize the overflowing penalty
As mentioned before, in the train procedure, these MARL baselines use the reward function as the cumulative profit plus a penalty equal to 50% of the procurement cost of overflowed products. This penalty coefficient trades off the constraint violation with the profit. We observe that these MARL baselines result in signification constraint violation, which calls for larger penalty coefficient. However, find that further increasing the coefficient will lead to instability, and this is the reason why we set the coefficient to 50%.
For the MARL baselines and our WIMSN implementation, additionally use the following techniques to improve the stability and the generalization of the model:
Observation normalization: Since different SKUs may vary greatly in prices, quantities, and demands, we normalize different features in the observation individually. Specifically, we divide the features with price/quantity/dmmadd as the unit with the selling price/the storage capacity/the average demand over the past 21 days respectively. For example, the feature "in-stock quantity" is equal 500 with the "quantity" unit and the storage capacity is 5000. Then, we normalize the feature to 500/5000 = 0.1.
Action normalization: Since different SKUs may vary greatly in the range of the demands, we use the quantity of the replenishment orders as the multiply of the action and the average demand of the corresponding SKU over the past 21 days. This greatly improve the generalization of the model over different SKUs.
State/reward normalization: We calculate the standard score (or z-score) (Huck al. 1986) of the state and the reward using the rolling average and standard deviation during the training.
Loss normalization: For WIMSN-d, since the Q values for different As have different value ranges, we normalize the loss separately for different dimensions of X.
We have also tried use team reward (TR) which is free of the penalty assignment problem. However, the experiment results suggested that using IR is better than equal using TR.
17

---

Published as a conference paper at ICLR 2024
Testing
MARL- Methods
Performance
Baselines
Ours
IPPO
IDQN
Primal Tralll-Dual
WIMSN-d
WIMSN-C
NT00-C2.5K
391
435
420
N100-C5k
400
432
423
N100-C1OK
390
432
428
N100-C20K
397
430
425
NTK-C25k
1222
1959
1207
2652 in total
2575 in total
NIK-C50k
1228
2061
1247
(One model
(One model for all tasks)
NIk-C100k
1227
2061
1183
for all tasks)
NIK-C200k
1190
2021
1253
N2K-C58K
1567
2728
1602
N2k-C115k
1618
2858
1689
N2k-C23IK
1593
2847
1549
N2k-C461k
1548
2703
1612
Table 4: The computational cost of different algorithms in different inventory management tasks with different numbers of SKUs (indicated by "N"") and capacity limits (indicated by "C"). See detailed description of the algorithms in comparison in Section minutes) of all the corresponding runs in [ Note that in WIMSN-c or WIMSN-d, we use the same model for all different tasks with different numbers of SKUs and capacity limits, this greatly reduces the computational costs of WIMSN-c and WIMSN-d.
We present the average wall time (in
C.3 IMPLEMENTATION OF WIMSN-C
For WIMSN-c and WIMSN-d, we only train one model (i.e., the Whittle index network) on all 2307 SKUs and use it for evaluation on all tasks with different combinations of SKUs and capacity limits. WIMSN-c is implemented based on the code the IDQN implementation, and we list the hyperparameters in Table
Thanks to Proposition 4.4, the outcome of this rule is equivalent to the rule we used in our algorithm. However, find that (3) more robust since the output of the Whittle index network may sometimes violate Proposition 4.4 slightly and it results in better empirical performance.
When initializing the Whittle index network, we add constant offset Aconst 4 the output. The offset chosen as reasonable prior for the cost which should correspond to the maximum affordable unit storage in typical supply chain. This trick makes the networks are trained initially over reasonable range of As.
C.4 IMPLEMENTATION OF WIMSN-D
We use K 51 and (AI...... ,A) = np wwnspaceeeeecceecceeeecc.e (0, 10, 51).
Given state s, the Q network outputs a matrix Q E BAXK, The Whittle index label for this sample f can be calculated as follows:
Q max forward np. wpm.maimmmm.mmm accumulate (Q, axis=0)
Q_ max_backwardd np. www.mummmmmmmmmmmmmmmmmmmmmmm.mm cocmulatee@ Q dii ff = np.abs abs (Q_max_forwaale
axis=0) [::-1]
2 q_max_backward
np.argmin (Q_diff, axis=1) [:-1]
D EXTENDED EXPERIMENT RESULTS
We show the computational costs (in terms of the wall time in minutes) of different algorithms in Table 4 We run our experiments on a machine with 24x Intel Xeon CPU ES-2690 CPU, 1x NVIDIA V100 16G GPU, and 512G memory. We also show the standard deviation of the daily profit per SKU for different MARL-based algorithms in T
This dataset used in this paper is provided by one of our retail partners. The experiments based on the history of 2307 SKUs from 2018-08-01 to 2019-12-13. The training validation testing sets are to be the subset 2015 8-08-01 to 2019 9-0-5--27 (300 days) 2019-05-28 to 2019-09-04 (100 days)
18

---

Published as a conference paper at ICLR 2024
Testing
MARL -Base Methods
Performance
Baselines
Ours
IPPO
IDQN
Primal-Dual
WIMSN-d
WIMSN-C
N100-C2.5K
0.07
0.06
0.17
0.03
0.03
N100-C5k
0.39
0.41
0.85
0.15
0.20
N100-C10K
0.07
0.08
0.07
0.01
0.07
N100-C20K
0.01
0.07
0.08
0.00
0.07
NIK-C25K
0.46
0.43
0,36
0.14
0.15
NIK-C50K
0.67
0.78
0,93
0.23
0.40
NIK-C100K
0.44
0.02
0,38
0.10
0.36
NIK-C200k
0.23
0.34
0.20
0.14
0.10
N2K-C58k
1.67
0.98
0.50
0.08
0.01
N2K-C115K
1.28
1.42
0.80
0.39
0.46
N2k-C23IL
0.47
0.33
0.15
0.45
0.27
N2k-C461k
0.13
0.30
0.01
0.18
0.19
Table 5: The standard deviation across different seeds on the daily profit per SKU for different algorithms in different inventory management tasks with different number of SKUs (indicated by "N") and capacity limit (indicated by "C").
2019-09-055 to 2019-12-13 (100 days) respectively. For the experiments on 100/1000 SKUs, we sample subset of SKUs uniformly randomly from the 2307 SKUs.
19


Reviews:
[{"review_id": "TcIqvY4KRS", "reviewer": "Reviewer_spvh", "summary": "This paper extended Whittle index method to a multi-agent reinforcement learning setting under the inventory management setup with multiple discrete actions (number of replenishments) and a global constraint on the state space (total inventory not exceeds a certain limit). The paper bridges two gaps to the restless bandits problem: the constraint is imposed on the state instead of the actions; there are multiple actions for each agent instead of binary actions; by measuring the cost of unit budget consumption and generates the critical points for different actions (changes index into vectors). Then the author proposed a algorithm combines their WIMS with a neural network to solve problem more efficiently. The real data experiments shows that the new policy performs good and efficiently with less constraint violations compared to existing policies.", "questions": "-\tBased on the above mentioned weaknesses, I feel that the comparison in Figure 2 does not complete or fair enough.  The author compared their adaptive algorithm with two IPPO models trained with two different capacity limits, but the author didn’t mention any training cost or training time for IPPO, so I am wondering why the IPPO users can’t re-train their models on the 50th day, since the reward was compared on a daily manner. \n-\tIn the neural network training phase, in the network update steps, they sample a batch of transitions from the replay buffer, could this word “batch” be more clarified? And is this replay buffer something important to computational efficient of the algorithm? What is the difference from regular learning algorithms that runs a sequence of data first and without a replay buffer?\n-\tOR-based methods was first mentioned without explanation, should add an abbreviation after first mentioning operation research.\n-\tIt's unclear from the given text what dataset was used for training and testing the proposed methods. This raises concerns about the generalizability and robustness of the proposed models. Are the datasets representative of real-world scenarios? Are they publicly available for verification and reproducibility?\n-\tWhile the paper mentions computational costs and standard deviations of daily profits, are there other metrics (like fill rate, stockout rate, etc.) used in inventory management that could provide a more comprehensive view of performance?\n-\t(Typo in Proposition 3.3) In the third case, Q(s,1) = Q(s,i) -> Q(s,1) = Q(s,0)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-\tThis paper introduces a novel adaptation of the Whittle Index tailored for a MARL setting. The adaptation addresses challenges that previously hindered the direct application of the Whittle Index such as multiple actions for each agent and constraints on state spaces instead of action spaces. The application of WIMS in a deep MARL algorithm, called WIMSN, further establishes its originality.\n-\tThis paper compares the proposed algorithm with both traditional operation research methods and other MARL baselines, providing a comprehensive evaluation. The adaptive nature of WIMSN to changes in constraints or combinations of SKUs without requiring retraining emphasizes the quality and flexibility of the proposed approach.\n-\tThis paper is well-structured and organized. Technical concepts such as WIMS are well explained using simple examples, making them easily accessible to readers. \n-\tThe ability to scale WIMSN to thousands of agents highlight its significant, especially in large-scale industrial scenarios. \n-\tThis paper is original in that it measures the cost of unit budget consumption and generates the whittle index to be a vector. Furthermore, the author gave a reasonable sufficient condition of indexability for the vector whittle index, (which is the optimal policy should not reduce the replenishment quantity when the inventory cost decreases.)\n-\tThis paper has high contributions since the methods generally applied to other problems where the global constraint of a multi-agent learning problem depends on states instead of on actions.", "weaknesses": "-\tThis paper builds upon the Whittle index, but there is no comprehensive exploration of the inherent limitations or challenges of using this index in a MARL context. This oversight could result in practical challenges or unintended outcomes when transitioning to real-world implementations. It would be beneficial for the authors to outline specific assumptions made when integrating the Whittle Index. One potential area of exploration could be the interplay of local constraints in conjunction with global constraints. Some SKUs might have their own unique storage.\n-\tThis paper does not have details on how the dataset is partitioned into training, validation, and testing subsets. Providing this information is essential for ensuring replicability and comprehending the robustness of the outcomes. It would be beneficial for the authors to explicitly define and justify their splitting methodology, particularly given the time series nature of the data.\n-\tIt would be helpful if the authors mentioned areas where their method (WIMSN) could be improved or further developed. By discussing potential future studies or enhancements, the authors would make their paper even more valuable.\n-\tThe literature review seems not very complete and up-to-date. The author reviewed a lot of works on the operation research and on independent learning based MARL without global constraints, and the main comparisons were made with works in 217,2018. I am wondering if there are more recent works focus on similar topic.\n-\tOne of the focuses, main assumptions and challenges in this paper, which is the limit of the total inventory capacity changes across the different seasons is not very reasonable to me. If the constraint is only changed once a long time like once a seasons, it seems to me that wrong those previous algorithms and re-train the model does not bring many troubles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CxZKIXDfqw", "reviewer": "Reviewer_xHVA", "summary": "This paper considered a multi-agent reinforcement learning setting with multiple actions and one coupling state constraint which has broad applicability across various fields, including inventory management.  The authors proposed WIMS by leveraging deep multi-agent reinforcement learning. Specifically, WIMS built on top of the state-of-the-art Whittle index policy, and generalize it to multi-action and dynamic state constraint settings. Experimental results were provided to validate the performance of WIMS.", "questions": "See comments in weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper considered a multi-agent reinforcement learning setting with multiple actions and one coupling state constraint which has broad applicability across various fields, including inventory management.  The authors proposed WIMS by leveraging deep multi-agent reinforcement learning.", "weaknesses": "1. In this paper, the authors assume that the multiple SKUs are independent and their total inventory level cannot exceed some capacity constraint. Unfortunately, this assumption is problematic and lack of justification. For example, rewards are not only independent in large-scale systems (a large number of SKU), and the substitution effect can occur. \n\n2. The proposed algorithm WIMS is mostly heuristic based and lack performance guarantees. First, this paper introduces a state constraint that must be satisfied in each time step $t$ in Section 4.1. However, WIMS is NOT guarantee to satisfy this constraint at all, and hence there is a gap between the formulated problem and the proposed solution. Second, it is not quite clear if the proposed WIMS is asymptotically optimal or not, which is a key performance in restless bandits literature. For example, the state-of-the-art Whittle index policy is provably asymptotically optimal. However, Whittle index policy is designed for restless bandits with a constraint on the action, rather than on the state. It is not straightforward to the reviewer if the asymptotic proof of Whittle index policy can be generalized to that for WIMS. Third, as discussed above, there is a gap between the problem formulation (with state constraint) and the proposed WIMS (ignore the constraint). To this end, this paper tunes the parameters of $\\lambda_g$. Though empirical results are provided, it is unclear if this is theoretically sound given that there is not performance guarantee in this paper. \n\n3. From the experimental results, the proposed WIMS outperformed the considered baselines in terms of accumulated profit. What about the computational complexity? The reviewer did not fully understand the results presented in Table 4. It seems that WIMS takes significant larger computational costs, measured in mins. Given these large numbers (mins), how could this algorithm adapt to dynamic settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lD8cngv2ql", "reviewer": "Reviewer_qxXR", "summary": "This paper is about computational studies of a Whittle index-based policy on a variant of the restless bandit problem. The restless bandit setting allows multiple actions for each bandit, and there is a constraint that controls the joint state of the bandits. In particular, the paper focuses on the inventory management of multiple stock-keeping units. Here, each stocking-keeping unit corresponds to a bandit, and the list of stock levels that a unit can maintain is the set of actions. The distinction with the existing works on restless bandits is that the problem is not about selecting one bandit but about choosing an action for each bandit. That said, it makes sense to define and compare the indices of multiple actions, instead of comparing some indices of distinct bandits. Moreover, the inventory level constraint imposes restrictions on taking which actions to different units. Therefore, the problem can be viewed as an instance of multi-agent reinforcement learning (MARL).", "questions": "* Is it possible to prove that without the joint inventory level constraint, the Whittle index-based policy of this paper achieves optimality?\n* Is it possible to discover and explain any connection between the setting of this paper and the MARL settings, e.g., fully competitive and cooperative settings?\n* Is it possible to compare the proposed framework of this paper and the existing single-agent inventory control methods combined with dual penalization as done in this paper?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper provides a novel definition of a Whittle index-type policy for a multi-agent inventory management problem. Indexability for the setting is defined, and some sufficient conditions for the notion of indexability are provided. The framework seems novel.\n* In general, implementing a Whittle index policy can be inefficient as computing Whittle indices is difficult. However, numerical results show that the proposed Whittle index-based policy can be efficiently implemented and at the same time, one can impose satisfying the joint inventory level constraint at least computationally.", "weaknesses": "* No theoretical guarantee on the proposed method is provided. It seems that the framework sits between restless bandits and multi-agent reinforcement learning (MARL). That said, the reader would wonder if any theoretical results on either restless bandits or MARL extend to the particular problem setting of this paper. \n* It is not clear how the joint inventory level constraint is satisfied by the WIMS policy. The WIMS policy controls individual stock-keeping units separately while the only joint control is on updating the dual variable. That said, in principle, one may use any algorithm for dealing with individual units. It is difficult to convince that the WIMS policy is particularly effective for the multi-agent setting studied in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)