Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Authors: Guangyi Liu
Suzan Iloglu
Michael Caldara
Joseph W Durham
Michael M. Zavlanos
Venue: ICML 2025 poster (2025)
Keywords: Distributionally Robust Reinforcement Learning
Multi-agent Reinforcement Learning
Group Distributionally Robust Optimization
Robotic Sortation Warehouse
Primary area: reinforcement_learning->multiagent
TL;DR: A robust multi-agent reinforcement learning framework using group DRO and contextual bandit that optimizes chute mapping in robotics sortation warehouses, reducing package recirculation by 80% under varying induction rates.

Abstract: In Amazon robotic warehouses, the destination-to-chute mapping problem is crucial for efficient package sorting. Often, however, this problem is complicated by uncertain and dynamic package induction rates, which can lead to increased package recirculation. To tackle this challenge, we introduce a Distributionally Robust Multi-Agent Reinforcement Learning (DRMARL) framework that learns a destination-to-chute mapping policy that is resilient to adversarial variations in induction rates. Specifically, DRMARL relies on group distributionally robust optimization (DRO) to learn a policy that performs well not only on average but also on each individual subpopulation of induction rates within the group that capture, for example, different seasonality or operation modes of the system. This approach is then combined with a novel contextual bandit-based estimator of the worst-case induction distribution for each state-action pair, significantly reducing the cost of exploration and thereby increasing the learning efficiency and scalability of our framework. Extensive simulations demonstrate that DRMARL achieves robust chute mapping in the presence of varying induction distributions, reducing package recirculation by an average of 80% in the simulation scenario.

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Guangyi Liu
Suzan Iloglu' Michael Caldara
Joseph W. Durham
Michael M. Zavlanos 12
Abstract
In Amazon robotic warchouses, the destination- to-chute mapping problem is crucial for efficient package sorting. Often, however, this problem is complicated by uncertain and dynamic package induction rates, which can lead to increased pack- age recirculation. To tackle this challenge, we introduce Distributionally Robust Multi-Agen Reinforcement Learning (DRMARL) framework that learns destination-to-chutte mapping pol- icy that is resilient to adversarial variations in induction rates. Specifically, DRMARI. relies on group distributionally robust optimization (DRO) to learn a policy that performs well not only on av- erage but also on each individual subpopulation of induction rates within the group that capture, for example, different seasonality operation modes of the system. This approach is then combined with novel contextual bandit-based estimator of the worst-case induction distribution for each state-action pair, significantly reducing the cost of exploration and thereby increasing the learning efficiency and scalability of our framework. Ex- tensive simulations demonstrate that DRMARL achieves robust chute mapping in the presence of varying induction distributions, reducing package recirculation by an average of 80% in the simula- tion scenario.
Figure I: Schematic layout of an Amazon robotic sortation ware- house featuring robot drives and eject chutes.
gins at induction stations, where packages are loaded onto mobile robots and subsequently transported designated eject chutes based on their destinations (Figure 1). A criti- cal factor determining the package throughput capacity of these facilities is the effective allocation of eject chutes to different destinations. Therefore, the destination-to-chut mapping policy plays a crucial role in optimizing the overall throughput performance of the robotic sortation warehouses.
Our previous work (Shen et al., 2023) addresses the des- tination assignment problem (DAP) (Boysen & Fliedner, 2010) in robotic sortation systems by developing a dynamic chute mapping policy. This policy determines the optimal allocation of eject chutes to destinations with the objective of minimizing the number of unsorted packages. We pro- posed model-free reinforcement learning approach that dynamically adjusts the number of chutes assigned each destination throughout the day. Our solution formulates the chute mapping problem within a Multi-Agent Rein- forcement Learning (MARL) framework (Lowe et al., 2017; Sunehag et al., 2017: Samvelyan et al., 2019; Rashid et al., 2020), where each destination is represented as an agent that controls its chute allocation at each time step.
1. Introduction
In Amazon robotic sortation warehouses, mobile robots deployed to transport and sort packages efficiently to different destinations (Wurman et al., 2008; Azadeh et al., 2019; Amazon, 2022b;a; 2023). The sorting process be-
Ammzoon Robotics, North Reading, MA, USA DDeparmment of Mechanical Engineering and Materials Science, Duke Uni- versity, Durham, NC, USA. Correspondence to: Guangyi Liu <gyliu@ amazon.com>>
While the MARL policy proposed in our previous work (Shen et al., 2023) demonstrates superior performance com- pared to traditional reactive chute mapping approaches often implemented in Amazon robotic sortation warehouses, its effectiveness assumes that the induction distribution dur- ing deployment matches the training distribution and that the daily induction rate remains close to its average value.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
In practice, however, induction patterns exhibit significant temporal variations, potentially compromising the MARL policy's performance when confrrnnted with unexpected dis- tribution changes.
tions on the sortation floor, developing stochastic approach with chance and robust constraints. For robotic sorting sys- tems, (Khir et al., 2021) proposes an integer programming method to solve DAPs that minimize sortation effort and satisfy package deadlines. A robust formulation address- ing demand uncertainty is presented in (Khir et al., 2022). While these approaches effectively optimize destination as- signment in sorting systems, they do not account for distri- butional uncertainty in demand and system dynamics. In contrast, our proposed DRMARL framework explicitly mod- els such uncertainties, ensuring robust performance under varying operational conditions.
To enhance robustness against such variations, in this pa- per, we propose a Distributionally Robust Multi-Agent Re- inforcement Learning (DRMARL) framework that learns chute mapping policies capable of maintaining near-optimal performance across diverse induction distributions. Specif- ically, we formulate this problem as group DRO problem, where each group represents a distinct induction distribu- tion pattern extracted from a subset of a historical dataset. Our DRMARL framework optimizes policies for the worst- case induction patterns across these distribution groups. To address the computational cost of exhaustively evaluating all distribution groups during training, we introduce con- textual bandit (CB)-based worst-case reward estimator for each state-action pair. Through extensive evaluation, we demonstrate that our DRMARL framework yields robust chute mapping policies that not only outperform baseline MARL policies on out-of-distribution (OOD) induction data but also maintain consistent performance across varying in- duction distributions.
MARL for Resource Allocation: MARL has previously been applied address resource allocation problems (Nie et al., 2021; Mei & Wang. 2024; Jun-Han et al., 2025). For exam- ple, a MARL framework for occan transportation networks was proposed in (Li et al., 2019). This framework devel- oped a multi-agent Q-learning algorithm where the local Q-networks depend on the joint states (including the lim- ited shared resources) and the joint actions. However, since the joint state-action space grows exponentially with the number of agents, the local Q-networks are hard to learn and this approach does not scale well in practice. This limitation was addressed in our previous work (Shen et al., 2023), where the local Q-networks are only loosely coupled, enhancing the scalability while still being interconnected enough to capture the impact of robot congestion on the sortation floor. Compared to (Li et al., 2019), the method proposed in (Shen et al., 2023) models resources explicitly as actions and considers budget constraints when taking joint actions. However, these MARL-based approaches do not incorporate distributional robustness, making them sen- sitive to demand fluctuations and uncertainty. which our DRMARL framework explicitly addresses to ensure reliable performance in dynamic sorting environments.
Our contributions are twofold: distributionally robust optimization in multi-agent reinforce- ment learning, developing a novel framework to learn poli- cies that are robust to distribution shifts in the reward func- tion. Second, we propose an novel contextual bandit-based method for efficient worst-case reward estimation, signifi- cantly reducing the computational complexity of DRMARL training by eliminating the need for exhaustive group explo- ration to estimate the worst-case reward. To the best of our knowledge, our framework is the first to integrate contextual bandits with group DRO and MARL, addressing a well- known challenge of distributionally robust reinforcement learning related its computational cost. Our proposed framework has broad applicability to various large- scale industrial applications beyond sortation systems, including resource allocation, collaborative robotics, and warehouse automation, where robustness to distribution shifts is cru- cial.
First, we introduce group
Robust and Distributionally Robust RI:: Robust Reinforce- ment Learning (Robust RL) (Morimoto & Doya, 2005; Wiesemann et al., 2013; Pinto et al., 2017; Panaganti & Kalathil, 2021; Moos et al., 2022; Panaganti et al., 2022; Goyal & Grand-Clement, 2023; Yamagata & Santos- Rodriguez, 2024) develops policies that maintain perfor- mance under worst-case conditions through adversarial per- turbations. Distributionally Robust Reinforcement Learning (DRRL) (Xu & Mannor, 2010; Smirnova et al., 2019; Hou et al., 2020; Wang et al., 2023; Ramesh et al., 2024; Zhang et al., 2024; Lu et al., 2024; Panaganti et al., 2024) extends this by optimizing across environment distributions rather than a single worst-case scenario. While traditional DRRL primarily addresses ambiguity in MDP transition probabili- ties, this approach inadequately captures induction distribu- tion changes in Amazon robotic sortation warehouses. Our problem requires a focus on distributionally robust optimiza-
1.1. Literature Review
Destination Assignment Problems: Mathematical program- ming has been used to optimize warehouse systems, in- cluding destination assignment problems (DAP) (Boysen & Fliedner, 2010) for sorting systems. The destination map- ping approach in (Fedtke & Boysen, 2017) optimizes pack- age flow by minimizing travel distances between inbound and outbound stations in bonveyor-based sorting systems, which leads to improved throughput. (Novoa et al., 2018) minimizes the worst-case flow imbalance across work sta-
2

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
tion of reward function distributions, building on (Ren & Majumdar, 2022; Liu al., 2022). Recent advances in (Dis- tributionallyy Robust Multi-Agent RL (Zhang et al., 2020; Bukharin et al., 2024; Shi et al., 2024b;a) have introduced frameworks like RMGs, ERNIE, and DRNVI to address en- vironmental uncertainties, adversarial dynamics, and model uncertainties. While existing methods primarily focus on robustness in transition dynamics, adversarial interactions, and general environmental uncertainties, they do not plicitly address distributional shifts in package induction, which is a critical challenge in sortation warehouses. Our ap- proach extends DRMARL to explicitly model and optimize against uncertainties in induction distributions, ensuring ro- bust and consistent performance under varying operational conditions.
Eject Chutes
Incoming Package
Induction Butfer
Laden Drive Buffer
Sorted Package
Recirculation Butfer
Recirculation Chutes
Figure 2: Flow of packages in the Amazon robotic sortation ware- house.
contains packages actively transported by robots from in- duct stations assigned eject chutes. Since the total number of chutes is limited, contention occurs when multiple desti- nations require chute capacity simultaneously. When chute availability is insufficient, excess packages are diverted to recirculation chutes, entering the recirculation buffer for another pass through the system. Chutes are reallocated to different destinations when destination vehicle reaches ca- pacity or when its scheduled departure time is reached. The dynamic chute mapping policy optimizes chute allocations across destinations in real time to minimize recirculation and maximize system throughput
Group DRO: Group Distributionally Robust Optimization aims to enhance model robustness across diverse subpopula- tions by optimizing for the worst-performing groups rather than the average performance (Sagawa et al., 2020). This ap- proach ensures fairness and resilience to distribution shifts, particularly for underrepresented groups. While initial work focused single-agent supervised learning (Hu et al., 2018; Oren et al., 2019), recent advances have extended these prin- ciples to more complex settings. Notably, (Soma et al., 2022) proposed a soft-weighting method on distribution groups with convergence guarantees, while (Wu & Fu, 2023) and (Xu et al., 2023) demonstrated the applicability of group DRO in multi-agent systems and reinforcement learning, respectively. Our work bridges a critical gap by introduc- ing group DRO principles to DRMARL. We begin by for- mulating the distributionally robust Bellman operator and addressing the computational challenges of exploring all dis- tribution groups during training. To tackle these challenges, we provide a DR Bellman operator specifically designed for MARL and introduce a contextual bandit (CB)-based worst- case distribution group estimator. This estimator enables efficient training by adaptively identifying the worst-case distribution groups
We formulate the dynamic chute mapping problem as a sequential (ecision-making problem, specifically as a multi- agent reinforcement learning (MARL) problem that deter- mines the optimal chute allocations to minimize package recirculation at each time step. To this end, we define a Markov game over N unique destinations, represented by the tuple (N ( {-A') N 1==1: P. (r (r=) 7, Pn).
where S is the joint state space, O² C S is the local obser- vation space for destination i, A' denotes the action space (i.e., the number of new chutes acquired) for destination i, P is the state transition probability, r' is the reward func- tion, E (0, 1) is the discount factor, and po is the initial state distribution. While the joint state provides complete system information, each destination's 's observation is lim- ited to local features relevant to its deciison-mmkinn. This partial observability requires destinations to act based on their own experiences and available information. We refer to Appendix C.1 for detailed MARL formulation.
The remainder of the paper is organized as follows. In Section 2, we formulate the dynamic chute mapping prob- lem within a multi-agent reinforcement learning framework. In Section 3, we extend this formulation by incorporat- ing group distributionally robust optimization into MARL, while in Section 4, we present novel contextual handit-hased worst-case reward estimator to enhance training efficiency. Finally, in Section 5, we demonstrate the effectiveness of our proposed framework through extensive simulations.
The system operates in discrete time steps, where, at each step t, individual agents (destinations) make chute allo- cation decisions. Each agent i employs a local policy T² : Oi x A + [0, 1] that maps local observations o* to probabilities over possible chute allocation actions. Each agent i learns its optimal local policy n² by maximizing the expected discounted return [[ = E Ei 7" "rrr) where r/. denotes the instantaneous reward at time L' and Ty is the discount factor. The expectation captures both the stochastic nature of the policy and the environment dy- namics. This formulation naturally aligns individual agent objectives with the global goal of minimizing recirculation
2. Problem Formulation
In Amazon robotic sortation warehouses, package flow is modeled using three buffers: induct, laden drive, and recir-
culation, as illustrated in Figure 2. The laden drive buffer
3

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
while maintaining the throughput. The instantaneous reward function for agent i time step is defined as:
Degradation (%)
100
74.45
75
51.01
50
reciicc - 22a
(1)
25
where recirct 0 represents the number of packages in recirculation for destination i and the action term of is included as penalty to prevent the learned policy from over-allocating chutes to only a few destinations. Due to the coupled nature of agent decisions, we utilize the joint action-value function to determine optimal local policies ;
Year 2
Year 3
Year 4
Figure 3: Relative degradation of MARI, policy (trained on Year 1) on OOD (Years 2-4) induction data, compared to Year 1,
distribution shifts in package induction patterns (see Fig- ure 3). This vulnerability to out-of-distribution scenarios motivates our robust formulation.
(= E
R a, =
In this paper, our objective to introduce robustness to distribution shifts in the learned chute mapping policies. We achieve this by incorporating group DRO into the MARL framework, giving rise the proposed DRMARL approach. Our proposed framework ensures reliable and robust perfor- mance in Amazon robotic sortation warehouses, even under unforeseen future induction scenarios.
which evaluates the expected return when taking the joint action a = =aa (a!,....." a²) in state and following the joint policy π thereafter.
To mitigate the exponentially growing policy space, we assume agents execute actions independently, such that I n². The optimal policy x² is learned using the Deep Q-Network (DQN) (Mnih et al., 2015), where neural network Q(s, a; 0) with parameters 0 approximates the optimal action-value function Q" as) The learning
3. Distributionally Robust Multi-Agent Reinforcement Learning with Group DRO
process minimizes the loss:
In this section, we enhance the MARL chute mapping frame- work described in Section 2 by incorporating group DRO to handle uncertainty and variability in package induction distributions. This approach enables to develop robust policies that perform well across diverse induction scenarios, including previously unseen induction distributions.
L(0; X) E, B,,.. [Q((s,;;))- - w)²].
where y -Ex~p/r(s, a; X]] + y max, Q(s', d':0) ap- proximates the optimal target values Ex~e/r(s, a; X) y max. (s', a'). Here, r(s, ax;) X) represents the instan- taneous reward under the current state-action pair and in- duction distribution X'. Stability of the learning process is enhanced through two mechanisms: a target network Q with periodic parameter updates using the most recent values of 0, and an experience replay buffer D that stores transition tuples (s,a,r,s'), The resulting optimal policy takes the form:
Unlike traditional stochastic optimization, which assumes that the true data-generating distribution is known and fixed, Distributionally Robust Optimization (DRO) addresses un- certainty in the underlying distribution by optimizing against family of plausible distributions, i.e., the ambiguity set, constructed from the available data (Delage & Ye, 2010; Shapiro, 2017; Rahimian & Mehrotra, 2022; Zhen et al., 2023; Liu et al., 2024; Kuhn et al., 2024; Konti et al., 2024). Instead of minimizing the expected loss under a single es- timated distribution, DRO seeks solutions that minimize the worst-case expected loss over all distributions within this ambiguity set. This leads policies that are robust distributional shifts, such those encountered in the chute mapping problem, where test-time conditions may deviate from those seen during training.
(A(a)) if a E A(s),
(4)
otherwise
where A(s) = arg max, a())a;0") and 0* arg min L(0; X).
To address the scalability of the state-action space and the computational feasibility of the expectation (3), we employ the Value Decomposition Network (VDN) (Shen et al., 2023) with budget constraints for computing feasi- ble joint actions, with implementation details provided in Appendix C.3 and Appendix C.4.
3.1. Group DRO
In DRO, the ambiguity set that captures uncertainty in the data Hata-generating distribution can be defined in various ways. In particular, Group DRO (Hu et al., 2018; Oren et al., 2019) offers an efficient way to define the ambiguity set using a finite collection of known distributions. In the context of
While the above MARL-based chute mapping policy demon- strates strong performance under standard operating condi- tions, it exhibits significant performance degradation under
See Appendix C.2 for details of the induction distribution.

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
3.2. DRMARL with Group DRO
P1
In the MARL framework described in Section 2, the policy parameters 0 are optimized as follows:
0.1P, 0.3.2 0.6P3
0
arg min Ex-e [L(;
(7)
F2'
3
P2
Directly applying group DRO to MARL then leads to:
Figure 4: Ambiguity set 2T in regular DRO (Rahimian & Mehrotra, 2022) (left) versus group DRO (Sagawa et al., 2020) (right).
max Ex-P, [I[0; [[(e
0 = arg min EEB
(8)
9Eg
robotic sortation warehouses, this approach is especially useful, as the ambiguity set can be constructed from histor- ical induction data that reflects the variability observed in real-world operations.
However, conventional group DRO approaches are not di- rectly applicable to MARL problems, as minimizing the worst-case Bellman error across groups does not necessar- ily yield policy that is optimal under worst-case rewards. This disconnect arises because the worst-case Bellman error captures the maximum deviation from the target Q-function across groups, but it does not guarantee convergence to a robust optimal Q-function. To address this limitation, we introduce the distributionally robust Bellman operator, for- mally defined in the following result.
(Sagawa et al., 2020), we define the unknown distribution I as a combination of m distributions Pg. each The ambiguity set 21 is then defined as a convex combination of these groups:
Following
indexed by group g in the set G = . m)
m
M
P
Σ gg Pg g=1
q
Am)
(5)
Lemma 3.2. For an ambiguity set M defined in (5) with group set G, the distributionally robust (DR) Bellman oper- ator is given by:
where Am denotes the (m didimennionnal probability simplex (Grünbaum et al., 1967) (see Figure 4).
(s, = inf gEg
max Q(5', 0(s',a)) d')
In the dynamic chute mapping problem, we assume that past years of operational data from sortation warchouses provide sufficiently rich historical induction distributions that can be used to obtain representative distribution groups G. With this assumption, it is reasonable to expect that most future induction distributions IP can be represented as combinations of the basis distributions P, with G. As shown in (Sagawa et al., 2020), evaluating the worst-case reward over all m groups in G is equivalent to evaluating the reward for the worst-case distribution within the ambiguity set Di defined in (5).
where Q is the distributionally robust Q-function. Moreover, the DR Bellman operator Tg is also a contraction mapping.
The proof of Lemma 3.2 is provided in Appendix A. Ac- cordingly, the distributionally robust loss is given by
i. x) = B. [({(s,a;0)
o' - inf Ex-P, |r(s, X)] max@(s' (s'x 0)) .EG
(9)
and the distributionally robust parameters 0g are obtained by solving
Lemma 3.1. Consider ambiguity set M. formed by P,s as defined in (5). For any state state-action pair (s, a) E x A.
0g == ar= min L(e; X). 8E0
(10)
the worst-case expected reward satisfies:
4. Contextual Bandit-based Worst-Case Reward Estimator for DRMARL.
inf Ex-r, [r(s, = inf Ex-r (xs, gEG
(6)
Petn
where G denotes the set of group indices.
group Solving the MARL group DRO problem (10) is theoretically feasible since the worst-case reward can be evaluated for each (s, a) by exhaustively searching over all distribution groups G. However, this approach is inefficient when the number of groups is large and forward simulation in the environment is costly. This is particularly the case in the dynamic chute mapping problem, where millions of pack- ages are sorted across many destinations. Common group DRO techniques, such as soft reweighting (Sagawa et al.,
The proof of Lemma 3.1 is provided in Appendix A. The above lemma highlights key advantage of group DRO: while general DRO problems are infinitediimensional and computationally challenging, group DRO reduces the opti- mization to a fnite-dimennional problem over m groups This reduction makes training of distributionally robust MARL (DRMARL) computationally tractable.
5

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Algorithm 1 CB-based Worst-Case Reward Estimator
Algorithm 2 DRMARL with CB-based Worst-Case Reward
1: Input: Learning rate (cB, initial parameters to, induc- tion distribution groups 9, MARL policy with QMARL exploration rate ECB
Estimator
Input:
Learning rate 4, initial parameters fl. induction
distribution groups G, pre-trained CB-based estimator
2: Initialize: v + to, replay buffer Dea <- 0
QCB. exploration rate
3: for episode = I,... lcB do
2: Initialize: 0 + 0o, replay buffer D +-
4: Initialize the environment with random group g' e eg
3: for episode = 1..... k do
and observe initial state Su
4: Initialize the environment with random group g' E g and observe initial state 50
5: for time step 0, o,.... T do
6: Select action at f- arg m QMARL(SE, a)
for time step = 0,.... do do
7:
With probability ECB, select g' ~ Uniform(G)) (g); otherwise, n' 1 arg mingeg (st, at, 9i 15)
With probability E, select ag Uniform(AA); oth-
erwise, ae +- arg maXue ((s,, a;
8:
Execute action e₁ observe reward r2 and next state 81+11 using group g'
7:
Estimate worst-case distribution group using CB:
g' + arg mingeg Qcn g)
9:
Store transition (st, q'.g', rt, St+1) in DeB Sample mini-batch from DeB and update sh:
8:
Execute a,, observe reward T; and next state St+1
10:
using group g'
11:
+ IcB V (CB((()))
Store transition (st, ats (st, g Tty 88+1) in Sample mini-batch from and update parameters:
12:
+ S reduce ECB
10:
13:
end for end for
θ - 0 1, V.L
14:
11:
St f- St+1, reduce
15: Output: Optimized CB estimator parameters v
12: end for
13: end for
14: Output: Optimized DRMARL policy parameters 0g
2020), may not perform well in MARL because, unlike re- gression tasks, the data distribution depends on the agent's policy. As the policy evolves, the groups that are underrep- resented or perform poorly will change dynamically. This dynamic nature of the problem makes it challenging apply static even adaptive reweighting schemes, which assume a relatively stable data distribution.
The CB is represented by a Q-function that approximates the expected reward of choosing distribution P, given context (s, a). For this purpose, use independent DQN (Mnih et al., 2015):
Qca(s,a,g;v) = =x Ex-F, as)s
(11)
To improve training efficiency, we propose a novel contex- tual bandit- bandit-based worst-case reward distribution estima- tor that learns identify the worst-case distribution group g G for each state-action pair (s, a) by training a con- textual bandit (CB) model (Li al., 2010; Russo et al., 2018).
where the reward function r(s, a; X) is observed after run- ning single-step forward simulation with (s, (s,a) under the distribution Fgg. The QCB function is learned by minimiz- ing the following loss (see the detailed training process in Algorithm I):
Lcn( = E.,a (Qcn(ss,a.g:)) Ex-P, (rss (12)
4.1. CB-based Worst-Cass Reward Estimator
The CB treats the current state-action pair (s, a) context, and its arms corresponding to the distribution groups in the set G. The goal is to identify the group g that minimizes the reward Ex.r, [r(s, a; X) which represents the worst-case reward among all groups. The CB constructed as follows:
In Algorithm the exploration of state-action pairs (s, guided by the existing MARL policy using QMARL. which ensures sufficient exploration of the context space for the chute-mapping problem. For other applications, different exploration policies can be employed, such as random action selection, to ensure adequate coverage of the state-action space.
A, where (s,a) a) E S A represents the state and the chute mapping actions at each step.
Context space:
Action space: G = 21, m). where g E G denotes group associated with an induction distribution.
4.2. DRMARL with CB-based Worst-Case Reward Estimator
Reward: A reward function r: (S x A) x G - R, where r(s, a; X) represents the observed reward for choosing dis- tribution g at the current state-action pair (sa).
Once the QCB function has been learned, we can rewrite the
6

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Table
Comparison of key sortation metrics across policies, averaged over m = 9 groups, each evaluated over 100 runs. See
2000
Appendix D.1 for detailed results.
1750
Policy
Recirculation Rate (_)
Throughput (1)
Recirculation
Amount (L)
Resircaleton
MARL
2.16% 2.35%
11740.98
259.02
1000
DRMARL (random)
1.56% +. 1.45%
11812.77
187.2 23
750
Ci-basen Predictor
Soarch
DRMARL (with Qus)
0.56% + 0.18%
11932.21
67.79
Random Group Switch
DRMARL (exhaustive)
0.35% = 0.13%
11933.68
66.32
500
MARL on Group #1
MARL (group-speciffe)
0.03% 0.145%
11936.60
63:40
250
100
150
200
3:00
350
400
Training Time (Seconds)
distributionally robust loss (9) as:
Figure 5: Training efficiency comparison in simplified warchouse.
i(0; X) Ex,,an (((s, 0)
(13)
1.00
- -ExP, asss,; - maxQ(s' ,d a'
Recirc Logs
3.75
Average Prediction Loss
1.7
3.50
Max Prediction
3.25
where o' = arg mingeg Qca a, g) is the index of the distribution group with the estimated worst-case reward.
3.00
0.75
0.50
The training procedure of DRMARL is shown in Algo- rithm 2. The key difference compared to MARL training is that DRMARL aims to train DQN that estimates the worst- case return across all groups, while MARL aims to estimate the observed return only for a specific induction distribution. Moreover, in contrast to traditional group DRO, the index of the worst-case distribution g' in DRMARL is not obtained via exhaustive search; instead, it estimated by QCB given the state-action pair (s, a). The independent Q-network Qcn is learned beforehand using Algorithm 1 and remains unchanged during DRMARL training. While it may seem counterintuitive that QCB estimates the worst-cass group after Q selects an action, this design enables QcB to provide the worst-case expected return for each (s, a) pair, thereby enabling the learning of robust policy.
50
200
Training Episode
150
200
250
200
250
Figure 6: Training recirculation rate estimation loss (12) for CB in
simplified robotic sortation warchouse.
We train the DRMARL. policy over 300 episodes using training data generated from 9 distinct induction distribution groups. Similarly, the regular MARL policies trained for 300 episodes, each on one of the same groups. Due to the stochastic nature of the induction- induction-generating distributions, the induction data varies across simulation trials.
Key metric comparisons are shown in Table DRMARL with QCB achieves the best performance across all met- rics. For reference, the last row shows the theoretical upper bound achieved by group-specific MARL policies, trained and evaluated on the same group. DRMARL performs only marginally below this oracle baseline, highlighting its strong balance between individual performance and robust- ness. Compared to random group selection and exhaustive worst-case search, QCB efficiently identifies challenging groups during training and achieves comparable robustness with significantly lower computational cost. These results demonstrate that QCB enables DRMARL to focus worst- case distributions effectively, yielding robust policies with- out sacrificing efficiency. We refer to Appendix D. for a detailed discussion of these results.
5. Numerical Experiments
In this section, we demonstrate the effectiveness of the proposed DRMARL policy under OOD induction changes in both a simplified simulation and a large-scale Amazon robotic sortation warehouse simulation environment.
5.1. Simplified Robotic Sortation Warehouse Simulation
In the simplified simulation environment, there are 10 eject chutes, one recirculation chute, and 20 unique destinations. Packages arrive the sortation warehouse according to the induction data X generated from an induction distribution P. When packages exceed the capacities of eject chutes, they are sent to the recirculation chute. One training or testing episode consists of 5 hours, with each time step being 30 minutes long, after which the environment is reset. An eject chute can be reallocated at each time step. The implementation details are provided in Appendix B.
Figure shows the training progress of QCB, with clear reductions in both average and maximum recirculation rate estimation errors over time. Training efficiency compar- isons across methods are shown in Figure 5. DRMARL with QCB converges in under 300 seconds, while exhaus- tive worst-case search exceeds 2900 seconds due to its full traversal of all groups In contrast, QCB achieves compa- rable robustness at significantly lower computational cost,
7

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Table 2: Relative improvement (1) DRMARL over MARL baseline, averaged across m 21 groups.
PE 90
1 $
Recire Rate Reduction
Throughput Increase
Recire Amount Reduction
60
61
62
63
64
65
66
67
Policy
Recirculation Percentage Improvement (%)
DRMARL MARL
79.97%
5.62%
33.64%
Figure 7: Recirculation rate improvement of DRMARL over two equally-performing MARL policies trained distributions inside and outside Mt.
85.42%
9.80%
40.50%
(group-specific)
Table 3: Relative reduction (1)) in CVaR, worst-case recirculation amount, and average recirculation rate across different Robust RL approaches. Method
validating its ability to efficiently identify worst-case groups. Although DRMARL with random group selection converges faster, it lacks the worst-case guarantees provided by QCB- Group-specific MARL converges fastest overall, reflecting the simplicity of its angle-group training setup.
CVaR Reduction
Worst-cuse Recire
Recire Rate
Reduction
Reduction
Adversarial Agent
61.11%
53.57%
66.
Adversarial Environment
58.48%
23.68%
77.09%
Robustified Reward
58.97%
60.23%
70.13%
5.2. Lcrge-Scale Robotic Sortation Warehouse Simulation
DRMARL
77.09%
76.61%
79.97%
5.2.1. IMPLEMENTATION DETAILS
the bottom row shows the theoretical optimal performance achieved by training and testing group-specific MARL poli- cies on each individual group. As expected, DRMARL performs marginally below these group-specific MARL policies, illustrating the trade-off between robustness and individual group performance. Detailed results are provided in Appendix D.2.
In the large-scale simulation environment, there are 187 eject chutes, one recirculation chute, and 120 unique destina- tions. Packages arrive the sortation warehouse according to the corresponding induction data X generated from the induction distribution pp?. When packages exceed chute ca- pacities or miss departure transportation schedules, they are sent to the recirculation chute and added to the queue of new packages at the next time step. One training/testing episode lasts 11 hours, with each time step lasting five minutes, after which the environment is reset. Every five minutes, des- tinations are assigned to chutes that become available for reallocation
As more production data becomes available, the distribution groups can be expanded to better capture year-to-year shifts without increasing the training complexity of DRMARL. To assess robustness beyond the ambiguity set 27, we evaluate the learned policy on induction distributions I where IP' &f M. As shown in Figure 7, the robust policy generalizes well to these -uttr--distribution settings. Notably, the Type-1 Wasserstein distance between I' and the closest distribution in M is 818. 19, compared to an average of 542.96 within 2R.
We train the DRMARL policy over 200 episodes using training data generated from 21 distinct induction distribu- tion groups spanning several years. Similarly, the regular MARL policy is trained for 200 episodes using induction data from Year 4. For testing, evaluate both policies newly generated induction data from 21 distinct distribution groups, conducting five experiments per group. Due to the stochastic nature of the induction-generatiio distributions, the test induction data remains unseen during training for both policies.
distance
We also compare DRMARL against three robust RL base- lines, with results summarized in Table 3: (i) an adversarial agent that deliberately blocks available chutes (Mandlekar et al., 2017); (ii) an adversarial environment with perturbed transition dynamics (Pinto et al., 2017); and (iii) robust MDP formulation that uses a worst-case reward function over an uncertainty set (Wiesemann et al., 2013). We evalu- ate performance using three metrics: cumulative recircula- tion over the 1 11-hour simulation; worst-case instantaneous recirculation; and Conditional Value-at-Risk (CVaR) at the 5% level (Rockafellar & Uryasev, 2000). To compute CVaR, we collect instantaneous recireulation values at all time steps and estimate their empirical distribution. CVaR at 5% corresponds to the expected recirculation conditional on exceeding the 95th percentile, capturing the expected sever- ity of the worst-case recireulation (higher values are more undesirable). While some robust RL approaches achieve
5.2.2. ROBUSTNESS OF THE CHUTE MAPPING POLICY
Table 2 presents the average relative performance improve- ment of DRMARL across all 21 distribution groups, us- ing MARI. as the baseline. DRMARI. demonstrates ro- bust performance across all induction groups, consistently outperforming the baseline MARL policy. For reference,
'Due to common industry confidentiality practices, we cannot disclose the specific data source and report only relative perfor- mance improvements. The data represents realistic package flow patterns typical of Amazon robotic sortation facilities.
8

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Redire Estimation Loss
10?
Max Estimation Loss
Average Loss
CB based Prodictor
10a
Max Loss
10a
500
MARL #20
10a
100
-1000
Exhaustive Search
-1500
10-1
22000
10-2
10-2
.2500
0
100
200
300
400
500
600
Simulation Minutes
Figure 8: QuB estimation loss for recirculation percentage, aged over 25 test simulations.
4000
Training Hour
cumulative recirculation comparable to DRMARL, only DRMARL consistently improves the worst-case recircula- tion performance by explicitly optimizing for it. DRMARL outperforms all robust RL baselines due its distributional awareness, which yields more accurate worst-case reward estimation and robust action-value functions. This robust- ness is critical in practice, as sudden surges in recirculation can lead to robot congestion and delays in the sorting pro- cess. Since robot congestion is not explicitly modeled in the simulator, the real-world advantage DRMARL over standard robust RL. methods likely even greater.
Figure 9; Training computational efficiency comparison in large- scale Amazon robotic sortation warehouse simulation environ- ments.
pronounced when dealing with complex environments larger group sets.
Figure also compares the training efficiency of differ- ent approaches. The group-specific MARL, which trains on group #20, shows the fastest convergence due to its simplified learning objective. Among distributionally ro- bust approaches, DRMARL with random group selection (g' < random(G)) initially converges faster than other bust variants but achieves suboptimal robustness, since it may miss critical worst-case scenarios. DRMARL with QCB strikes a balance between training speed and performance, converging significantly faster than exhaustive search while maintaining near-optimal worst-case performance guaran- tees. As expected, DRMARL with exhaustive search over all distribution groups requires the longest training time, though it serves valuable baseline for validating the efficiency of our QCB- baseb approach.
5.2.3. CB-BASED VORST-CASE REWARD ESTIMATOR
Following Section train an independent Q-network, QCa, to estimate the worst-case recirculation (reward) for all state-action pairs (s,a) across groups G. The trained Qcn achieves high accuracy, with estimation errors below 1% of the recirculation rate, enabling reliable identification of worst-case scenarios among the groups g E g.
During each day's 11-hour simulation, illustrated in Fig- ure 8, the estimation accuracy improves substantially after the first hour. Although initially suboptimal, Qcn't 's perfor- mance remains sufficient for DRMARL training, since the impact of the worst-case distributions on recirculation be- comes more pronounced later stages when fewer chutes are available.
6. Conclusion
In this paper, we introduced DRMARL, a framework that integrates group DRO into MARL enhance policy robust- against OOD distribution shifts in warehouse sortation systems. To address the computational cost of identifying the worst-case group, we developed CB-based estima- tor that significantly reduces the complexity of worst-case identification from (mn) to O Experimental results from both simplified and large-scale warehouse environ- ments demonstrate that DRMARL achieves robust and near- optimal performance across all distribution groups while maintaining computational efficiency. The framework also shows strong generalization to distributions outside the train- ing set, and its design principles can be extended to other MARL applications where distributional robustness is cru- cial.
5.2.4. EFFICIENT TRAINING WITH CB-BASED WORST-CASE REWARD ESTIMATOR
The CB-based worst-case reward estimator, QCB, substan- tially improves training efficiency by eliminating the need for exhaustive group evaluation at each time step, reducing computational complexity of the worst-case group iden- tification from (m) (1). As demonstrated in Figure 9, training with QCB achieves significantly faster convergence compared exhaustive evaluation over G, which requires approximately 924 hours on a cloud instance with 64 vCPUs (Intel Xeon Scalable 4th generation) and 128 GB RAM. The lightweight Q-network updates enabled CPU-only training, with most of the computation time spent on environment simulation. This efficiency advantage becomes even more

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Acknowledgments
news-rreleasee-eetllls///// com Announces- -ourth-uuuhhhhh Seaeth--uutteereesslleeee 2023. Amazon.com, Inc. Amazon.com announces fourth quarter results. aboutamazon com rewo//ewwsrrlesse news re ease lstitalss222225// A0azzoo. con Announces Aootrce--oooot Fourth-Quarter Qearttrrrrrreeeeeeeee Results/, 2024.
We would like to express our sincere gratitude to our col- leagues at Amazon Robotics for their support and valuable contributions throughout this research. In particular, we extend special thanks to Rahul Chandan and Mouhacine Benosman for their insightful feedback and constructive discussions related to this work.
https://ir.
Impact Statement
This work advances both machine learning research and sustainable warehouse operations. The proposed DRMARL framework reduces package recirculation, directly contribut- ing to lower energy consumption and carbon emissions in e-commerce logistics. From a societal perspective, more efficient package sorting translates to both lower consumer costs and faster delivery times, particularly benefiting those who rely on e-commerce for essential goods. For warehouse operations, the system creates more predictable workflows, potentially reducing physical strain workers and improv- ing labor conditions. The framework's robust performance across varying distribution patterns ensures consistent ser- vice levels regardless of package volume or ordering pat- terns, promoting equitable service delivery across different communities. Beyond warehouses, our approach distri- butional robustness contributes the broader development of reliable AI systems for real-world applications where performance consistency is crucial.
Azadeh, K., De Koster, R., and Roy, D. Robotized and automated warehouse systems: Review and recent devel- opments. Transportation Science, 53(4)9917-945, 2019.
Boysen, N. and Fliedner, M. Cross dock scheduling: Classi- fication, literature review and research agenda. Omega, 38(6):41--42,, 2010.
Bukharin, A., Li, Y., Yu, Y., Zhang, Q., Chen, Z., Zuo, S., Zhang, C., Zhang, S., and Zhao, T. Robust multi-agent re- inforcement learning via adversarial regularization: The- oretical foundation and stable algorithms. Advances in Neural Information Processing Systems, 36, 2024.
Delage, E. and Ye, Y. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 58(3):595-612, 2010.
Fedtke, S. and Boysen, N. Layout planning of sortation conveyors in parcel distribution centers. Transportation Science, 51(1):3-18, 2017.
References
Amazon. 10 years of amazon robotics: how robots help sort packages, move product, and improve safety.
FICO.
Xpress
optimizer.
help-ss haniaae---ooe---ddnunt--------po Accessed: 2023-01-23.
optimization, 2023. Accessed: 2023-01-23.
2022a.
Goyal, V. and Grand -ennm-Ceement J. Robust markov decision processes: Beyond rectangularity. Mathematics of Oper- ations Research, 48(1):443)))) (8(1):203322 220033222,1222 2023.
Amazon. How amazon robots navigate conges- amazon--baas-naiiieeecogggtttiii 2022b.
tion.
Grünbaum, B., Klee, V., Perles, M. A., and Shephard, G. C. Comvex polytopes, volume 16. Springer, 1967.
Amazon. Tour amazon fulfillment center.
2023. Accessed: 2023-01-23.
Hou, L., Pang, L., Hong, X., Lan, Y., Ma, Z., and Yin, D. Ro- bust reinforcement learning with wasserstein constraint. arXiv preprint arXiv:2006. 00945, 2020.
Amazon.com, Inc. Amazon.com announces fourth quarter results. aboutamazon.c con/ hwwm///wwwrreeeeee/// aboottammzoo..com//eeeeereeeee com
https://ir.
Hu, W., Niu, G., Sato, I., and Sugiyama, M. Does distribu- tionally robust supervised learning give robust classifiers? International Conference on Machine Learning, pp- 2029 2037. PMLR, 2018.
tews==releasse-etaall//303333
Quarter- hutthhuuuuurerr-estsss../e
2022.
Amazon.com, Inc. fourth quarter results.
Amazon.com
announces
Iyengar, G. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
/ttrr://ir.
10

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Jun-Han, W., He, H., Cha, J., Jeong, I., and Chang-Jun, A. Multi-agent reinforcement learning for efficient resource allocation in internet of vehicles. Electronics, 14(1):192, 2025.
Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L.., and Savarese, S. Adversarially robust policy learning: Active con- struction of physically-plassible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3932-3939. IEEE, 2017.
Khir, R., Erera, A., and Toriello, A. Two-stage sort planning for express parcel delivery. IISE Transactions, 53(12): 1353-1368, 2021.
Mei, R. and Wang. Z. Multi-agent deep reinforcement learning-based resource allocation for cognitive radio networks. IEEE Transactions Vehicular Technology, 2024.
Khir, R., Erera, A., and Toriello, A. Robust planning of sorting operations in express delivery systems. European Journal of Operational Research, 2022.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529-533, 2015.
Kim, S., Pasupathy, R., and Henderson, S. G. A guide to sample average approximation. Handbook of simulation optimization, pp. 207-243, 2015.
Konti, X., Riess, H., Giannopoulos, M., Shen, Y., Pencina, M. J., Economou-Zavlane N. J., and Za- vlanos, M. M. Distributionally robust clustered feder- ated learning: case study healthcare. arXiv preprint arXiv:2410.07039, 2024.
Moos, J., Hansel, K., Abdulsamad, H., Stark, S., Clever, D., and Peters, J. Robust reinforcement learning: A review foundations and recent advances. Machine Learning and Knowledge Extraction, 4(1 4(1):276-315, 2022.
Morimoto, and Doya, K. Robust reinforcement learning. Neural computation, 17(2):335-35, 2005.
Kuhn, D., Shafiee, S., and Wiesemann, W. Distribution- ally robust optimization, 2024. URL htwpt: //axiv.. org/ org/abs// /aa//b///// / 2411.11111 02549.
Nie, H., Li, S., and Liu, Y. Multi-agent deep reinforcement learning for resource allocation in the multi-objective hetnet. In 2021 International Wireless Communications and Mobile Computing (IWCMC), pp. 116-121. IEEE, 2021.
Li, L., Chu, W., Langford, J., and Schapire, R. E. A contextual bandii approach to personalized news article recommendation. In Proceedings of the 19th interna- tional conference on World wide web, pp. 661-670, 2010.
Li, X., Zhang, J., Bian, J., Tong, Y., and Liu, T.-Y. A co- operative multi-agent reinforcement learning framework for resource balancing in complex logistics network. In Proceedings of the 18th International Conference on Au- tonomous Agents and MultiAgent Systems. pp. 980-988, 2019.
Novoa, L. Jarrah, A. I., and Morton, D. P. Flow balancing with uncertain demand for automated package sorting centers. Transportation Science, 52(1):210-22 2018.
Oren, Y., Sagawa, S., Hashimoto, B., and Liang, P. Dis- tributionally robust language modeling. In P Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4227-4237, 2019.
Liu, G., Amini, A., Pandey, V., and Motee, N. Data-driven distributionally robust mitigation of risk of cascading failures. In 2024 American Control Conference (ACC), pp. 3264 3269. IEEE, 2024.
Panaganti, K. and Kalathil, D. Robust reinforcement learn- ing using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pp- 51 511-520. PMLR, 2021.
Liu, Z., Bai, Q. Blanchet, J., Dong, P., Xu, W., Zhou, Z., and Zhou, Z. Distributionally robust q-learning. In- ternational Conference on Machine Learning, pp. 13623- 13643. PMLR, 2022.
Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. Robust reinforcement learning using offline data. Ad- vances in neural information processing systems, 35: 2022.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, Multi-agent actor-critic for mixed cooperative-competiiie environments. Advances in neu- ral information processing systems, 30, 2017.
Panaganti, K., Wierman, A., and Mazumdar, E. Model-free robust é-divergence reinforcement learning using both offline and online data. In Proceedings of the 41st Inter- national Conference on Machine Learning, pp. 39324 39363, 2024.
Lu, M., Zhong, H., Zhang, T., and Blanchet, J. Distribu- tionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithm. arXiv preprint arXiv:2404.03578 2024.

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Perron, L. and Furnon, V. Or-tools. Google, 2024. URL deeel gooplll..goooo..o. laaggee.comm com/ optimizationi
Shi, L.,, Gai, J., Mazumdar, E., Chi, Y., and Wierman, A. Breaking the curse of multiagency in robust multi-agent reinforcement learning, arXiv preprint arXiv:2409.20067, 2024a.
Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. Ro- bust adversarial reinforcement learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Shi, L.,, Mazumdar, E., Chi, Y., and Wierman, A. Sample- efficient robust multi-agent reinforcement learning in the face of environmental uncertainty. In International Con- ference on Machine Learning, pp. 44909 44959. PMLR, 2024b.
Rahimian, H. and Mehrotra, S. Distributionally robust op- timization: A review. Open Journal of Mathematical Optimization, Volume 3, 2022.
Smirnova, E., Dohmatob, E., and Mary, J. Distributionally robust reinforcement learning. In Reinforcement Learning for Real Life, ICML Workshop, 2019.
Ramesh, S. S., Sessa, P. G., Hu, Y., Krause, A., and Bo- gunovic, I. Distributionally robust model-based reinforce- ment learning with large state spaces. International Conference on Artificial Intelligence and Statistics, pp. 100-108. PMLR, 2024.
Soma, T., Gatmiry, K., and Jegelka, S. Optimal algorithms for group distributionally robust optimization and beyond. arXiv preprint arXiv:2212. 3669, 2022.
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning, arXiv preprint arXiv: / /00.0700660666 2017.
Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
Villani, C. et al. Optimal transport: old and new, volume 338. Springer, 2008
Ren, A. 7. and Majumdar, A. Distributionally robust policy learning via adversarial environment generation. IEEE Robotics and Automation Letters, 7(2):1379-1386, 2022.
Wang. S., Si, N., Blanchet, J., and Zhou, Z. On the foun- dation of distributionally robust reinforcement learning. arXiv preprint Stti::::3110091111 2023.
Rockafellar, R. T. and Uryasev, S. Optimization of con- ditional value-at-risk. The Journal of Risk, 2(3):21-41, 2000.
Wiesemann, W., Kuhn, D., and Rustem, B. Robust markov decision processes. Mathematics of Operations Research, 38(1): 2013.
Rudin, W. Principles of mathematical analysis, 2021.
Russo, D. J., Van Roy, B., Kazerouni, A., Osband, Wen, Z., et al. A tutorial thompson sampling. Foundations and Trends@ in Machine Learning, 1 11(1):19,, 2018.
Wu, X. and Fu, J. Distributed robust optimization for multi- agent systems with guaranteed finite-time convergence. arXiv preprint arXiv:2309.0120 2023.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
Wurman, P. R.,, D' D'Andrea, R., and Mountz, M. Coordi- nating hundreds of cooperative, autonomous vehicles in warehouses. Al magazine, 29(1):9-9, 2008
Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902.04440 2019.
Xu, H. and Mannor, S. Distributionally robust markov decision processes. Advances in Neural Information Pro- cessing Systems, 23, 2010.
Xu, M., Huang, P., Niu, Y., Kumar, V., Qiu, J., Fang, C., Lee, K.-H., Qi. X., Lam, H., Li, B., et al. Group distribu- tionally robust reinforcement learning with hierarchical latent variables. In International Conference on Artificial Intelligence and Statistics, pp. 2677-2703, PMLR, 2023.
Shapiro, A. Distributionally robust stochastic programming. SIAM Journal on Optimization, 27(4):2258-2255, 2017.
Shen, Y., McClosky, B., Durham, J. W., and Zavlanos, M. M. Multi-agent reinforcement learning for resource alloca- tion in large-scale robotic warehouse sortation centers. In 2023 62nd IEEE Conference on Decision and Control (CDC), pp. 7137-7143. IEEF, 2023.
Yamagata, T. and Rodriguez, R. Safe and robust reinforcement-leamee learning: Principles and practice. arXiv preprint arXiv:2403. 18539, 2024.
12

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Zhang, K., Sun, T., Tao, Y., Gene, S., Mallya, S., and Basar, T. Robust multi-agent reinforcement learning with model uncertainty. Advances in neural information processing systems, 33:10571-10583, 2020.
Zhang, Z., Panaganti, K., Shi, L., Sui, Y., Wierman, A., and Yue, Y. Distributionally robust constrained rein- forcement learning under strong duality, arXiv preprint arXiv:2406. 15788, 2024.
Zhen, J., Kuhn, D., and Wiesemann, W. A unified the- ory of robust and distributionally robust optimization via the primal- worst-cquals-duall dual-bbst principle. Operaaionss Research, 2023.
13

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Appendix
A. Proofs of Theoretical Results
A.I. Proof of Lemma 3.1
Recall the definition of Dt in (5) and for any probability distribution P E 2t, we have
Ex-e [r(s,;
r(s,,ax))PXXxd
= f r() EapPics dx
(14)
= Σ 9g / [(())::X)P(X)) Σ qg Ex-r, [x)s
Then, taking the infimum both sides yields:
inf Ex<p [r(s,ax)]] Femt
inf Σ Ex [r(s,a;))) g.am g=1
(15)
inf Ex-P, [r(,, s6g
since optimum of linear program over simplex Am is obtained at vertices.
A.2. Proof of Lemma 3.2
In order to find the group distributionally robust action-valus function Q, we consider the worst-case immediate reward at each state-action pair (s, a) as:
f(s a) min Ex~,, [r(s, X)) < Ex-e (xs,;a(x;;;
(16)
gEG
for all unknown distribution E D.. Then, the worst-case return is given by:
R
x
a(s, <
Ex~P, (s,
(17)
k=0
k=0
where 7 [0, is the discount factor and the inequality holds for all possible sequences (P= with IPx Mt. Then, the worst-case action-value function under policy π can be expressed as:
( a) = Fx (((8,a) + + (xc+1: (e41)| 8, 8, ar
(18)
-Eg-P((1s.a) (F(s.a) a) + B (2((s'a))) (s'
and the optimal worst-case action-value function satisfies the Bellman optimality equation:
(" a) = F(s, a) + +yEss-((ssn)) max (ss''aa')
(19)
Following the result from (Liu et al., 2022), the distributionally robust Bellman Operator with ambiguity sets Q and R is given by
T (Q)
inf Er,.. (r(s, a)) + Ep... [max Q(6' ((d,
(20)
where Ps.a and Te.a denote the distributions of state transitions probabilities and reward functions respectively, with their corresponding ambiguity sets & and M. Since the distribution shift in the random variable X only affects the reward, i.e.,
14

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
the distribution of the reward function Ve.ar have:
Ton (s) - inf { (Er ((s, +7 max Q(s [ T2,
= =((s,a) +7 max Q(s',a')
(21)
inf (Ex-r, [r(s, a; X]]} + + max =(((,)) g6g
a'
We further show the distributionally robust Bellman operator is contraction mapping under the lo norm. Consider two arbitrary robust action-value functions Qi and Q2 such that
{)))))(s, =((,a)))))))))))) max ( ,')
(22)
= + max (2(5',
Finding the difference yields
- S max a'
- Q2((',2))
(23)
and taking the maximum over all feasible state-action pair (s, a) implies
S <1110 - -2llea
(24)
Since Y E [0, I]. this establishes that DR Bellman operator contraction mapping under the l- norm. By Banach's Fixed Point Theorem (Rudin, 2021), there exists unique fixed point Q such that Ta(Q") - Q*. Consequently, iteratively applying the operator ensures convergence to Q", proving the stability of the robust Q-learning algorithm. This contraction property of the DR Bellman operator was also addressed in (lyengar, 2005) when the ambiguity set is defined for transition probability.
B. Implementation Details for Simplified Robotic Sortation Warehouse
We define Markov
game for N agents
(representing
unique destinations) by the tuple
(N, N (A P, ( where:
(a) Agents: The set of N agents, each corresponding to unique destination.
(b) State Space: S denotes the joint state space.
(c) Observation Space: For each agent O C S represents its local observation at each time step, consisting of:
The total number of assignable chutes (uniform across all agents)
The number of chutes currently assigned to agent
(d) Action Space:
For each agent i, A' [0, 1] represents its action space, where each action determines if new chute will be allocated. An action value of 1 indicates the assignment of a new chute to destination i. The joint action space is defined as A = I A
(e) Transition Probability: P: A - [0, 1] specifies the probability of transitioning between states, representing the likelihood of packages being either successfully sorted or diverted to the recirculation buffer.
(f) Reward Function: For each agent i, r² S x A x X + RR defines the reward function, which penalizes the number of packages in the recirculation buffer resulting from the current chute allocation.
The model is completed with discount factor y E (0, 1) and initial state distribution po- In Section 5.1, we employ the Value Decomposition Network (VDN) (Shen al., 2023) combined with budget constraints in computing joint actions.
15

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
In the simplified robotic sortation environment, we fix the total induction volume at each time step 1200 packages. The number of incoming packages for each destination 1,..., N follows unknown normal distribution (u,, o). For each destination i, the probability that an incoming package is assigned to destination i is given by:
Φ ((-=#) Φ N
p incoming package belongs to i
(25)
where (22 = 1
dt is the cumulative distribution function of the standard normal distribution. This formulation ensures Σx= P(incoming package belongs to i} = 1. The distribution of packages across destinations is then determined by sampling 120 packages according to the probabilities defined in (25) at each time step.
v2r 1_x0 e
In Section 1, we assume the destination transportation vehicle has infinite capacity, meaning packages enter the recirculation buffer only when incoming packages are destined for location without an assigned eject chute. In this example, we construct the ambiguity as:
21 = P =
4g P P
q
(26)
9=1
where each Pg represents a normal distribution N (Mgg σ) with different Mgs. For our simulation, we construct the ambiguity
set using m = 9 groups with means (gg E {-4, 3, .,......., 0,...,4}. standard deviation a = 2, and index set G (1.2....,9);
C. Implementation Details for Large-Scale Robotic Sortation Warehouse C.1. MARL Structure
We define a Markov game over N agents (unique destinations) by the tuple (N, (4') P, (r')) 7, P.P).
where:
(a) Agents: N agents, each corresponding to a unique destination.
(b) State Space: S denotes the joint state space.
(c) Observation Space: For each agent i, O C S represents its local observation at each time step, consisting of: Number of packages recirculated until time for agent i
Total number of available chutes that can be assigned (uniform across all agents)
- Number of chutes currently assigned to agent i
(d) Action Space: For each agent A' represents its action space, determining the number of new chutes required. Actions take values in [0, 10], where an action value indicates the assignment of new chutes to agent at that time step. The joint action space is defined as A IT A'.
Transition Probability: P: x x x - [o, 1] specifies the probability of packages being either sorted by chutes or sent to the recirculation buffer. In the large-scale robotic sortation warehouse setting, the transition probability is function of the induction distribution P with random variable X X, which is addressed detail Appendix C.5.
(f) Reward Function: For each agent i, r' : S A X - IR defines the reward function, penalizing both the number of allocated chutes and the number of packages in the recirculation buffer. The recirculation is function of the induction distribution, which is defined Appendix C.2.
The model is completed with discount factor y E (0, 1) and initial state distribution Pa-
C.2. Induction Distribution
For a given sortation warchouse with D destinations and T time intervals (e.g., hours or minutes) in a day, we consider the random vector X = (X1,..., XDD TRDT where each X; 2 0 denotes the number of packages inducted for a specific
16

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Induction Volume Heatmap
ATL
LAX
ORD
8000
Induction Vol Volume
Destination
DFW
DEN
6000
JFK
SFO
4000
SEA
LAS
2000
MIA
0
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Hour of Day
Figure 10: A realization of the random variable X. The values are synthetic and do not represent actual Amazon induction volumes.
destination- time pair (d, {L...., D} TI. That X(d,t) represents the number of packages inducted for destination d during time interval t. A realization of the random variable X is shown in Figure 10.
We model the daily induction pattern as a random variable X drawn from an unknown distribution P., which we refer to as the induction generating distribution. We assume that the total daily induction volume is fixed at V across all days, as the MARL chute mapping policy is primarily influenced by the spatial-tempora distribution pattern rather than variations in total volume. The daily induction thus generated by sampling a realization X from
Definition C.1 (Induction Generating Distribution). We define the induction generating distribution as a multinomial distribution
= ( .,) ~ M(V.P....... .P).
where V N the total number packages inducted in a day, and the probability vector (PI....,P4 Pk) satisfies 2 0 and > 11=1 Pj 1. The support of X is the set
h Σe, 21 = )
2x) E No
1=1
Its probability mass function is given by
n!!
P(X1 = 21,..., Xx =xx)=
21122! ... zx! Pi ...pp"
for all (2))...,24) in the support.
For induction random variables X corresponding temporally proximate dates (e.g., within the week), assume they share common induction generating distribution IP MMVV, (((VPPi... PI.... PDT). .DD) In practice, this distribution is estimated via the Sample Average Approximation (SAA) method (Kim et al., 2015) using historical induction data from the relevant dates. Specifically, we collect historical induction records from Years 1-4 and partition them into 21 groups based on week numbers. For each group 9, we construct an empirical induction generating distribution Pg via SAA, modeled as a multinomial distribution using all induction observations within that group.
The group ambiguity set M in (5) then formed from the collection P2.., P21). The probability py of assigning an incoming package to the j-th destination- pair is estimated from the corresponding empirical multinomial distribution P., Given a total volume of V packages, the daily induction is generated by single sample from this distribution.
Amazon has publicly reported double-digit Year-over- Year (YoY) growth in retail sales in its Q4 carnings press releases over multiple consecutive years (Amazon.com) Inc., 2022; 2023; 2024). This sustained growth translates directly to the package sortation context, resulting in significant YoY increases in induction volumes. practice, package volume and its spatial distribution exhibit substantial variability across both seasonal and annual timescales. Seasonal effects introduce
17

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Table Averag Type-1 Wasserstein distances each week compared Week of the same year.
Table Average Type-1 Wasserstein distance Year (}).
Year
Week 2
Week 3
Week 4
Week 5
Year
76.87
202.96
147.17
241.80
Year
Year
Year
Year 2
36.46
70.47
301.25
123.45
1172.85
585.98
288.86
Year 25.81
23.87
72.91
78.82
Year 4
72.99
45.04
24.92
62.39
predictable fluctuations driven by major sales events and holidays (c.g., Black Friday, Cyber Monday, and year-end gifting), while annual trends reflect more gradual changes due to evolving customer behavior, shifts in logistics strategies, and overall business expansion.
We further quantify these distributional changes using the empirical Type-1 Wasserstein distance (Villani et al., 2008), shown in Table 4 and Table Specifically, Table 4 compares four representative weeks Week within each year, revealing substantial seasonal variation in induction distributions. Additionally, we find that the Wasserstein distance between consecutive days is typically on the order of 20, indicating moderate short-term variability. Table 5 illustrates yearly distributional shifts by comparing the same temporal slices across different years to the baseline in Year 1. These results highlight that inter-annual changes tend to be even more pronounced than intra-annual seasonal effects, posing sigmificant challenges for policies that rely on static assumptions or historical averages.
C.3. Dimension Reduction of the State-Action Space
To manage the dimensionality of the state-action space, we decompose the joint Q-network into a sum of local Q-networks. Each local network captures the expected return of an individual agent's chute mapping actions, while the joint network represents the expected return of the complete chute assignment across all agents. Specifically, we express the joint Q-network as:
Q(s, 0, ((s,a,)==>" -,- N' d'i,
(27)
where the input space scales linearly with the number of agents. While this decomposition is similar to that proposed in (Shen et al., 2023), approach learns single shared Q' network for all agents, rather than separate networks for each agent, resulting in improved computational efficiency.
C.4. Feasibility of Joint Actions
In unconstrained settings, agents would simply select actions that maximize their individual Q-networks, with the joint action being the collection of these individual choices. However, the chute mapping problem introduces resource constraints, as agents must share a limited number of available chutes. This necessitates coordination to allocate resources optimally among the agents based on their state-action values.
Given a budget constraint M on the joint actions, such that ΣN as S M, we formulate the following integer program to determine the optimal joint action that maximizes the joint Q-network for any state
maximize a...
s'
(28)
Σn a' M, a' EN
This integer program, which can be efficiently solved using commercial solvers such as Google OR- Tools (Perron & Furnon, 2024) or Xpress (FICO, 2023), serves two purposes: it generates feasible data for the replay buffer to compute the expectation in (3) during training, and it determines the optimal actions once learning has converged. Notably, this optimization step separate from the Q-leanning process.
18

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
CB Predicted
-51530.00
Observed
-51823.00
50000
-40000
-30000
-20000
-10000
Worst-Case Return
Figure 11: Comparison worst-case returns: estimation using Qcn for both immediate rewards and transition probabilities compared against the actual observed values from extensive state-action space exploration. The close alignment validates our approach of using QCB to approximate both components of the worst-case scenario in (30).
C.5. Distributionally Robust Bellman Operator in Large-Scale Robotic Sortation Warehouses
In lcage-scale robotic sortation warchouses, observe that the transition probability is dependent on the induction random variable X, which violates the assumption in Lemma 3.2. Consequently, we must compute:
inf y (x). max ((s's d)]
PER
for the distributionally robust Bellman operator. This leads
Tox (Q) a) = inf {Ex~r, [r(s, a; X)]} + inf { (x., max ( {
(29)
gEg
FEM
where the computation becomes infinitediimensional and practically intractable. To address this, during training, we approximate the distributionally robust Bellman operator with:
(+)( =inf [Ex~2,1r(x a; X)] + 7 E. p... max @(s', ((s'a')]3. 9Eg
(30)
which provides upper bound for the distributionally robust Bellman operator, shown by:
(Q)())( a) 2 inf gEg
{Ex=e, (rs,,a [)]} + inf
(X) max ((d
$69
2 inf gEg
(Ex~e, (r( + inf
PERR { (x
max Q Q(x)
(31)
(Q.
During training, we use (30) to construct the loss function (13) for DRMARI, where the optimization problem within (30) solved using the solution from the CB-based worst-case reward estimator QCB- In practice, this approximation (30) proves highly effective the worst-case return, with relative approximation error of Uox (s, a) Tox (Q) a) being less than 0.57% (see Figure 11). This small error margin indicates that U4y (s) a) does not impede DRMARL's ability to capture the worst-case return.
D. Additional Simulation Results and Discussions
D.1. Simplified Sortation Environments
As shown in Table 6, the DRMARL policy with QeB outperforms both the MARL policy and the DRMARL policy trained with random group selection across metrics. Here, random group selection refers to replacing Line 7 of Algorithm 2 with:
g' Uniform(G),
instead of:
g' - arg min QCB (st,a,, a4.g).
gEg
This confirms that QuB effectively enables the DRMARL policy to explore worst-case reward functions during training. Furthermore, when compared to a DRMARL. policy trained using exhaustive search over worst-case rewards for each
19

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Table Comparison of key sortation metrics across policies, averaged over m groups, each evaluated over 100 runs.
Policy
Recirculation Rate (+)
Throughput (t)
Recirculation Amount (1)
MARL
2.16% + 2.35%
11740.98 + 42.30
259.02 + 19.81
DRMARL (random)
1.56% + 1.45%
11812.77 30.14
187.23 17.26
DRMARL (with QCB)
0.56% + 0.18%
11932.21 + 24.12
67.79 + 5.16
DRMARL (exhaustive)
0.55% + 0.13%
11933.68 22.48
66.32 4.83
MARL (group-specific)
0.53% + 0.14%
11936.60 + 23.11
63.40 + 5.02
2000
1750
1500
Recirculation
1250
1000
CB-based Predictor
750
Exhaustive Search
Random Group Switch
500
MARL on Group #
250
50
100
150
200
250
300
350
400
Training Time (Seconds)
Figure 12: Training efficiency comparison in simplified robotic sortation warchouses.
state-action pair (8, a), the computationally more efficient. The last row presents the theoretical optimal performance of group-specific MARL policies (trained and tested the same group). DRMARL performs only marginally below this optimal baseline, demonstrating an effective balance between individual performance and distributional robustness.
approach achieves equivalent policy performance and robustness while being
The training efficiency comparison across different approaches is presented in Figure 12. DRMARL with exhaustive search requires the longest training time due to its comprehensive exploration across all distribution groups. In contrast, DRMARL with QсB converges significantly faster while matching the recirculation performance of the exhaustive search in initial stages, validating QCB 's ability to identify worst-case groups. DRMARL with random group selection shows faster convergence than the Qcy-based approach, but this is because random exploration does not guarantee capturing the worst-case reward functions. The group-specific MARL policy exhibits the fastest convergence due to the relative simplicity of its training task.
D.2. Large-Scale Robotic Sortation Environments
D.2.1. DETAILED EVALUATION RESULTS ACROSS ALL GROUPS
Table 7 presents detailed validation results comparing MARL and DRMARL chute mapping policies across all induction distribution groups from Years 1-4. The DRMARL policy demonstrates superior performance across most groups, achieving both higher package sortation throughput and lower recirculation rates. The only exceptions are two groups in Year 4, where the MARL policy shows marginally better throughput but at the cost of higher recirculation rates. This is expected behavior since the MARL policy is specifically trained on Year induction data, while DRMARL optimizes for robustness rather than throughput maximization. Overall, DRMARL achieves significant improvements, reducing recirculation by 80% on average while simultaneously increasing throughput by 5.62% on average.
20

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Table Relative key metrics improvements over MARL baseline trained on Year data large-scale robotic sortation warchouses.
RECIRCULATION RATE REDUCTION (%)
PACKAGE THROUGHPUT
PACKAGE RECIRCULATION AMOUNT REDUCTION (%)
GROUP NUMBER
YEAR
INCREASE (%)
94.75
31.66
94.21
93.19
23.47
92.55
3
94.62
35,69
94.26
93.85
25.94
93.85
95.90
19.50
95.67
2
90.47
5.79
90.44
7
2
91.27
5.24
91.11
8
2
77.22
4.13
77.22
2
83.34
2.49
83.34
10
2
85.12
5.07
85.12
11
2
92.85
6.97
92.82
12
3
84.22
4.54
84.59
13
3
83.14
15.89
81.57
14
3
85.53
4.17
86.04
15
3
83.63
5.51
83.58
16
3
84.24
5.52
84.19
17
4
34.18
-0.99
36.71
18
4
57.04
-5.85
62.34
19
75.78
4.47
75.73
20
4
66.59
9.85
64.18
21
4
75.07
5.64
75.83
D.2.2. ABLATION STUDY
Ablation studies were conducted across variety of warehouse layouts, focusing primarily varying the total number of eject chutes and unique induction destinations. These factors directly influence the action space size and the underlying transition probabilities, thereby significantly affecting policy performance. By systematically modifying these environmental parameters, we assess the adaptability and robustness of DRMARL compared to the standard MARL baseline under diverse operational conditions.
In Figure 13, reduce the total number of eject chutes available in the warehouse to model different warehouse operational conditions. As the number of chutes decreases, competition among destinations for shared eject chute resources intensifies, amplifying the complexity of the allocation problem. Despite this increased challenge, DRMARL consistently outperforms the standard MARL baseline across all tested chute configurations. This superior performance highlights DRMARL's robust ability to handle both the complexity and uncertainty inherent in real-world resource allocation tasks.
Meanwhile, Figure 14 explores the impact of progressively reducing the number of induction destinations on policy effectiveness. DRMARL maintains stable performance throughout most scenarios, demonstrating robustness to changes in destination diversity. However, when only 50% of destinations remain active, MARL surpasses DRMARL in performance. This phenomenon arises because DRMARL adopts conservative strategy that reserves chutes for potential future destina- tions, including those that may not materialize when half of the destinations are inactive. In contrast, MARL employs more aggressive allocation approach, prioritizing immediate assignment of chutes to currently active destinations without accounting for the risk of obstructing future inductions. This trade-off underscores DRMARL's cautious, risk-aware design, which generally yields better performance but can be slightly less optimal in highly reduced destination scenarios.
D.2.3. POTENTIAL LIMITATIONS
natural concern with DRMARL's contextual bandit-based worst-case reward estimator is whether it might induce overly pessimistic learning dynamics. Specifically, whether selecting the most adverse induction distribution each step could lead to excessively negative reward signals, resulting in instability or premature convergence to suboptimal policies, which is a well-known challenge in robust policy learning. To mitigate such effects, our framework aggregates induction data across multiple operational days rather than evaluating each day in isolation. This aggregation dampens the influence of extreme outliers and stabilizes the worst-case signal used for training. In our experiments, this approach proved effective, as no instability was observed due to extreme worst-case distributions. Nonetheless, we acknowledge that, in practical
21

---

Distributionally Robust Multi-Agent Reinforcement Learning for Dynamic Chute Mapping
Relative improvement to MARL (187 chutes)
80.0%
79.2%
50.5%
Improvement to MARL (same # of chutes)
9.1%
-60.6%
S1o0.
-115.9%
80.0%
-197.3%
-200
-138.0%
-190.3
60
-400
37.9%
-499.0%
-40
-600
30,
30.2%
4 25.9%
.17.0%
27.9%
4
20
MARL
DRMARL improvement to MARL
-800
-733.00 01%
180
160
140
120
100
Number of Total Chutes
Figure 13: Relative improvement recirculation rate (f) over the bascline MARL across different numbers of available chutes.
100
Relative Recirculation Improvement(S))
91.9%
91.6%
88.6%
87.0%
82.8%
80.0%
81.7%
75.7%
60
58.8%
47 77.3%
20
DRMARL
0
MARL
0.0%
100
90
80
70
60
50
% of Destinations Remain
Figure 14: Relative improvement recirculation rate (f) compared to the baseline MARL, shown across varying percentages of induction destinations remaining active.
deployment, highly anomalous induction patterns could arise. such cases, extreme outliers can be filtered prior to training, and fallback a heuristic policy can used when necessary. This trade-off reflects a known limitation of distributionally robust methods: while they offer improved resilience under uncertainty, they may become overly conservative in the face of extreme but low-pre owwprobabiility events.
22


Reviews:
[{"review_id": "h0nLU6rr4C", "reviewer": "Reviewer_AFKN", "summary": "This paper proposes the Distributionally Robust Multi-Agent Reinforcement Learning (DRMARL) framework for dynamic chute mapping in robotic warehouses. The integration of group Distributionally Robust Optimization (DRO) with a contextual bandit-based predictor to handle induction rate variations is the main contribution. However, there are some presentation issues for this paper. I will consider increase the score if the authors can make proper improvement.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, thoughtful questions, and encouraging feedback. Your suggestions greatly improve the paper. Below are our responses, following the order of the comments, with references to tables and papers prefixed by “R-” for clarity. If any referred content is missing here, please find it in the responses to other reviewers.\n\nThe contraction property of the DR Bellman operator is shown in the proof of Lemma 3.2 (Line 668). We explicitly highlight this in the revision for clarity, and replace the paragraph starting from Line 681 with:\n\nSince $\\gamma \\in [0,1]$, this establishes that DR Bellman operator is a contraction mapping under the $\\ell\\infty$ norm. By Banach’s Fixed Point Theorem [R-1], there exists a unique fixed point $\\tilde{Q}^*$ such that $\\tilde{\\mathcal{T}}_{\\mathcal{G}}(\\tilde{Q}^*) = \\tilde{Q}^*$. Consequently, iteratively applying the operator ensures convergence to $\\tilde{Q}^*$, proving the stability of the robust Q-learning algorithm. This contraction property of the DR Bellman operator was also addressed in [R-2] when the ambiguity set is defined for transition probability. \n\nDue to the character limit, the induction rate, in short, is a vector that contains number (integer) of packages inducted per destination and hour. A comprehensive definition of induction rate/pattern is provided in the revision.\n\nWe provide a more detailed explanation of DRO in the revision.\n\nIn this formulation, the state space $\\mathcal{S}$ represents the global state, while each agent $i$ has a local observation $\\mathcal{O}_i \\subset \\mathcal{S}$ containing partial information about the global state. The observation features are detailed in Appendix C.1, Line 739. While the state provides full system information, each agent’s observation is limited to local aspects relevant to its decision-making. This partial observability requires agents to act based on their own experiences and available information. We will refine our explanation in Line 162 to better highlight this distinction.\n\nWe place the Figure 5 near Line 176 in the revision.\n\nLemma 3.2 defines the DR Bellman operator, derives its explicit form for group DRO with MARL, and establishes its contraction property. Its formal definition appears in Equation (20), Line 655, with benefits discussed in Line 220. Notably, minimizing the worst-case Bellman error among groups does not necessarily yield a policy optimal under worst-case rewards. This is because the worst-case Bellman error reflects the worst-case deviation from the target Q-function but does not guarantee convergence to the optimal robust Q-function. Instead, the DR Bellman operator and its corresponding DR Bellman error ensure robustness. We refine Lemma 3.2 and the corresponding text for clarity in the revision.\n\nIn Table 1, each group is tested 100 times. STDs are omitted for table readability, and are included in the revision.\n\nIn Line 372, $X$ denotes the induction pattern (packages per destination per hour) and is a realization of a random variable following the induction generating distribution. Thus, the immediate reward function (recirculation) depends on $X$.\n\nConvex combinations of distribution groups aim to span the space of potential induction distributions, using available distribution groups to model the space of potential target distributions is a common approach in DRO. As we collect more production data, the distribution groups will expand to better capture year-to-year shifts without increasing the DRMARL training complexity. Figure 6 shows that the robust policy generalizes well to test distributions specifically designed outside $\\mathfrak{M}$ without theoretical guarantees. Its Type-1 Wasserstein distance to $\\mathfrak{M}$ is 818.19, while the average distance among distributions within $\\mathfrak{M}$ is 542.96.\n\nIn Figure 9, the reward is indeed increasing as the y-axis is flipped. Fixed in the revision.\n\nThe years are ordered from past to present. Year 4 was chosen arbitrarily. In Table R-1, training the MARL policy on Year 1 also results in significant performance degradation on OOD data, with similar outcomes as MARL trained on Year 4 when compared with DRMARL.\n\n|Year|2|3|4|\n|-|-|-|-|\n|Recirc Degradation(↓)|51.01% ± 0.17%|74.45% ± 0.22%|42.89% ± 0.64%|\n\n**Table R-1**: Relative recirculation rate degradation compared to the baseline MARL (trained on Year 1) on induction data from Year 2-3. \n\nThe detailed chute allocation action and strategy difference are omitted due to the character limit and are presented in the revision. Please kindly refer to the text under Table R-4 and Table R-5 for high-level strategy difference and step-wise recirculation outcomes between MARL and DRMARL.\n\n**References:**\n\n[R-1] Rudin, W. Principles of Mathematical Analysis (3rd ed.). McGraw-Hill, 1976.\n\n[R-2] Iyengar, G. Robust dynamic programming. Mathematics of Operations Research, 2005.", "claims_and_evidence": "This paper proposes DRMARL algorithm which is validated in simulated and large-scale environment.", "methods_and_evaluation_criteria": "The proposed algorithm makes sense.", "theoretical_claims": "The theoretical part is weak, but the current statements are reasonable.", "experimental_designs_or_analyses": "The experiments are reasonable.", "relation_to_broader_scientific_literature": "The paper proposes a new method for the chute mapping tasks in the field of operation research. This method has some potential to be applied to broader fields.", "essential_references_not_discussed": "The paper provides extended literature reviews.", "other_strengths_and_weaknesses": "Strengths\n•\tThe proposed algorithm that combines group DRO with MARL is novel and reasonable.\n•\tThe literature review is extended and well organized.\n•\tThe experiment results are comprehensive.\n•\tThe experiment shows significant improvement over the baseline MARL algorithm.\n\nSee weaknesses in the below part.", "other_comments_or_suggestions": "•\tOverall, the presentation is not very clear in several places. \n•\tFor the theory part, the two lemmas provided are not very strong. Could the author provide, e.g., a convergence analysis/the contraction property of the distributionally robust Bellman operator\n•\tInduction rate is the key concept of this paper, but the paper never gives a clear description of it. This makes the paper inaccessible to the audiences of ICML. Is it a real number or does it contain other information?\n•\tA similar problem applies to DRO.\n•\tWhat are the features of the observation? What are the differences between state and observation in this chute mapping problem since the paper models both state and observation in L162.\n•\tL176 refers to Figure 5 that appears several pages after the description.\n•\tFor Lemma 3.2, is it a definition or a theoretical result? What is the definition of distributionally robust Bellman operator? What is the benefit of using it in the algorithm?\n•\tHow many times do you run the policies for each group in Table 1? Why do the authors not show the std of throughput and recirculation amount?\n•\tThe notation X appears in L372 seems to conflict with the notation $X$ that appears in e.g., Eq. 11~13.\n•\tThe authors define the ambiguity set as a convex combination of past patterns. Could this modeling capture the patterns that gradually change year by year (e.g., the more and more packages for promoting seasoning of each year)? The authors try to address this in experiment results shown in Figure 6. But how far is the test distribution from the ambiguity set?\n•\tIn Figure 9, why the rewards decrease along with training?\n•\tFor Figure 5, are Year 1-4 sorted from past to current? A natural test would be training the model on Year 1 and test it on the later years. Why do the authors reverse?\n•\tForm the statistical values, DRMARL improves significantly over the MARL baseline. But could the authors provide detailed cases where the DRMARL policy outperforms the baseline? E.g., the strategies of it in extremely busy scenarios?", "questions_for_authors": "See the above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEo0CdZ2bU", "reviewer": "Reviewer_iKz7", "summary": "The paper addresses the “dynamic chute mapping” task in robotic warehouses, where packages must be assigned to chutes in the face of uncertain and shifting arrival (induction) patterns. It proposes a distributionally robust multi-agent reinforcement learning (DRMARL) framework, combining group distributionally robust optimization with MARL to prepare for worst-case variations in package flow. The authors introduce a contextual bandit–based predictor that selects the likely worst-case distribution group for each state-action pair, reducing computational overhead compared to exhaustively checking every group. They demonstrate in both a simplified and a large-scale warehouse simulator that their DRMARL approach consistently lowers package “recirculation” (re-sorting) and achieves improved throughput across a range of induction scenarios, including out-of-distribution ones.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and thoughtful questions.  Your suggestions greatly improve the paper. Below are our responses, following the order of the reviewer comments, with references to tables prefixed by “R-” for clarity. If any referred content is missing here, please find it in the responses to other reviewers.\n\nThe proposed framework builds on existing methods, but their integration is novel and strategically bridges gaps in DRRL, particularly related to scalability for large-scale multi-agent RL problems. In the robotic sortation warehouse, the reward (recirculation) function is implicit and highly nonlinear. Consequently, unlike typical well-structured DRO formulations with finite-dimensional convex equivalent formulations, applying similar methods to the complex, large-scale chute mapping problem is infeasible. With millions of packages sorted daily, estimating the worst-case reward for each state-action pair becomes computationally impractical using existing DRRL techniques. Although group-DRO makes worst-case reward evaluation finite-dimensional, the complexity modeling of the warehouse sortation and the large number of groups render exhaustive search still infeasible. Unlike supervised learning, where soft reweighting can avoid exhaustive search, such techniques cannot be directly applied to RL, as the worst-case group and reward vary across state-action pairs. An important contribution of our work is to leverage a contextual bandit to significantly reduce the exploration cost for estimating the reward of the worst case group. This approach effectively bridges the gap in **scalability** and **applicability**.\n\nWe conducted extensive experiments for the comparison between DRMARL and naive Robust RL. Please kindly find the detailed result in Table R-5. Compared to naive Robust RL, DRMARL benefits from its distribution awareness for more accurate estimation of the worst-case reward and robust action value functions. Even certain naive Robust RL policy (<2>) achieves a comparable cumulative recirculation throughout the day, DRMARL still significantly outperforms it in terms of the step-wise recirculation amount (both worst-case and CVaR), which is the direct metric for robustness when comparing the performance of robust policies.\n\nThe proposed DRMARL framework is designed to address general RL problems where the reward function undergoes distribution shifts. Since Lemmas 3.1 and 3.2 are not specifically tailored to resource allocation problems, the framework is broadly applicable to other DRRL settings. It can also benefit large-scale RL problems with group-DRO, particularly when an exhaustive search is impractical. We see strong potential for extending this approach beyond sorting problems, as DRMARL directly addresses reward function ambiguity (i.e., environmental uncertainty). The core DRMARL framework (Algorithms 1 and 2) remains unchanged when applied to other resource allocation problems, with only the environment varying. We are currently working on extending this framework to general RL problems beyond resource allocation, with early-stage promising results, and we will report our findings in a separate manuscript.\n\nMajor e-commerce companies have publicly reported double digit growth in retail sales YoY (Year-over-Year) in their Q4 earnings press releases on multiple years which translates in the package sortation context to large swings in induction rates YoY. Due to the double-blind policy, we will provide reference in the final paper. Package volume and distribution exhibit substantial seasonal and yearly shifts. Seasonality drives fluctuations during sales events and holidays, while yearly shifts reflect evolving customer behavior and business growth. To support our claims, we quantify distribution changes using the empirical Type-1 Wasserstein Distance [R-3], as shown in Table R-2. By comparing four selected weeks to Week 1 within each year, we reveal significant seasonal variations, while the Wasserstein distance between consecutive days is typically on the order of 20. The last column highlights yearly shifts by comparing the same period across years to Year 1, showing that yearly changes are even more pronounced than seasonal ones. Figure 5 and Table R-1, evaluated on production data, demonstrate that MARL struggles with large induction pattern changes, underscoring the necessity of DRMARL.\n\n|Week #|2|3|4|5|Avg Dist to Year 1|\n|-|-|-|-|-|-|\n|Year 1|76.87|202.96|147.17|241.80|0.0|\n|Year 2|36.46| 70.47|301.25|123.45|1172.85|\n|Year 3|25.81| 23.87|72.91|78.82|585.98|\n|Year 4|72.99| 45.04|24.92|62.39|288.86|\n\n**Table R-2**: The first four columns represent Type-1 Wasserstein distances for each week compared to Week 1 of the same year. The last column represents the average Wasserstein distance of each year to Year 1.\n\n**References:**\n[R-3] Villani, Cédric. Optimal transport: old and new. Vol. 338. Berlin: springer, 2008.", "claims_and_evidence": "The paper applies a group DRO (distributionally robust optimization) approach to multi-agent reinforcement learning (MARL) for a warehouse “chute mapping” problem. Their claim is that, by anticipating worst-case changes in package arrival rates, the learned policy is more “robust” than a regular MARL policy. They present some experiments in both small and large simulation settings that show lower “recirculation” (which is re-sorting effort) and better throughput when induction rates deviate from the usual pattern. The evidence is mainly the simulation results comparing their DRMARL approach to standard MARL or simpler baselines.\n\nWhile the simulation outcomes look decent, the evidence is somewhat specialized to their chute mapping scenario. They also have some basic group DRO math (like rewriting the Bellman operator) to explain how they handle uncertain reward distributions. But it’s not a deep theoretical contribution—more of an application.", "methods_and_evaluation_criteria": "They start from a standard MARL setting, add group DRO for different induction patterns, and then do a contextual bandit trick to pick the “worst-case” induction group. They evaluate performance by how many packages get recirculated, how many total packages get sorted (“throughput”), and how stable the policy is across multiple induction distributions.\n\nThe method is essentially an existing robust RL idea (DRO + MARL) applied to their warehouse environment. Their evaluation metrics make sense in that domain (recirculation rate, throughput). But it doesn’t push the boundaries for new RL or new optimization ideas. It’s more about showing that existing robust RL methods can handle uncertain arrival distributions.", "theoretical_claims": "The paper does not have theoretical contribution.", "experimental_designs_or_analyses": "The main experiment is to train a policy on multiple “groups” (induction patterns), then test on new patterns. They show that their approach has better worst-case performance than normal MARL. This setup is straightforward and in line with standard RL or robust RL experiments.\n\nThe chute system is a bit niche, so it’s unclear if this generalizes to bigger or different warehouse tasks. Also, they only compare to baseline MARL or a naive robust policy, so we don’t see how it might stack up against other specialized robust RL frameworks. The experiments seem valid but narrow.", "relation_to_broader_scientific_literature": "They cite work in robust RL, distributionally robust optimization, and multi-agent resource allocation, and they connect it to existing RL-based sorting approaches (like the MARL method from Shen et al. 2023). The main novelty is combining group DRO and multi-agent RL with a “contextual bandit” step for computational speed. But conceptually, these are known techniques—there’s not a big new theory angle here. They haven’t proven new theorems that break ground on robust MARL.", "essential_references_not_discussed": "Nothing noteworthy to me.", "other_strengths_and_weaknesses": "Strength:\n1. The paper tests an actual warehouse scenario, which is relevant to industry.\n\nWeakness:\n1. Conceptually, it’s mostly applying known ideas (group DRO, multi-agent RL, contextual bandits) rather than introducing a fundamentally new theory.\n2. The chute problem is a narrow application, so broader impact could be limited unless it’s widely adopted in automated warehouses.\n3. It’s not fully clear if the “worst-case distribution group” approach is truly necessary if the environment doesn’t vary that much in practice.", "other_comments_or_suggestions": "It would help if you provided concrete evidence or references showing that real warehouses do experience large swings in induction rates, especially since many readers at ICML may not be as familiar with that domain. Could you cite any operational research studies, internal data, or industrial reports that quantify these variations and explain why they're significant enough to justify a robust RL approach?", "questions_for_authors": "1. Do you see a path for applying the same idea to other resource-allocation tasks beyond warehouse sorting? If so, have you tested on simpler domains?\n2. Could a standard robust RL approach (without group DRO or contextual bandit) handle moderate variability in induction rates almost as well? It’d be helpful to see a direct comparison.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ENi2tf5mUj", "reviewer": "Reviewer_kJH7", "summary": "This paper attempts to optimize the dynamic chute mapping problem to optimize throughput/reduce recirculation rates by formulating the problem as a multi-agent RL problem. The authors then extend the vanilla MARL framework by introducing concepts from group distributionally robust optimization into the framework and further optimize the computation efficiency of the proposed framework by using a contextual-bandit based worse-case reward predictor. Experimental results on a simple toy problem and large scale simulations shows the efficacy of the proposed method.\n\n## Update after rebuttal\n\nI thank the authors for addressing my comments and I find the response sufficient. As such, I will maintain my previous recommendation of accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, thoughtful questions, and encouraging feedback. Your suggestions greatly improve the paper. Below are our responses, following the order of the reviewer comments, with references to tables and papers prefixed by “R-” for clarity.\n\nWe address all readability and presentation concerns in the revision.\n\nIn (1), actions are introduced as penalty to prevent MARL from over-allocating chutes to a few destinations, which could block incoming packages from other destinations. However, MARL still fails to generalize to OOD induction patterns. We will clarify this in the revision.\n\nFor small-scale problems, the observation space excludes recirculated packages, making the transition probability independent of induction $X$ and aligning with the assumptions in Lemma 3.2. For large-scale problems, as shown in Appendix C.5, the DR Bellman operator in Lemma 3.2 acts as an accurate estimator when the transition probability depends on $X$, which does not hinder DRMARL from learning an effective robust policy.\n\nAblation studies were performed under various warehouse layouts, primarily varying the number of chutes and unique induction destinations, which directly affect the action space and transition probability. STDs omitted due to character limits.\n\n|# of Chutes|50|100|115|125|135|150|187|\n|-|-|-|-|-|-|-|-|\n|DRMARL|**-499.00%**|**-197.33%**|**-115.54%**|**-60.59%**|**-28.64%**|**50.52%**|**79.97%**|\n|MARL|-730.97%|-358.70%|-190.91%|-138.85%|-84.25%|-47.42%|0% (baseline)|\n\n**TABLE R-3**: Relative recirculation rate improvement (↑) compared to the baseline MARL.\n\n|% of Destinations|50|60|70| 80|90|100 (max)|\n|-|-|-|-|-|-|-|\n| DRMARL| 83.58% | **82.82%** | **88.60%** | **91.57%** | **91.88%** | **79.97%** |\n| MARL| **86.99%** | 81.71% | 75.66% | 58.81% | 37.31% | 0% (baseline) |\n\n**TABLE R-4**: Relative recirculation rate improvement (↑) compared to the baseline MARL across different percentages of 120 destinations remains in the induction.\n\nIn Table R-3, DRMARL consistently outperforms MARL, demonstrating its effectiveness and robustness across different environments. In Table R-4, we examined the performance of both policies while gradually reducing the number of induction destinations. DRMARL maintained its performance, but MARL outperformed it when only 50% destinations remained. This occurs because DRMARL conservatively reserves available chutes for potential future inducts, even for destinations that have not yet appeared and may never appear with 50% destinations. In contrast, MARL aggressively allocates chutes to existing destinations without considering the risk of blocking newly inducted destinations.\n\nOverly pessimistic decisions is a common risk in robust policies. We mitigate this by grouping induction data from multiple days rather than treating each day separately, reducing the impact of extreme outliers. Our experiments did not exhibit instability from extreme worst cases. However, if such cases arise, one can remove extreme outliers to stabilize training and switch to a heuristic policy in practice when extreme induction patterns are detected. This is a foreseeable drawback compared to MARL since the policy may become overly conservative with extreme inductions.\n\nWe compare DRMARL with 3 Robust RL (RRL) approaches: <1> an adversarial agent deliberately blocking available chutes [R-4], <2> an adversarial environment with perturbed transition probabilities [Pinto et al., 2017], and <3> a (non-distributionally) robustified reward function [Wiesemann et al., 2013]. Implementation and result details are provided in the revision. While certain robust policy (<2>) achieve comparable cumulative recirculation performance to DRMARL, DRMARL additionally guarantees improved worst-case performance by explicitly optimizing for it. To validate this, we analyze the empirical distributions of recirculated packages across all time steps and groups. In Table R-5, we report the relative reduction in both the worst-case and CVaR (5% confidence level) of recirculated packages. DRMARL outperforms all RRL due to its distribution awareness, leading to more accurate worst-case reward estimation and robust action value functions, ultimately optimizing worst-case performance. This is crucial, as sudden surges in recirculation can cause robot congestion, delaying the entire sorting process. Since congestion is not explicitly modeled in the simulation, DRMARL’s advantage over RRL is expected to be even greater in practice.\n\n|Method|CVaR reduction(↑)|Worst-case recirc reduction(↑)|Recirc rate reduction (↑)|\n|-|-|-|-|\n|<1>|61.11%|53.57%|66.19%|\n|<2>| 58.48%|23.68%|77.09%|\n|<3>|58.97%|60.23%|70.43|\n|DRMARL|**77.09%**|**76.61%**|**79.97%**|\n**TABLE R-5**: Relative step-wise worst-case recirc amount, CVaR, and recirc rate reduction compared to baseline MARL.\n\n**References:**\n\n[R-4] Mandlekar et al., Adversarially Robust Policy Learning: Active construction of physically-plausible perturbations, IROS 2017.", "claims_and_evidence": "Yes, claims are supported", "methods_and_evaluation_criteria": "Yes, proposed method and evaluation criteria makes sense", "theoretical_claims": "Yes, the proofs of Lemma 3.1 and 3.2", "experimental_designs_or_analyses": "Yes, experiments are sound", "relation_to_broader_scientific_literature": "I believe this paper is generally related to the area of robust MARL as well as relevant to the areas of operation research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \n\nOverall, I believe this is a strong paper. The paper is well-written, organized and structured. The idea of applying Group DRO concepts to MARL and the introduction of a contextual bandit-based reward predictor to improve efficiency is novel to the best of my knowledge. The experimental results also supports the claims with sufficient theoretical justification and I believe the method introduced in this paper may also translate beyond the application of dynamic chute mapping. \n\nWeakness:\n\nCertain parts of the paper could be better organized in order to improve the readability for reader either not familiar with RL/Dynamic Chute mapping. The paper is also lacking of additional experiments and ablation studies for different parameters of the environment.", "other_comments_or_suggestions": "In Eq. 1, it is not clear why is there an addition $-2a^i_t$ in the reward function\n\nI would suggest mentioning the detail listed in the appendix of using an external tool to resolve the feasibility of joint actions in the main paper as it seems like a critical part of the framework. \n\nAdditionally, a slightly more detail explanation of the problem (dynamic chute mapping) would also improve the readability of the paper for readers not familiar with this application. \n\nIt is not clear why are the observation spaces different for the small scale problem vs the large scale problem, as listed in the appendix. \n\nI understand that the large scale experiments are already conducted on real world data, but as always, additional experiments on different simulation parameters, for example, varying the number of destination points/number of chutes and demonstrating that the conclusions still hold would make the paper stronger.", "questions_for_authors": "1. Given that the contextual bandit-based worst-case reward predictor selects the most adverse induction distribution at each step, have the authors observed scenarios where the worst-case reward signal is excessively negative, leading to instability in learning dynamics or premature convergence to a suboptimal policy? Additionally, how does the framework mitigate the risk of overly pessimistic policy updates that may arise from extreme worst-case scenarios?\n\n2. The paper compares the proposed method primarily against MARL and exhaustive search, but how does it perform relative to other robust RL approaches, such as adversarial training methods that explicitly perturb the environment to simulate worst-case scenarios?\n\n3. Are there any foreseeable drawbacks of the proposed method over existing methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: This paper proposes to use distributional robust multi-agent reinforcement learning for solving the destination-to-chute mapping problem in package sorting where there is uncertainty over the package induction rates.  

This paper was somewhat borderline. All reviewers appreciated the fact that this was an application-driven paper with an interesting and non-trivial application, and that the algorithm was well suited to the problem. Nevertheless, there were some concerns around the theoretical/algorithmic contributions with some reviewers pointing out the lack of theoretical results. Overall, however I believe that this paper is a good application-driven paper and so recommend acceptance.
Decision: Accept (poster)