Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL
Authors: Juan Claude Formanek
Omayma Mahjoub
Louay Ben Nessir
Sasha Abramowitz
Ruan John de Kock
Wiem Khlifi
Daniel Rajaonarivonivelomanantsoa
Simon Verster Du Toit
Arnol Manuel Fokam
Siddarth Singh
Ulrich Armel Mbou Sob
Felix Chalumeau
Arnu Pretorius
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
offline reinforcement learning
multi-agent reinforcement learning
offline multi-agent reinforcement learning
Primary area: reinforcement_learning
TL;DR: We extend autoregressive multi-agent sequence models, including Sable and MAT, to the Offline MARL setting and demonstrate that they significanlty outperform current state-of-the-art methods across a diverse set of benchmarks with up to 50 agents.

Abstract: A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to directly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable and combines it with a sequential form of implicit constraint Q-learning (ICQ), to develop a novel offline autoregressive policy update scheme. This allows Oryx to solve complex coordination challenges while maintaining temporal coherence over long trajectories. We evaluate Oryx across a diverse set of benchmarks from prior works—SMAC, RWARE, and Multi-Agent MuJoCo—covering tasks of both discrete and continuous control, varying in scale and difficulty. Oryx achieves state-of-the-art performance on more than 80% of the 65 tested datasets, outperforming prior offline MARL methods and demonstrating robust generalisation across domains with many agents and long horizons. Finally, we introduce new datasets to push the limits of many-agent coordination in offline MARL, and demonstrate Oryx's superior ability to scale effectively in such settings.

Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL
Claude Formanek* Omayma Mahjoub'
Daniel Louay Ben Nessir' Sasha Abramowitz' Simon Du Toit Felix Chalumeau
Ruan de Kock!
Wiem Khlifi!
Arnol Fokam
Siddarth Singh*
UIrich Mbou Sob
Arnu Pretorius
IeetaDeep
University Cape Town
SStellenbosch University
Abstract
A key challenge in offline multi-agent reinforcement learning (MARL) is achieving effective many-agent multi-step coordination in complex environments. In this work, we propose Oryx, a novel algorithm for offline cooperative MARL to di- rectly address this challenge. Oryx adapts the recently proposed retention-based architecture Sable (Mahjoub et al., 2025) and combines it with sequential form of implicit constraint Q-learning (ICQ) (Yang et al., 2021), develop novel offline autoregressive policy update scheme. This allows Oryx to solve complex coor- dination challenges while maintaining temporal coherence over long trajectories. We evaluate Oryx across diverse set of benchmarks from prior works-SMAC RWARE, and Multi-Agent MuJoCo Maaooo---vveringgg tasks of both discrete and con- tinuous control, varying in scale and difficulty. Oryx achieves state-of-the-art performance on more than 80 % the 65 tested datasets, outperforming prior offline MARL methods and demonstrating robust generalisation across domains with many agents and long horizons. Finally, we introduce new datasets to push the limits of many-agent coordination in offline MARL, and demonstrate Oryx's s superior ability to scale effectively such settings.
Introduction
Cooperative Multi-Agent Reinforcement Learning (MARL) holds significant potential across diverse real-world domains, including autonomous driving (Cornelisse et al., 2025), warehouse logistics (Krnjacc et al., 2024), satellite assignment (Holder et al., 2025), and intelligent rail network manage- ment (Schneider et al., 2024). Yet. deploying MARL in realistic settings remains challenging, as learning effective multi-agent policies typically requires extensive and costly environment interaction. This limits applicability to safety-critica economically constrained domains, where the cost of real-world experimentation prohibitively high. Fortunately, in many such settings, large volumes of historical train schedules, traffic records, or robot navigation trajectories available. By developing effective methods to distil robust, coordinated policies from these static datasets, we may unlock their full potential.
logged data-sue
Offline MARL aims to address this exact challenge, training multi-agent policies solely from pre- collected data without further environment interaction. However, learning in the offline multi- agent setting introduces two primary difficulties. The first, well-studied challenge is accumulating extrapolation error, which occurs when agents select actions during training that fall outside the distribution of the offline dataset. This issue compounds rapidly as the joint action space grows
"Correspondence: c.formanek @ @instadeep.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

exponentially with the number of agents (Yang et al., 2021). Recent works (Wang et al., 2023; Matsunaga et al., 2023; Shao et al., 2023; Bui al., 2025) have made progress on addressing extrapolation error through policy constraints or conservative value estimation. However, such methods were typically only tested on settings with relatively few agents. Leaving open the question of whether such methods are able to scale to more agents.
The second issue is miscoordination, arising from the inability of agents to actively interact in the environment. Offline training forces agents to rely entirely on historical behaviours observed in the dataset collected from other (often suboptimal) policies, risking the development of incompatible policies. Tilbury et al. (2024) highlight how this miscoordination problem can significantly degrade performance in cooperative settings. Some recent works have tried address this problem (Barde et al., 2024: Qiao et al., 2025). However, it's unclear how well these approaches scale, particularly when long temporal dependencies and many agents are involved.
To jointly tackle the two fundamental challenges of extrapolation error and miscoordination in many-agent settings, we propose Oryx, novel offline MARL algorithm that unifies scalable sequence modelling with constrained offline policy improvement. Oryx integrates the retention- based sequence modelling architecture of Sable (Mahjoub et al., 2025) with an enhanced offline multi-agent objective based on ICQ (Yang et al., 2021). Furthermore. by leveraging a dual-decoder architecture- simultaneously predicting actions and Q-values Oryx is able to use counterfactual advantage (Foerster et al., 2018), enabling robust and extrapolation-set safe policy updates. Finally, Oryx's sequential policy updating scheme explicitly addresses miscoordination by conditioning each agent's 's policy update on the actions already executed by other agents in sequence, thus ensuring stable policy improvement.
We extensively evaluate Oryx across a broad set of challenging Sanchmarks-SMAGE RWARE and Multi-Agent MuJoCo-that include discrete and continuous control, varying episode lengths. and diverse agent densities. Oryx sets a new state-of-the-ar in offline MARL, outperforming existing approaches in more than 80% of the 65 evaluated datasets. Furthermore, we specifically test Oryx in large many-agen settings. To achieve this, we create new datasets (with up to 50 agents) and show that Oryx maintains its superior performance at such scales. Overall, by addressing both extrapolation error and miscoordination in scalable sequence modelling framework, Oryx significantly ad advances offline MARI,, bringing us closer to being able to reliably deploy cooperative, multi-agent policies learned entirely from static data in complex, real-world domains. To accelarete future research in this direction, we make all of our datasets and code available on GitHub'.
2 Background
Preliminaries problem formulation and notation. We model cooperative MARL as a Dec- POMDP (Kaelbling al., 1998) specified by the tuple LN,S, S. P. R. (2' (E, hex, 7)- At each timestep f, the system is in state s E S. Each agent E A selects an action of E A', based on its local acttio-obseerattio history F/ (oh ......, of), contributing A = II A'. Executing a, in the environment gives a shared reward r, = R(s,, ax), transitions the system to a new state St+1 ~ P(-ss;, at), and provides each agent i with a new observation of ~ E- a.) to update its history as T/41 (T', at, o The goal is to learn joint policy π (a) that maximizes the expected sum of discounted rewards, [(T) E,, 1E 7're]-
joint action a,
In our work, we adopt the multi-agent notation from Zhong et al. (2024b). Specifically, let i1.m denote an ordered subset (i,,..., km} of N, then ii refers to its complement, and for m = 1. we have i and - i, respectively. The h4th agent in the ordered subset is indexed as ig. The action-value function is then defined as
Qiam (T, (T,a'l.n)
F. f1om -ammm/mm---- from [Q(T, a f110 (1.m) )}
Here, Qéem (T, a fm:m) evaluates the value of agents it taking actions a f having observed while marginalizing out a $1:m When m = n (the joint action), then i1.3 E Sym(n), where Sym (n) denotes the set of permutations of integers L,.... n. which results in Q'1n (T, a²tm) being equivalent to Q(T, a). When m = 0, the function takes the form of canonical state-value function VTT)..
Moreover, consider two disjoint subsets of agents. Jisk and im.. Then, the multi-agent advantage

---

function of d1mm with respect to J1:k is defined as
(1) The advantage function f (T, a/1=,, a evaluates the advantage of agents dim taking actions aft.m having observed T and given the actions taken by agents J1:k are a with the rest of the agents' actions marginalized out in expectation.
A (T, a Fiik a 1mm) ) = Qu (T, alis a alim) - Qliak a1*)).
Heterogeneous Agent RL framework principled algorithm design for MARL. Zhong et al. (2024b) show how practical and performant multi-agent policy iteration algorithms can be designed by leveraging sequential policy updates. Underlying much of this work is the multi-agent advantage decomposition theorem (Kuba et al., 2021), which states that
n1
A(T, a) A (T, (a = Σ (Ta
j=1
In essence, the above theorem ensures that if policy iteration algorithm is able to update its policy sequentially across agents while maintaining positive advantage, i.e. A ( (T, att--t,ad a²) > 0.jj guarantees monotonic improvement. This result has guided the design of several recent algorithms under the heterogeneous agent RL framework (Zhong et al., 2024b), including multi-agent variants of PPO (Kuba et al., 2022a), MADDPG (Kuba et al., 2022b) and SAC (Liu et al., 2024a).
Sable efficient sequence modelling with long-context memory. Sable (Mahjoub et al., 2025) is a recently proposed online, on-policy sequence model for MARL. It is specifically designed for environments with long-term dependencies and large agent populations. Its key component the retention mechanism, inspired by RetNet (Sun et al., 2023), which replaces coftmax-basee attention with decaying matrix component. This allows Sable to model sequences flexibly: either as recurrent (RNN-like), parallel (attention- like), or chunkwise (a hybrid of both). During training, Sable uses chunkwise retention for efficient parallel computation and gradient flow, while execution relies on recurrent mode that maintains hidden state to capture temporal dependencies and ensure memory- efficient inference. Sable's s architecture consists of an encoder that processes per-agent observations into compact observation embeddings and value estimates, and a decoder that outputs predicted logits and actions. Training is performed via standard policy gradient, using the PPO objective (Schulman et al., 2017). Further details Sable's retention mechanism can be found in the Appendix.
Implicit constraint Q-learning (ICQ) - effective offline regularisation. The key idea in ICQ (Yang et al., 2021) is to avoid out-of-distribution actions in the offline setting by computing target Q-values sampled from the behaviour policy μ (extracted from the dataset), instead of æ, such that the Bellman operator becomes (T*Q) a) = >E a' Q')Q(T'' (Q(TT'a'),, a')]. where p(T', a') x(a'ir)) is an importance sampling weight. However, obtaining accurate μ is itself difficuult. Therfore, ICQ instead constrains policy updates such that DKL(T ll p)[T] S e, with the corresponding optimal policy taking the form xb41(a/T) ((alT) exp a
(2.(7.a)
Here, > is Lagrangian coefficient in the unconstrained optimisation objective and Z(T) = Σ ((ā (T) exp (+ (T, ū)) is p(T', = Fe+1 (a'lr") p(aa((")
the normalisation function. Finally, solving for only requires sampling from (1) as
gives the ICQ operator (that
Bla'' m Z(7') exp (Q(''aa))
Q(T (rr''a']
TicoQ(T,
During training, the critic network parameters Φ, and the policy network parameters 8, are optimised over a data batch B by minimising the following losses:
Critic loss:
1 jo(0) B' r + Z(7")
(Qo(())a)) 2o (r'a')-QQ(TTo)) ao(T,a)
exp
a
Policy loss
1 log(To(a|7)) exp [((,,
(+) = ErLB [
[T)[
Er,anD
Z(r)
Here, o= denotes the target network parameters. In practice, the normalising partition function computed E(rajer exp Q(T, a)/a, over the mini-batch B sampled from the dataset D.
3

---

Sbrervation
Ereaded
Pollcy
Incoder
Seteriie
0
ActSone
V-Valuen
Decader
Bark
Baak
Figure 1: Oryx's model architecture. The green blocks indicate the inputs to the model (in yellow), sourced from the dataset online experiences (in blue). First, sequence of agent observations from timestep to 1+k is passed through the encoder. Inside each retention block, the network performs joint reasoning over the agents (A1..... an) and temporal context (t.... t + k), producing encoded representations each timestep. These encoded observations, along with the actions from the dataset,
are passed to the decoder, which has two heads. One head returns Q-values, while the second returns policy distribution for each agent for the full sequence.
3 Method
Oryx integrates the strengths of Sable's auto-regressive retention-based architecture and ICQ's s robust offline regularisation to address the specific challenges of long-horizon miscoordination and accumulating extrapolation error in offline MARL. particularly effective model class that can naturally instantiate autoregressive policies is sequence models, with notable examples the multi- agent transformer (MAT) (Wen et al., 2022) and Sable (Mahjoub et al., 2025). These models naturally represent the joint policy
product of factors that similarly decompose ruto-regressively as (alr) = I[= T's (n's (T, al15-1).
Network architecture. Oryx modifies the Sable architecture, which features retention blocks to efficiently handle long sequential dependencies among agents. In particular, we employ dual-output decoder structure where the decoder outputs both policy logits and Q-value estimates. The logits correspond to the action probabilities for each agent and the Q-value estimates explicitly capture the relative value of each agent's 's actions given the current historical context. Unlike the original Sable network, we do not have value head as part of the encoder and instead train the encoder and decoder end-to-end with the combined critic and policy losses. A diagram of the computational flow of the Oryx architecture is provided in Figure 1.
Autoregressive ICQ loss. Next, we use the Oryx network's autoregressive policy structure and the advantage decomposition theorem (Kuba et al., 2021) to derive Theorem 1 (for proof see Appendix). Theorem For an auto-regressive model, the multi-agent joint-policy under ICQ regularisation can be optimised sequentially for = 1,... over data batch B as follows:
A (T, a
arginax E a NB
t11s
Zo:d (a
| [T,())-1) exp
x²
where t (T) IT- Eas n² (att |T, a aftttt) exp (
To arrive sequential SARSA-like algorithm similar ICQ, we update the critic by sampling target actions from the dataset and update the Q-vuuue function with implicit importance weights as
27
Q²2 (T' (r'a"(1
exp
Q = argmin Eg
Q¹ (T') - (T) a's)
r r
Z(T')
Q'
Finally, the centralised advantage estimate in the multi-agent policy gradient is susceptible high variance. In particular, Kuba et al. (2021) provides upper bound the difference in gradient

---

variance between independent and centralised learning when using the standard V-value function as (eB,)" baseline as (n (n--))-- 1- where B; = supr,a .a Ve log (
[T) = supr,a-t,a' [A'(T, a a=","") ). and = max; Eg- This bound grows linearly the number of agents. We can arrive at a better bound that removes the (n 1) term by employing a counterfactual baseline (Kuba et al., 2021) 1) as used in
Foerster et al. (2018) such that
A (T. a f1
Σ
Q(T a alim)
Σ (a'maa" atm
mm
T, a fiom ()Q(T, a afum)
This completes the sequential updating scheme for Oryx. full algorithm description is provided in Algorithm 1. Next, describe the architectural details allowing Oryx model long-term dependencies across sampled trajectories from the dataset.
Algorithm 1 Oryx's 's sequential updating scheme with autoregressive ICQ regularisation
I: Initialise the joint policy and critic network parameters 0,
2: for 1,... do
3: Sample mini batch {TT a a r', a (d } of trajectories from the offfine dataset D.
4: Draw permutation di:n of agents at random.
5: for 1 : do
6:
Update the critic:
(T' (
exp
Q argmin EB 4'
r+7
Q2 (T', a a(tsis) Q3 a
Z(T')
Calculate advantage:
A (T, (1))
Q(T, a a
Σ fm (a T,a flmm 2)Q(T, a (1:m)
a'm
8:
Update the policy:
AArss (A (11))
#k+1 argmin Es
log(n>(('>) a(,a exp
(
Z(TT)
9: end for
10:
end for
4 Results
In this section, we conduct detailed empirical evaluations to substantiate the core contributions of our proposed algorithm, Oryx. We begin by rigorously validating its key design components: (i) sequential action selection for agent coordination, (ii) memory mechanism for temporal coherence. and (iii) autoregressive ICQ for stable offline training. To assess scalability, we subject Oryx to a demanding multi-agent setting involving up to 50 agents. Specifically, we use the Connector environment (Bonnet et al., 2024), which progressively increases coordination complexity as the number of agents grows. Finally, we perform an extensive and diverse benchmarking study across more than 65 datasets from prominent MARL benchmarks, covering a wide range of tasks in SMAC (Samvelyan et al., 2019), MAMuJoCo (Peng et al., 2021), and RWARE (Papoudakis et al., 2021).
4.1 Validating Oryx's core mechanisms
Inspired by Osband et (2020), we designed a T-Maze environment (see Figure 2) to specifically isolate long-horizon multi-agent coordination challenge. On the first timestep, two agents each independently select colour ("green" or "orange").. On the subsequent timestep, they spawn randomly at the bottom of the maze and briefly observe their choice of goal action and the goal locations (e.g..

---

T-MAZE
Algorithm
Replay
Expert
I-CQ
0.0 = 0.0
0.0 = 0.0
MAICQ
0.0
0.0 0.0
Oryx- w/o Ret-eeesee Actions
0.0 1. 0.0
0.0 0.0
Oryx Memory
0.63 0.04
Oryx w/o ICQ
0.0 1 0.0
0.0 0.0
Oryx
0.99 0.94 0.03
(a) T-Maze
(b) Performance across baselines and ablations
Figure 2: Evaluating long horizon coordination. To issolate the importance of the different compo- nents of Oryx minimal two-agent environment, T-Maze was designed. In the environment the target states are revealed only at the first timestep, requiring agents to retain goal information throughout the episode and carefully coordinate at the end. Oryx successfully solves the task only when all components are present, while baseline methods fail to perform across both the replay and expert datasets.
orange-left, green-right). Without further observing their goal, agents must navigate to their respective targets, manoeuvring the junction to avoid collision and delay. Success requires: (i) selecting different colours initially for effective coordination, (ii) retaining memory of their goal choice action, and (iii) efficiently manoeuvring around one another. Agents receive a team reward of 1 if both agents reach the goal. We generated two datasets: a mixed dataset containing primarily unsuccessful trajectories and smaller number of successful examples, and expert dataset consisting solely of successful trajectories. Dataset statistics are detailed in the appendix following Formanek et al. (2024a). We evaluated several baselines: fully independent ICQ learners (I-ICQ), ii) the CTDE variant with mixing networks (MAICQ (Yang et al., 2021)), and iii) targeted Oryx ablations disabling autoregressive action selection, memory, and offline sequential ICQ regularisation individually to isolate their contributions.
As shown in Figure 2, Oryx consistently achieves optimal performance with both datasets, while baseline ICQ variants fail regardless of value decomposition. The ablation results confirm that each of Oryx's 's core components sequential action selection, memory, and [CQ-based offline regularisation individually crucial for enabling effective long-horizon coordination among agents.
4.2 Testing coordination in complex many-agent settings
Having validated the importance of Oryx's xs core components in smaller-scale setting, we now stress- test the algorithm in significantly larger and more challenging many-agent coordination scenarios. To effectively evaluate this, we select the Connector environment from Bonnet et al. (2024), which inherently becomes more complex as agent density increases. It is important to highlight that naively increasing agent numbers does not necessarily enhance task complexity; it may even simplify certain problems where coordination is less critical. Despite its popularity, many SMAC scenarios tend to become easier as the number of agents increase (for example, reported performance often takes on the following task order, 5m_vs_66 > 8m_vs. vs __m 10m_vs_ 1_11 1m (Wen et al., 2022; Yu et al., 2022; Hu et al., 2023)). In Connector, agents spawn randomly on a fixed-size grid and are each assigned random target location. Agents must navigate to their targets, leaving impassable trails behind them that can obstruct other agents. The necessity for careful coordination sharply increases as agent density grows, making Connector ideal for testing Oryx.
Datasets for Connector were generated by recording replay data from online training sessions using Sable (Mahjoub al., 2025). Due to the substantial volume of data generated by the online systems (20M timesteps). we applied random uniform subsampling on each dataset to select IM timesteps. Statistical summaries of these datasets are provided in the appendix. For comparison, we again use MAICQ (Yang et al., 2) as our baseline. MAICQ is particularly valuable as comparative algorithm because it incorporates several widely used components that differ notably from Oryx, specifically: (i) RNN-based memory, (ii) global state- catte-conditioned value decomposition, and (iii) the original noneeuvoeeeeessive ICQ loss. We normalise the results in Fu et al. (2020) score- meore-rendom score where random_ _ccre is the average score achieved by
morm_sse score
expert_score cactoorerranoon....m randonn__coee

---

(Ours)
1.00
0.75
0.50
0.25
0.00
-0.25
-050
10 23
Number agents
(a) 23-Agent Connector
(b) Results on Connector
Figure 3: Evaluating Oryx on muny-agent settings. We compare Oryx, with its autoregressive ICQ loss and sequence model architecture, to MAICQ which is non-autoregressive CTDE algorithm. The two algorithms are trained on datasets from Connector (Bonnet al., 2024) scenarios with increasing numbers of agents. While the performance of MAICQ dramatically degrades scenarios with large numbers of agents, Oryx's performance remains robust.
agents taking random actions and the expert_score is the performance of the online system at the end of training.
Our results indicate that at lower agent counts, performance differences between Oryx and MAICQ modest. However, from around 23 agents, Oryx significantly outperforms MAICQ. achieving near-expert performance compared to only 25% of expert performance by MAICQ. Interestingly, between 23 and 30 agents, MAICQ's performance temporarily improves. However, this is likely due to the fact that we had to marginally increase the grid size at this point, making the environment slightly less dense (see Figure 7). Nonetheless, as agent count grows from 30 50 (with grid size held constant), coordination complexity dramatically escalates, clearly underscoring Oryx's 's superior capability to manage increasingly challenging coordination demands.
4.3
Demonstrating state-of-the-art performance existing offline MARL datasets
Finally, we evaluate Oryx against a wide range of current state-of-the-arn offline MARL algorithms across various widely recognised benchmark datasets. The datasets come from various original works, including Formanek et al. (2023a); Pan et al. (2022); Shao et al. (2023); Wang et al. (2023); Matsunaga et (2023) and were obtained via OG-MARL (Formanek et al., 2024a), a public datasets repository for offline MARL. We compare Oryx against the latest published performances on each dataset, including reported results from Matsunaga et al. (2023); Shao et (2023); Bui al. (2025); Li al. (2025). We follow similar methodology to that of Formanck et al. (2024b), where trained our algorithm for a fixed number of updates on each dataset and reported the mean episode return at the end of training over 320 rollouts. We repeated this procedure 10 times with different random seeds. We summarise our results in Figure and provide detailed tabular data in the appendix, ie. mean and standard deviations of episode returns across all random seeds for each dataset. We perform a simple heteroscedastic, two-sided t-test with 95% confidence interval for testing statistical significance, following Papoudakis al. (2021) and Formanek et al. (2024b).
SMAC the most widely used environment the offline MARL literature. However, as Formanek et al. (2024a) pointed out, different authors tend to not only use different datasets in their experiments but also entirely different scenarios. This makes comparing across works very challenging. As such, tested Oryx across as many SMAC datasets that were available us (Meng al., 2022; Formanck et al., 2023a; Shao et al., 2023) and compared its performance against the latest state-of state-of-the-art result could find in the literature (Shao et al., 2023; Bui et al., 2025; Li et al., 2025). In total, tested on 43 datasets, spanning 9 unique scenarios (2c_vs. vs_ 22_2_42g.622 3s vs 3s_vs___________________________ 52, 3m, 2s32, 5m_ Sm_vs__ ___vv_66_ 6m, 2h_vs_8z 8m, 3s52_ve vs_ 3s62 and corridor). In total, Oryx matched surpased the current state-of-the-art on 34 of the datasets (79%).
MAMuJoCo is the most widely used multi-agent continuous control environment and second most popular source of datasets for testing offline MARI algorithms. We tested Oryx on datasets from Pan et al. (2022) and Wang et al. (2023). These span several scenarios with varying numbers of agents
7

---

MAKO
3.5
275
238
1.75
1.5-
Nomalaan
0.75
0.5-
028
consuderr Musphhor
wo Acthol meeeti
del bettes suptettinn
bastaii Bodding
soaiii the seveen
Dalasot
(a) SMAC
Ore
OPOI
OPICA
ONLY
IOOL
UpRERS
ICU
In corlidnn happoshecas skachh surag
Datave:
Delare
(C) RWARE
(b) MAMuloCo
Figure 4: Performance of Oryx across diverse benchmark datasets fromm prior literature. Scores are normalised relative to the current state-of-the-art, with values above indicating that Oryx surpasses previous best-known results. Unnormalized scores are provided in the appendix. Gold stars indicate instances where Oryx matches or exceeds state-of-the-art performance, while black stars denote otherwise.
including 2x3 HalfCheetah, 6x1 HalfCheetah, 2x4 Ant and 3x1 Hopper. We compared our results to recent state-of-the-ar results by Shao et al. (2023) and Bui et al. (2025). On 14 out of the 16 datasets tested, Oryx matched or surpassed the current ttaee-of-the-art.
challenging MARL environment that requires long-horizon coordination. Agents must learn to split up and avoid overlapping in order to optimally cover the warehouse. Moreover, episodes are quite long and have a very sparse reward signal, spanning 500 timesteps as compared to less than 100 and a dense reward on most SMAC scenarios. Matsunaga et al. (2023) generated and shared a number of datasets on RWARE, with varying numbers of agents (n [2, 4, 6)) and warehouse sizes. Oryx set a new highest score on all six of the available datasets, and in several cases improved the ttaee- by nearly 20%.
RWARE (Papoudakis et al., 2021) a
4.4 Comparing ruto-regressive sequence modeling architectures
Oryx extends auregressive MARL sequence modeling into the offline MARL setting by leveraging Sable (Mahjoub et al., 2025) as its network architecture backbone. While prior sequence models such as MAT (Wen et al., 2022) have demonstrated strong performance in online settings, the decision
8

---

Table 1: Comparing Oryx and MAT+ICQ. In order to isolate the effect of using the Sable network architecture as the backbone of the Oryx network, we compare it to an offline varient of MAT that uses the same autoregressive ICQ loss as Oryx (MAT+ICQ). We report several aggregated metries with 95% stratified bootstrap confidence intervals (Agarwal et al., 2021) across all SMAC and RWARE datasets. To Facilitate aggregation across scenarios with different episode returns, scores were first normalised by the highest return in each dataset respectivly. We see from the results that on both environments the mean, median, interquartile mean (IQM), and optimality gap of Oryx is superior to MAT+ICQ.
SMAC
RWARE
MAT+ICQ
Oryx
MAT+ICQ
Oryx
Median 1
0.71 [0.62, 0.74]
0.91 [0.88, 0.92]
0.85 [0.84, 0.86]
0.89 [0.88, 0.90]
IQM*
0.67 [0.66. 0.69]
0.87 [0.86, 0.88]
0.85 [0. 84, 0.86]
0.89 [0.89, 0.90]
Mean f
0.63 [0.62. 0.64]
0.77 [0.76, 0.79]
0.84 [0.83, 0.84]
0.89 [0.88, 0.90]
Optimality Gap &
0.38 [0.36. 0.39]
0.23 (0.22, 0.25]
0.16 [0.16, 0.17]
0.11 [0.10, 0.12]
to adopt Sable as the backbone for Oryx was motivated by its superior scalability, efficiency, and stability when trained across large populations of agents.
However, further quantify the advantages of our design choices, we conducted an ablation compar- ing Oryx with an offline variant of MAT using identical training procedures and the same autoregres- sive ICQ loss. This experiment isolates the impact of choosing Sable over MAT for Oryx's sequence model component. Following the evaluation methodology proposed by Gorsane et al. (2022), we report aggregated metries across all SMAC and RWARE datasets. To facilitate aggregation across scenarios with different expected episode returns, results were normalised by the highest episode return in each dataset. The results in Table demonstrate that while MAT+ICQ a strong baseline, Oryx still ourperforms it across all metrics and environments, validating our design decision to build on the Sable network architecture instead using MAT.
5 Related Work
Finally, we provide an overview of prior literature addressing the key challenges of offline MARL and works that approach MARL as a sequence modeling problem, highlighting connections and distinctions relative to our contributions.
Offline MARL. Early works such as Jiang and Lu (2021) and Yang et al. (2021) introduced methods to mitigate extrapolation errors through constrained Q-value estimation within the training distribution. Pan et al. (2022) combined first-order gradient methods with zeroth-order optimisation to guide policies toward high-value actions, with further refinements provided by Shao et al. (2023), which applied conservative regularisation using per-agent counterfactual reasoning, and Wang et al. (2023), employing global-to-local value decomposition approach. Distributional constraints have been another promising direction, notably in works like Matsunaga et al. (2023) and Bui et al. (2025). Numerous additional works have also tackled extrapolation error, coordination issues, and offline stability (Zhang et al., 2023a; Wu et al., 2023; Eldeeb et al., 2024; Liu et al., 2024b; Barde et al., 2024; Liu et al., 2025; Zhou et al., 2025). Our approach builds directly upon these insights, specifically extending the early ICQ framework ( (Yang et al., 2021) to enhance learning stability from offline trajectories.
Moreover, theoretical advancements have enhanced understanding of the guarantees and limitations inherent to offline MARL (Cui and Du, 2022b,a; Zhong et al., 2022: Zhang et al., 2023b; Xiong et al., 2023; Wu et al., 2023). Complementary studies investigated opponent modeling in offline contexts (Jing et al., 2024), and explorations into offline-to-online transitions (Zhong et al., 2024a; Formanek et al., 2023b) reveal potential pathways to bridge static offline training with dynamic online adaptation.
MARL as a Sequence Modeling Problem. Capturing long-term dependencies and joint-agent behaviour is critical, particularly in partially observable scenarios and tasks with extended horizons. Prior works such as Meng et al. (2022) and Tseng et al. (2022) addressed these challenges through Transformer-ben based architectures, effectively modeling trajectory data in offline MARL settings.

---

Attention- irentioo-bbssed and diffusion- based strategies have also been explored (Zhu et al., 2024; Qiao et al., 2025; Li al., 2025; Fu al., 2025). Online MARL research further contributed with influential works like by Wen et al. (2022), which introduced transformer-base autoregressive action selection mechanism, alongside newer architectures aimed at better scalability with number of agents (Daniel et al., 2025; Mahjoub et al., 2025).
6 Conclusion
In this work, we introduced Oryx, a novel algorithm explicitly designed to address the critical challenges of coordination amongst many agents in offline MARL. We derived sequential policy updating scheme that leverages implicit constraint Q-learning (Yang et al., 2021) and the advantage decomposition theorem (Kuba et al., 2021). By integrating this sequential ICQ-based updating scheme with modified version of the Sable network (Mahjoub et al., 2025), Oryx effectively mitigates two fundamental problems in offline MARL: extrapolation error and miscoordination. Our extensive SMAC, RWARE, Multi-Agent MuJoCo, that Oryx consistently achieves state-of-the -he-trr-rttt results. Notably, Oryx excels in scenarios characterised by high agent densities and complex coordination requirements, distinguishing from prior approaches for offline multi-agent learning. To help accelerate research in this direction, we make all our datasets on Connector, with to 50 agents, openly accessible to the community.
empirical evaluation across diverse benchmarks-inchute and
Limitations and future work While Oryx demonstrates robust performance across diverse research benchmarks, its evaluation is naturally limited compared to the true complexity of large-scale, real- world industrial settings. Consequently, important future work includes extending Oryx to hybrid offline-onlie settings and evaluating effectiveness broader, more varied domains, particularly real-world applications. Furthermore, our work introduces autoregressive policies offline MARL, demonstrating the significant promise of utlising such policies in other architectures and setups. Future research could extend existing and novel offline MARL methods utilise autoregressive policies (Fu et al., 2022) and sequence models. Integrating diverse offline learning techniques into this paradigm may lead to the discovery of even more effective sequence models for multi-agent coordination.
Acknowledgments
We would like thank Louise Beyers for her valuable contributions the outset of this project, which were crucial for its inception. We thank our MLOps team for developing our model training and experiment orchestration platform Alchor. We thank the Python and JAX communities for developing tools that made this research possible. We thank the anonymous reviewers for their constructive feedback and valuable suggestions. Finally, we thank Google's ' TPU Research Cloud (TRC) for supporting our research with Cloud TPUs.
10

---

References
R. Agarwal, M. Schwarzer,
Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural information processing systems, 34, 2021. URL https::/ wwwpooeeeeednnnnnnneii..e.. www.ips..u//eeeeeeeeeeee ptpepppprrfleeppppre//////// /001/ file/f514cec81chh 1cccccc6555cccf77774666e555
P. Barde, J. Foerster, D. Nowrouzezahrai, and A. Zhang. A model-based solution to the offline multi-agent reinforcement learning coordination problem. International Conference on Au- tonomous Agents and Multiagent Systems, 2024. URL https: httppss///aaaas....// csc, liv.ac.. uk/
Proceedings aamas2024 (es222024dddf///p/ ppfs/pp41.ppl
C. Bonnet, D. Luo, D. Byme, Surana, Abramowitz, P. Duckworth, V. Coyette, L. I. Midgley, E. Tegegn, T. Kalloniatis, O. Mahjoub, M. Macfarlane, A. P. Smit, N. Grinsztajn, R. Boige, C. N. Waters, M. A. Mimouni, U. A. M. Sob, R. de Kock, S. Singh, D. purelos-Blance V. Le, A. Pretorius, and A. Laterre. Jumanji: diverse suite of scalable reinforcement learning environments in jax. In International Conference on Learning Representations, 2024. URL https htte::///pppeeee http///ppeeeeeeeeeeeeeeeeeeeeeee net/ pdf (eedddiAdcdpppnnn
T. V. Bui, T. H. Nguyen, and T. Mai. Comadice: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. In International Conference on Learning Representations, 2025, URL
D. Cornelisse, A. Pandya, K. Joseph, J. Suárez, and E. Vinitsky. Building reliable sim driving agents by scaling self self-play, 2025. URL https://arxiv..o hergg//////00..../ 14706.
Q. Cui and S. S. Du. When are offline two-player zero-sum markov games solv- able? In Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, ed- itors, Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022a. URL 777488884544445771116662222-8823 Conferencc pdf.
wwweins...pp.... cc/ ccppaper_.... fomerrffiless wwl_fffesspppppee http//pppp/////////////
Q Cui and S. S. Du. Provably efficient offline multi-agent reinforcement learning via strategy- wise bonus. In Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022b. URL https: htttp::/pproceeding.eeeee neurips. cc wwwppcppppppppeeeeteeeeeeeeee faapprr_flees www.fflllppppeee filles/paperr/022/fiis 0cc55400BTb110ccff10000000e33833 Pawpre/aaaa-PPape--oooeeeeeeeeee pdf.
J. Daniel, John de Kock, I.. Ben Nessir, S. Abramowitz, O. Mahjoub, W. Khlifi, J. C. Formanek, and A. Pretorius. Multi-agent reinforcement learning with selective state-space models. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, AAMAS '25, Richland, SC, 2025. International Foundation for Autonomous Agents and Multiagent Systems. URL https: /dl..... acm. org/ doi/ a00/10...../// 5555/370934/ 00145//3709377.74399999 3743910.
E. Eldeeb, H. Sifaou, O. Simeone, M. Shehab, and H. Alves. Conservative and awsk-aware offline multi- agent reinforcement learning. IEEE Transactions on Cognitive Communications and Networking, page 1-1, 2024. ISSN 2372-2045. URL http: / dx doi. org/ prrg/10 /00. 1109/ /CCON 2024. 3499357.
B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. N. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. 2023, URL
org/ abs 2212 07489.
J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the Thirty- AAAI Conference Artificial Intelligence. AAAI Press, 2018. URL https: : // /ojs s. aaai..rg.aa..................... org/ index. phpppppp///aa wptp/AAAI/articaaaapppppppo aarticlleiiiiii 11794.
C. Formanck, A. Jeewa, J. Shock, and A. Pretorius. Off-the-gue -rid marl: Datasets and baselines for offline multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, Richland, SC, 2023a. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450394321. URL https://wwy, afaamas... org. org/Proceedings/ aawas
3ttpaa/aaoaaaooee//////a2322333 httpa//a//aaaaa2/22ddd//ss////// http:///////ppp//////2/ pdf....d....
C. Formanek, Beyers, C. Tilbury, J. P. Shock, and A. Pretorius. Putting data the centre of offline multi-agent reinforcement learning, 2024a. URL https / aarxivii org/ abs/2409. 12001.
11

---

C. Formanek, C. R. Tilbury, L. Beyers, J. Shock, and A. Pretorius. Dispelling the mirage of progress in offline marl through standardised baselines and evaluation. In A. Glober- son, I. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, edi- tors, Advances in Neural Information Processing Systems, volume 37. Curran Associates, Inc., 2024b. URL httph: btstts::/prooe https::/ neurips cc/ wwwpapprr_eeeeeeeeeeeeee lf fiw.f/lllaaaeeeee httpp/pppp/22////111 Kc521TcBBcff07773110002266666843
-atee---aar----aaateeeee aeaataasttt_and_Bencaaaaaaa wwweeeeennnneeeeeee Track, pafaa
C. Formanek, C. R. Tilbury, Shock, K. ab Tessera, and Pretorius. Reduce, reuse, recycle: Selective reincamattin in multi-agent reinforcement learning. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023b. URL https://openrrriew..nt wtwpiiewwwee/foommmm// _Nx21t2qQfV.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020. URL https///://aps httt::::/aap.eemanniccccooo semanticscholar.
:D 215827910.
W. Fu, C. Yu, Z. Xu, J. Yang, and Y. Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research. PMLR, 17-23 Jul 2022. URL https:/ /ttppe///prcceeingg.. mlr. wal..ppeessvvvie v162/ wttpp///....ee/eeeeeeeeee (tto2//df22dd.pdde
Y. Fu, Y. Zhu, J. Zhao, J. Chai, and D. Zhao. INS: Interaction-aware synthesis to enhance offline multi-agent reinforcement learning. In The Thirteenth International Conference Learning Representations, 2025. URL https: http///ppppeeeeeeeeeeee/ee net/ forum? 2rrummdd=kxx2211P440..
R. Gorsane, O. Mahjoub, R. de Kock, Dubb, S. Singh, and A. Pretorius. Towards a standardised performance evaluation protocol for cooperative marl, 2022. URL htt.:: //axxiv../// org/ abs/ 2209. 10485.
Holder, N. Jaques, and M. Mesbahi. Multi agent reinforcement learning for sequential satellite assignment problems. Proceedings of the AAAI Conference on Artificial Intelligence, 39(25): 66516-26522 Apr. 2025, doi: 10. 10.1669/aai.vv992..438888 URL https://oja aaa./ai..saaaaa..a..aaaaaaaaa... org/ index andex.php/Aee
J. Hu, S. Wang, S. Jiang, and M. Wang. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning. In The Second Blogpost Track at ICLR 2023, 2023. URL https: /ttp:////pe/eiieee///e////e aet//forumm1dd-VB00Wbbbss
J. Jiang and Z. Lu. Offline decentralized multi-agent reinforcement learning, 2021. URL https: //arxiv. org/ atbs/2108 01832.
Y. Jing, K. Li, B. Liu, Y. Zang. H. Fu, Q. FU, J. Xing, and J. Cheng. Towards offline opponent model- ing with in-context learning. The Twelfth International Conference on Learning Representations, 2024. URL https: /eattps://oppe http///pppeeeeeeeeeeeeeee/eeeeee www.eeettuuune i iididdd2liiii dadSSwngtthigg.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains, Artificial intelligence, 1998. URL https: / http://wwwwwcccceeec scweecedreee.................... com/ science/
A. Krnjaic, R. D. Steleac, J. D. Thomas, G. Papoudakis, L. Schäfer, A. W. Keung To, K.-H. Lao, M. Cubuktepe, M. Haley, P. Börsting, and S. V. Albrecht. Scalable multi-agent reinforcement learning for warehouse logistics with robotic and human co-workers. In International Confer- ence on Intelligent Robots and Systems (IROS), 2024. URL https://ieeexplore ieee org/ document/ 10802813.
J. G. Kuba, M. Wen, L. Meng, H. Zhang, D. Mguni. J. Wang, Y. Yang, et al. Settling the variance of multi-agent policy gradients. Advances in Neural Information Processing Sys- tems, 34, 2021. URL https://prooeeddnggs http:/eeeededn..nennnneeeeeeeeee neurips. hww.ips.cc.pappr..ti..... files/ /wwes//appee///2222 ftle/ffeccaeeerr/SSssecccacaceoe
.df.
J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, 2022a. URL http:::////eerreee...... wwwp://www.enndttiddddeeeeea /ttp:p/dff7---EGGFaTTxx..
12

---

J. G. Kuba, X. Feng, S. Ding, H. Dong, J. Wang, and Y. Yang. Heterogeneous agent mirror learning: A continuum of solutions cooperative marl, arXiv preprint arXiv:2208.0168. 2022b. URL atttp://arxii.. http:/axxivv....aaaaa. http//bb//222/// 01682.
C. Li, Z. Deng. C. Lin, W. Chen, Y. Fu, W. Liu, C. Wen, C. Wang, and S. Shen. Dof: A diffu- sion factorization framework for offline multi-agent reinforcement learning. The Thirteenth International Conference on Learning Representations, 2025. URL https: https:///ppnneeee //wpp//eeep/eeeieeeee/me net/ Korum?id=0TFKVest
Liu, Y. Zhong, S. Hu, H. Fu, FUFUUX X. Chang, and Y. Yang. Maximum entropy heterogeneoou--ggen reinforcement learning. In The Twelfth International Conference Learning Representations, 2024a. URL
net
S. Liu, Y. Shu, C. Guo, and B. Yang. Learning generalizable skills from offline multi-task data for multi-agent cooperation, 2025. URL htttps://arxiv.oot wtg./bs/////...o../...// 21200.
Z. Liu, Q. Lin, C. Yu, X. Wu, Y. Liang, D. Li, and X. Ding. Offiie multi-agent reinforcement learning via in-sample sequential policy optimization, 2024b. URL https: //axxii org/an abs/ 07639.
O. Mahjoub, S. Abramowitz, R. De Kock, W. Khlifi, S. V. D. Toit, J. Daniel, L.. B. Nessir, L.. Beyers, J. C. Formanek, I.. Clark, and A. Pretorius. Sable: a performant, efficient and scalable sequence model for MARI.. In A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research. PMLR, 2025. URL https: httpas//pprcceeeee http//pooceeeddi/gee/e mlr vrees/v267/ j oub25a.
D. E. Matsunaga, J. Lee, J. Yoon, S. Leonardos, P. Abbeel, and K.-E. Kim. Alberdice: Addressing out outoof-distribution joint actions in offline multi-agent rl via alternating stationary distribution correction estimation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 72648-72677 Cur- ran Associates, Inc., 2023. URL https: / h/tpoc/ee////ooeeeeeeeeee neurips cc/paper fil.ffees//leellll files/paper/ Cowferenceeeeeccccccommoe pdf.
L. Meng, M. Wen, Yang, C. Le, X. Li, Zhang. Y. Wen, Zhang, Wang, and B. Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks, 2022. URL https: //ap::::///xiiii /aaxivxiiivv./.a//aaaa/aaaaaa.aa org/ /b///////// /2122 02845.
I. Osband, Y. Doron, M. Hessel, Aslanides, E. Sezener, A. Saraiva, K. McKinney, T. Lattimore, C. Szepesvari, S. Singh, B. V. Roy, R. Sutton, D. Silver, and H. V. Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreviee net/ met/foounti forum?id=rygf-kat kSYwH.
L. Pan, L. Huang, T. Ma, and H. Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In K. Chaudhuri, S. Jegelka, L.. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research. PMLR, 2022. URL /ttp///eercc.....e/eee/p mlr. press/v162 http///222//p/2eee////// html.
G. Papoudakis, F. Christianos, I.. Schäfer, and S. V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: Sn5n.
l/openreview wwtt/oorrmiidcctteeeeeee
B. Peng. T. Rashid, C. Schroeder de Witt, P.-.. Kamienny. P. Torr, W. Boehmer, and S. Whiteson. Facmac: Factored multi-agent centralised policy gradients. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 12208-12221. Curran Associates, Inc., 2021. URL https: http:/::/pocceednneeee wewpiiinnnnnnnuuiiepeeee.e. wwtppppp..pppp....pe files/ patps/pppp//p//////. 6519eea6e1 ieglaaafeccbbbrroaaaa///aaa. Paaerr.appprp.re
D. Qiao, W. Li, Yang, Zha, and B. Wang. Offline multi-agent reinforcement learning with se- quential score decomposition, 2025. URL
13

---

A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. a. Ingvarsson, T. Willi, R. Hammond, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan. M. Jiang, R. Lange, S. Whiteson, B. Lacerda, N. Hawes, T. Rocktäschel, C. Lu, and Foerster. Jaxmarl: Multi-agent rl environments and algorithms in jax. In Advances in Neural Information Processing Systems, volume 37. Curran Associates, Inc., 2024. URL pttppe://procceedingg. nwwrine..nncmmm httpc//pppppe.ieeeppaape////e/// 2aee122f6022200222666ffffffffff2
Paper wwwp:e/ewweeeeeeeeaaaattttttetee Bew.reetss.ddddeeeennnnrereeesee Track, pdf.
M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. J. Rudner, C.-M. Hung. P. H. S. Torr, J. Foerster, and Whiteson. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Richland, SC, 2019, URL https: www. afaamas. org/ Ptt.peeeoceeeee///ee/eeeeeccceee attpeedddnnnaammaa2011119 aataaa01aaaaaaaaaaaa/ / p2186. pdf.
S. Schneider, A. Ramesh, A. Roets, C. Stirbu, F. Safaei, F. Ghriss, J. Wülfing, M. Güra, N. Sibon, R. Gentry, et al. Intelligent railway capacity and traffic management using multi-agent deep reinforcement learning. In 2024 IEEE 27th International Conference Intelligent Transportation Systems (ITSC). IEEE, 2024. URL httph: 10919999.
ieee org/ att////ttttttoooooiaa
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. URL https: //tr:v////iiiii
111110007700.777 06347.
Shao, Y. Qu, C. Chen, H. Zhang, and X. Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances Neural Information Processing Systems, volume 36, pages 77290-77312. Curran Associates, Inc., 2023. URL https:
neurips. wwtpppppppeaaaeeeeeeeee fhaaprr_flee_ httll///ppper222//E122 Paper- Conference, pdf.
Y. Sun, L. Dong. S. Huang, S. Ma, Y. Xia, J. X.e. Wang, and F. Wei. Retentive network: A successor to transformer for large language models, 2023. URL https: //tp:v//a.ivvv..a abs/2307 08621.
C. Tilbury, J. C. Formanek, L. Beyers, P. Shock, and Pretorius. Coordination failure in cooper- ative offline MARL. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024. URL https: //tpp:///////eeeeeeeeeee net/ wwt.:eeet/oouue /ttr/
W.-C. Tseng, T.-H. Wang, Y.-C. Lin, and P. Isola. Offline multi-agent reinforcement learning with knowledge distillation. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. URL htt.ttt:://opereeaieeee wwtpevviwwiieeeeonnniiiii yipUuqxveCy.
X. Wang, H. Xu, Y. Zheng, and X. Zhan. Offline multi-agent reinforcement learn- ing with implicit global-to-local value regularization. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 52413-52429. Curran Associates, Inc., 2023. URL wttp:////oooooo........e 000c647766a42494bbbbee66666 -Paparrr----ppa-eeeaae Conferencee..eeeee pdf..
cc/paper www.s/illeeeeeeee /ppp///////////ilee/
M. Wen, J. Kuba, R. Lin, W. Zhang, Y. Wen, J. Wang, and Y. Yang. Multi- agent reinforcement learning is sequence modeling problem. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh. editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 16509-16521. Curran Associates, ittpeeeeddinnneneuiisseeee neurips.
Inc., 2022. URL https: http///proeeenn.eeeeee///e aat.aa499770011100009997700000 Paper Conferr-coeeeeeeeee pdf.
C. Wu, P. Tang, J. Yang, Y. Hu, T. L.v, C. Fan, and C. Zhang. Conservative offline policy adaptation in multi-agent games. shirth-seventh Conference on Neural Information Processing Systems, 2023. URL
h:tp:://ppeeeiiiee net/forum? iatprrui@@dcCppLL@@0tt
W. Xiong, H. Zhong, C. Shi, C. Shen, L. Wang, and T. Zhang. Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game, 2023. URL https: // /arxiv. org/ abs /abs//205..02 15512.
14

---

Y. Yang, X. Ma, C. Li, Z. Zheng, Q. Zhang. G. Huang. J. Yang, and Q. Zhao. Be- lieve what you see: Implicit constraint approach for offline multi-agent reinforcement learn- ing. In Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021. URL https::/proceeedeeeeeeee 100114122e63411a5500003333333333
C. Yu, A. Velu, E. Vinitsky, Gao, Y. Wang, A. Bayen, and Y. WU. The surprising effectiveness of ppo in cooperative multi-agent games. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35. Curran As- sociates, Inc., 2022. URL https: http:///////ceeeeeeeeeeeee neurips www..ps.ccppppppeeeeee cc/paper httppppparrpllalllllrlllllleeeee ftles//aperr/022/ 0aap:/25555222077433334444111112
and www.eeenncshepeeeeeeeeee pdf.
F. Zhang, C. Jia, Y.CC. Li, L. Yuan, Y. Yu, and Z. Zhang. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh International Conference on Learning Representations, 2023a. URL https: //wprr/viie/eeeeeee///// htt.://w...nttooroorrooaeaaaaaae
Zhang, Bai, and N. Jiang. Offline learning markov games with general function approximation, 2023b. URL https: aarxiv org/ abs/ /202./232..2222 2302.02571.
H. Zhong, W. Xiong, J. Tan, L. Wang, T. Zhang, Z. Wang, and Z. Yang. Pessimistic minimax value iteration: Provably efficient equilibrium learning from offline datasets, 2022. URL https: // //arxiv. org/ abaaab//2202 2202. 220200755 07511.
H. Zhong, X. Wang, Z. Li, and L. Huang. Offline-to-online multi-agent reinforcement learning with offline value function memory and sequential exploration, 2024a. URL https://arxiv org/ abs/2410. 19450.
Y. Zhong, J. G. Kuba, X. Feng, Hu, J. Ji, and Y. Yang. Heterogeneouu--agent reinforcement learning. Journal of Machine Learning Research, 2024b. URL http://jmlr... warrg/apprs/v25.. 23- 13-0488.00 html.
Y. Zhou, Y. Zheng, Y. Hu, K. Chen, T. Zheng, Song, M. Song, and S. Liu. Cooperative policy agreement: Learning diverse policy for offline marl. Proceedings of the AAAI Conference Artificial Intelligence, 39(11):30188
Apr. 2025, doi: 10. 16t0///aa..3392....... https: //op::///js aaai. org/ index. http://A//aaticce///ee//////////
URL
7. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang. Madiff: Offline multi-agent learning with diffusion models. In Advances Neural Information Processing Systems, volume 37. Curran Associates, Inc., 2024. URL https: / p/tpoeee////oooeeinggg..g.ms neurips htppp/aperrrrra// files/
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The results of an extensive benchmark are reported in Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Included in the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification: A derivation of our policy update is included in the main text, and the detailed proof is in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the code and datasets used are provided. Training details and hyperparame- ters are included in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All the code and datasets are available and can be run.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss cttpssccccpbccccccceececcccccccc /ttp:/eeeeeeeoeeemiiieeeee /Codeeeomiissseeeeee cessionnoocyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All training and test details are carefully outlined the Appendix. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Careful consideration was taken to test statistical significance. Details included in the main text and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details on compute resources are carefully outlined in the Apppendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /nnurrirs.... cc http:/pulc/tttisssuiit
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Included in the Conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The datasets contribute well documented and made publicly available.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21

---

A Additional Algorithm Details
A.1 Proof of Theorem
Theorem For an auto-regressive model, the multi-agent joint-policy under ICQ regularisation can be optimised for j = I,.., n follows:
(A (T, a'1o)
THe arginax IE, 1113 wB
Zu: log( ( a))
exp
x²1
where ( = IT--1 Ear | |T a exp
(Atu((aaa))) is the normalisation function.
Proof. For an regreseegeee model, we can decompose the joint policy as
(aaT) = ~ (a 1:m (T)
II
(T 1)
j=1
Furthermore, from the advantage decomposition theorem by Kuba et al. (2021), we have that
A(T,a) A (T, aft))
A (T,
j=1
Let In denote the joint-policq loss as Yang et al. (2021). Using the above decompositions we can re-wir ee-ir the loss follows
1
A(T,a)
Jx Br
log(T(alTT) exp
((T
1
n
En ||=| A (T,
Tog(T>b(())
(a's
T, a 1))
exp
((T)
a
Eis A (
E B
(log( ( (a aTaa exp
Z(T)
1
El (T
E 11 BB
log(T' (T,a exp
Z(T))
a
j=1
Lk=j+1 A (T, Li 3.a%))
exp
log(T>>(a*) (a's
T exp
BB
7in:1 (T)
1=1
1
Ekmj+1 A (T, (
Bra
7ts+
exp
a
( (T, aداد)
log(T
/ a exp
11 BB
where the normalising partition function is given
A' AT, altim)
Z(T)
II 11 7m
Σ
p tm (ā im |T a exp
a
and for any specific j = 1..... n can be factorised as
A
A (T, at::
A (Ahit(T, $1:k
((T) 11 d (a T, 21 )xx exp
II
Σ th (a*t (T a exp
a
a
1=1 64
k=j+1 5t
- (T) Z (T)
22

---

A.2 About Sable
Sable employs Retention-basse architecture designed to efficiently model temporal dependencies amongst many agents. The model operates in two distinct modes depending on the phase: recurrent execution and chunkwise training.
Execution Phase. During interaction with the environment, Sable runs in recurrent mode, main- taining a retention state that evolves over time. This mode supports memory- and compute-effin efficient inference, as it scales linearly with the number of agents and is constant with respect to time.
Training Phase. For gradient-basee based optimisation, Sable leverages chunkwise mode, parallel variant of Retention that processes fixed-length temporal chunks. Unlike fully parallel mechanisms, the chunkwise representation preserves hidden state transitions across chunk boundaries. This design allows the model to propagate temporal signals through time respecting episode boundaries (resettable retention). The choice of chunkwise training over parallel mode ensures that the retention state {S,} can be passed between training chunks maintaining temporal coherence and improving convergence stability.
Oryx Adaptation. In our offine variant, Oryx, we adopt only the chunkwise training mode from Sable. Since training occurs entirely off-policy datasets (offline), we do not maintain a persistent recurrent retention state {S,} Instead, whenever we initiate the training phase, we begin with a None hidden state, ensuring no temporal leakage from prior training phases.
As mentioned in the main text, adaptation of Sable to Oryx includes additional structural modifications to better support the ICQ loss. Specifically, we remove the value head from the encoder and use it solely to produce observation representations. The decoder, in turn, is extended to output both the action distribution and Q-values, enabling compatibility with the ICQ training objective.
23

---

B T-MAZE
T-Maze step
(b) T-Maze middle step
(c) T-Maze last step
Figure 5: Environment visualisation for Connector.
B.1 Environment Details
The T-Maze environment is intentionally designed minimalist setting that isolates key challenges for multi-agent reinforcement learning: interdependent action selection, reliance on memory from previous timesteps, and effective coordination achieve a common goal. The environment unfolds in two distinct phases.
Phase 1: Initial Target Color Selection
This phase spans a single timestep at the beginning of each episode and is visualized in Figure Figure (a). During this step:
Observation: Agents receive no specific environmental observation.
Action Space: Each agent must independently choose one of two actions: choose orange or choose green.
Coordination Requirement: For the episode to be solvable, the two agents must select different target colors. If both agents choose the same color, they will be assigned the same goal location, making successful completion impossible. This necessitates a coordinated action selection strategy at the outset.
Phase 2: Navigation and Goal Achievement This phase encompasses all subsequent timesteps until episode terminates. It is visualised Figure Figure 5 (b) and (c)
Initial Setup (Start of Phase 2):
Agent Placement: The two agents are randomly assigned to one of two distinct starting squares located at the base of the T-maze stem.
Goal Assignment: The two target locations (at the ends the T-maze arms) are randomly assigned the colors green and orange, ensuring goal is green and the other is orange.
Observation Space (During Navigation): In each step of this phase, agents receive the following information:
3x3 local grid view, centered on the agent's current position, showing the maze structure and potentially the other agent if within this vicinity.
Information indicating which corridor contains the green target. From this, the location of the orange target in the opposite corridor can be inferred.
Their own action taken in the immediately preceding timestep.
Action Space (During Navigation): Agents can choose to move in any of the four cardinal directions (up, down, left, right)
take do nothing action.
Memory Requirement: The critical memory challenge arises from the observation structure. On the first timestep of Phase (the second timestep of the episode overall), an agent observes the outcome of its choose target action from Phase (i.c., it knows its chosen color

---

Replay
5000
Frequency
3000
2000
1000
0.0
0.5
Episode return
Figure 6: Distribution of episode returns of the recorded Replay Data for T-Maze.
and the corresponding goal location). For all subsequent timesteps in Phase 2, the agent only observes more recent movement actions as its "previous action." " Therefore, each agent must retain the memory of its initially chosen target color for the remainder of the episode to navigate correctly.
Dynamics and Rewards
Movement and Collision: If an agent attempts move into wall or into square occupied by the other agent, its position remains unchanged for that timestep.
Reward Structure: Agents receive sparse team reward. reward of +1 is given if both agents are simultancously positioned on their correctly colored target locations. In all other timesteps, and for all other outcomes (including collisions incorrect goal locations), the reward is O.
This design forces agents first coordinate on distinct objectives, then remember their individual objective over potentially long horizon while navigating a shared space and avoiding interference at junctions.
B.2
Dataset Generation
For the T-Maze experiment, we generate two types of offline datasets: replay and expert. Both collected from training Sable for 20 million timesteps. The replay dataset is recorded continuously during training by logging trajectories sampled throughout the execution phase. In contrast, the expert dataset is generated post-training by evaluating the final policy parameters for a fixed number of steps, producing trajectories that reflect near-expert behaviour.
The replay dataset contains over million transitions with a mean episode return of 0.559, reflecting mix of behaviours observed throughout training. While, the good dataset comprises 100,000 transitions collected by evaluating the final policy, with all episodes achieving a perfect return of 1.
B.3
Algorithms Details
B.3.1 Implementation Details
For baselines, we select MAICQ (Yang et al., 2021) and its fully independent variant (I-ICQ), in which we remove the QMixer component to enable decentralised training across agents.
Besides the baselines and Oryx, we test the effect of isolating Oryx's key components. The first ablation concerns disabling auto-regressive actions, in the original implementation, agent + 1 receives the actions of agents i, 1, 1 as an input, within the same timestep. to support sequential coordination. To disable this mechanism, we instead feed a constant placeholder value of as the previous action input to all agents during both training and evaluation. The second ablation targets removing temporal dependency, to achieve this, we reduce the sequence length from default value of 20 (used in Oryx) to 2, such that only timesteps and + are provided during training. This limited context prevents the model from capturing or leveraging long-term temporal dependencies, effectively disabling its ability to retain memory across steps. The third ablation is about removing
25

---

the ICQ loss component, specifically, we eliminate the advantage estimation and policy regression terms from the loss function, retaining only standard Q-learning.
B.3.2 Evaluation Details and Hyperparameters
Each offline system is trained for 100,000 gradient updates. Final performance is evaluated over 32 parallel episodes, with results aggregated across 10 independent training runs using different random seeds.
Table 2: Default hyperparameters for MAICQ
Table 3: Default hyperparameters for Oryx
and I-ICQ
and its ablation variants
Parameter
Value
Parameter
Value
Sample sequence length
20
Sample sequence length
20
Sample batch size
64
Sample batch size
64
Learning rate
3c-4
Learning rate
3c-4
ICQ Value temperature
1000
ICQ Value temperature
1000
ICQ Policy temperature
0.1
ICQ Policy temperature
0.1
Linear layer dimension
64
Model embedding dimension
64
Recurrent layer dimension
64
Number retention heads
Mixer embedding dimension
32
Number retention blocks
Mixer hypernetwork dimension
Retention heads k scaling
0.5
26

---

C Connector
C.1 Environment Details
5x5x3a
(b) 15x15x23a
(c) 25x25x50a
Figure 7: Environment visualisation for Connector.
The Connector environment (Bonnet al., 2024) cooperative multi-agent grid world where each agent is randomly assigned start and end position and must construct path to connect the two. As agents move, they leave behind impassable trails, introducing the need for coordination to avoid blocking others. The action space is discrete with five options: up, down, left, right, and no-op. Agents observe local d x view centered on their position, including visible trails, their own coordinates, and all target locations. The reward function assigns +1 when an agent successfully connects to target and -0.03 every other timestep, with no further reward once connected.
For scenario naming, we adopt the convention con- <x_ xx_size>xyy___aaa ser_ssszeennnue aauu_aggets>a, where each task defined by the grid dimensions and the number agents. In our experiments, we use the original Connector scenarios from the Sable paper (Mahjoub et al., 2025): 5x5x3a, 7x7x5a, 10x10x10aa and 15x15x23a. To evaluate scalability beyond 23 agents, we introduce three new sce- narios 18x18x30a, 22x22x40a, and 25x25x50a while approximately preserving agent density across scenarios.
C.2 Dataset Details
To generate the offline datasets, we train Sable using the hyperparameters reported in the original Sable paper for each Connector task; for scenarios with more than 30 agents, we reuse the parameters from the 23-agent setting. For all tasks involving 3 to 30 agents, we train Sable for 20M timesteps and record the full execution data, resulting in 20M samples per task. We then uniformly subsample 1M transitions using the tools introduced in Formanek et al. (2024a), which randomly shuffle the set of available episodes and iteratively select full episodes until the target number of transitions is reached. For the larger 40-agent and 50-agent scenarios, due memory constraints, directly record less than IM transitions during training and use them as-is without additional subsampling.
Table 4: Summary of Recorded Offline Datasets
Task
Samples
Mean Return
Max Return
Min Return
con-5x5x3a
1.17M
0.59
0.97
-0.75
con-7x
1.13M
0.48
0.97
-1.23
con-10x 10x
1.09M
0.40
0.97
-1.53
con-15x15x23a
1.06M
0.34
0,97
-1.56
con-18x18x30a
1.00M
0.25
0.97
-2.43
con-22x22x40a
624,640
0,4
0.97
-2.61
con-25x25x50a
624,640
0.33
0.97
-3.06
C,3 Extra Results on Default Connector
We evaluate Oryx against several well-established baselines on the Connector tasks with a 3, 5, 10, 23: (i) MAICQ, along with its decentralised variant I-ICQ, (iii Multi-Agent Decision
27

---

con-5x5x3a
con-7x7x5a
con-10x con-10x10x1oa
100000
80000
60000
Frequency
80000
Frequency
60000
Frequency
60000
40000
40000
20000
Episode return
Episode return
Episode return
con-15x15x40 1x23a
con-18x1 8x30a
con-22x22x40a
60000
Frequency
Frequency
40000
Frequency
6000
40000
30000
4000
20000
2000
-1
1
-2
-2
-1
1
Episode
Episode
con-25x25x50a
8000
Frequency
6000
0 Episode
-2
Figure 8: Distribution of episode returns of the recorded Replay Data for Connector.
Transformer (MADT) (Meng al., 2022), and (iii) IQL-CQL. Both MAICQ and IQL-CQL the implementations from OG-MARL (Formanek et al., 2024b). Based on their relative performance, select the most competitive baseline for the scaling experiments (n 30, 40, 50) reported in subsection 4.2.
- Oryx (Burs)
186-00
HDO
MAICO - MAST
Normalised
Number agents
Figure 9: Performance on the smaller Connector scenarios- While all methods perform similarly in low-agent settings, Oryx begins outperform the other baselines the number of agents increases. I-IQQ and MADT follow. However, despite our best efforts, we could not run MADT on the larger agent instances without running out of Memory on our compute infrastructure. As such, I-ICQ offered a better compute- perpurte-eeformmnce trade-off, making it the preferred baseline for scaling beyond 23 agents.
28

---

C.4 Evaluation Details and Hyperparameters
For evaluation, we train Oryx and MAICQ for 100k gradient updates across 10 seeds (reduced to 3 seeds for settings with 40 and 50 agents due to computational constraints). Final performance is reported based on evaluation over 320 episodes.
We conducted preliminary comparisons between I-ICQ and MAICQ and observed that I-ICQ consis- tently outperformed the centralised training system (see Figure 9).
All the Connector experiments were distributed across NVIDIA A100 GPUs (40GB and 80GB VRAM).
Table 5: Default hyperparameters for MAICQ and I-ICQ Parameter
Value
Sample sequence length
20
Sample batch size
64
Learning rate
3e-4
ICQ Value temperature
1000
ICQ Policy temperature
0.1
Linear layer dimension
128
Recurrent layer dimension
64
Mixer embedding dimension (MAICQ only)
32
Mixer hypernetwork dimension (MAICQ only)
64
Table 6:
Default hyperparameters for IQL--QII.
Parameter
Value
Sample sequence length
20
Sample batch size
64
Learning rate
3e-4
Linear layer dimension
128
Recurrent layer dimension
64
CQL weight
3.0
Table 7: Default hyperparameters for Oryx
Parameter
Value
Sample sequence length
20
Sample batch size
64
Learning rate
3e-4
ICQ Value temperature
1000
ICQ Policy temperature
0.1
Model embedding dimension
128
Number retention heads
Number retention blocks
1
Retention heads scaling parameter
0.5
For MADT, adopt the default hyperparameters from the official repository of the original paper. However, we evaluated two configurations: (i) the default MADT setup, and (ii) an enhanced variant that includes reward-to-go, state, and action in the centralised observation. We found the latter consistently outperformed the default, and therefore use it in the reported results.
29

---

D SMAX
Before trying scaling experiments on Connector, we first evaluated Oryx on SMAX, an adaptation of SMAC(v2)EElis et al., 2023) implemented in JAX via JaxMAALL(Rutherfrr et al., 2024), to assess its performance on a widely adopted benchmark MARL.
D.1 About SMAX
SMAX (Rutherford et al., 2024) is JAX-based reimplementation of SMAC (Samvelyan et al., 2019) and SMAC(v2) (Ellis al., 2023), designed for efficient experimentation without the need for the StarCraft II engine. In this environment, agents form teams of heterogeneous units and cooperate to win battles in real-time strategy setting. Each agent observes local information-ssuct as positions, health, unit types, and recent actions of nearby allies and enemies and selects from discrete action space including movement and attack commands. Unlike SMAC. SMAX balances reward signals between tactical engagements (damage dealt) and final success (winning the episode).
For the scaling experiments, focus SMAC(v2), evaluating performance across four sce- narios of increasing agent count: mmacv2__uuitac smacv2_ 10un_tt, smacv2 smacv2_20unitar and 2macv2_3Ounize
D.2 Scaling Agents in SMAX
COLCOL MADT
1.0
Hormalised Epizode
0.6
0.4
0.0
10
20
30
Number of agents
Figure 10: Performance on SMAC(v2) across varying agent populations (5 to 30 agents) While SMAC scenarios do not always become harder with scale-some higher-agent tasks may in fact be less coordination-intensive -Oryx remains robust and achieves superior results across the board.
D.3 Evaluation Details and Hyperparameters
For data collection, follow the same protocol used for Connector (3-30 agents) as described in Section C.2. The evaluation procedure and the chosen hyperparameters are also consistent with those applied in the Connector experiments.
30

---

E Tabulated Benchmarking Results
Here we provide the tabulated results from our benchmark on SMAC, RWARE and MAMuJoCo. The algorithm with the highest mean episode return is denoted in bold. Algorithm that are not significantly different the best, based on the two-sided t-test we conducted (Papoudakis et al., 2021; Formanek et al., 2024b), are denoted with an asterisk. All Oryx results are reported over 10 random seeds, with the mean and standard deviation given.
Table 8: Results (Win Rate) on SMAC datasets from OMIGA (Wang et al., 2023). Other algorithm results from ComaDICE (Bui al., 2025).
Sul Sn
ceriid
Algeeith
Good
Mediun
Poor
Good
Mellon
Good
Melion
Cood
Molian
BC
31.2 9.
19: 1.5
0.0 0.0
2.5 23
19:
255 1.3
88 1.2"
1
00:000 0.1
30.
250
000
BCQ
0.0"
1.:
44.4
25
1,2=15
15 1.5
0.=000"
1.
=4
10Q
38.7 4.6 2
1.0 1.5
0000000"
3.
12
13 1.5
20"
2
00=0.
425
225==31
0,6:213 1.3
OMAR
12-11
0:000000
11-11
06
86- 1.2
86+13
1.
00=000
11-000
110=23
00:+:00
OyDICE
37.5
Lbs
000000
000000
00:000
00:00P
33
119
00:4:00
AberiiCC
6. 6.4
16.2
on
3.21 14
0.000.
2.
45.1264
94.16
00: 00:1:00
ComaDICC
55.0 15
13: 70"
86 16:111
2.5.25
4..1422
112 544
31 21"
19
488.2
21.52-34
05.2 1.3"
Table 9: Results (Episode Return) on SMAC datasets from OG-MARL (Formanek et al., 2023a). Results from DoF (Li et al., 2025).
Migiitees
1.0 100 14.1 a-1.1
36
65
34
17..4.0
15-E55
142 --17
139-002
146-
115
118406
166-86
06-02
170+81-
155
115-92
14-331
Manur
19. mm
19-15
15-11
52-06
67406
12
Table 10: Results (Win Rate) on CFCQL (Shao et al., 2023) datasets. Other algorithm results also from CFCQL.
Aile
No. 1110
Roday
Expuc
Miod
N
1I4 Rocly
Exyur
Mincc
Mic. Roday
Expes
Vinu
Hil
Regioe
Digant
117 0.
D37
S1R 1110
211 E111
011
111 -EID
011
01201 $11
611
112 -0ST
319 -1:1
610
115-00
090-100
071-
115 -000
022-130
-11
415
0123 61h
13-4:11
810 8/3-1017
839 09--68
22
833 618
L12
211
6. 418
B22 122-00
bat 200
6.1 km
5.0 410
0kys
6.6 118
312-02)
6
121 4ear
(31-6:1
860 4U
Table 11: Results (Episode Return) on RWARE datasets from Alberdice Matsunaga et al. (2023). Other algorithm results are also from AlberDICE.
Tiny (IIIII)
Small (IIx20)
Algorithm
N-2
N-4
N-6
N-2
N-4
N-6
0.45
11.12 0.35
5.54
7.88
I0Q
9.38 +: 1.30
12.13
1: 0.76
14.59 1 0.28
5.43 2: 0.33
7.93 2: 0.33
8.87 0.38
OMAR
6.77 1 1.11
14.39 1 1.58
16.13 1 2.10"
4,10 1: 0.59
7.12
1: 0.66
8.41 1: 0.85
MADTKD
6.24 + 1.04
9.36
13.06
= 0.33
3.65
# 0.59
6.85
4: 0.62
7.85 + 0.00
OPiDICE
8.70
0.10
11.13
14,02 0.62
4,84 0.55
7.68
0.16
8.47 0.45
AlbaDDICE
11.15
13.11
0.62
5.97
0.19
Oryx
13.23 1 0.25
16.71 1 0.32
18.64 1: 0.47
6.95 1: 0.44
9.86 J: 0.32
10.13 1: 0.41
Table 12: Results (Normalised Episode Return) MAMuloCo datasets from OMAR (Pan et al., 2022). Other algorithm results from CFCQL (Shao et al., 2023).
HaltChettah
Algorithm
Random
Medium-Rnne Replay
Medium
Esper
ICQ
7:40 2 0.00
35.60 : 2.70
75.60 0: 5.00
110.60 = 3.30
TD34BC
7.40 0.00
27.30 5.500
75.50 3.70
114,40 3.50
ICQL
7.40 0.00
41.20 10.10
$0,40 2: 10.50
24.500
OMAR
13.50 + 7.00
57.50 + 5:10
90.40 + 10.20
(.335
MADQI
5.90++0.50
33.00 + 7.10
$1.50
26.70
20:00+220.10
IOL.
7.40 2:000:0:00
58.50 = 6.50
81
3.70
115.60
= 4.20
AWAC
7.30
30,00 = 1.60
71.20 4.20
113.50 = 4.10
CRCOL
39,70 2: 4.00
59.50 : 8.20
80.50 4: 9.60
118.50 = 4.90
Oryx
7.90
91.70 à. 11.00
93.80 :: 10.90
128,70 8.40
31

---

Table 13: Results (Episode Return) on MAMuJoCo datasets from OMIGA Wang al. (2023). Other algorithm results from ComaDICE (Bui et al., 2025).
Sel Hegper
Algrefia
Netizn-Sappte
Mediam Eupeet
Alguriitea
Eapen
Makkar
Nidiee- Ruploy
Mucilan- Eogeet
BCQ
27.50 39.00
44.00 =: 20.4
26.50 = 34.00
54,30 23.70
1317 1117-70
230-31
2056.00
91.30
653.0-1 10.90
3230.0
342.70
(00
$33.00.
1766-07
373.00
OMBIAA
899 60
1100
171.30 191.30 937
700.0 995.70
ONEGA
3005-93
146
1038-0044 5.90
1106 10 30.90
1134 32 113:60
120.10
AlteDIFT
801.00
311-40 15:36
1119 30 341.50
515 301.00
AlboRICE
1990,90
23.30
1204300::2200
1062-90 93.30
3190.00 23,07"
OleaDICC
3827.100 82.96
$22.50 06.25
506.30 242.10
1392.40
522
206.99
5:3.0"
22220
1122/00::91000 01,00
18:3.00
1299
927.08
1223.00
1490
2068.0 $:900
1436.00 7.89
121340-2 115.20
1963.00 118.0
Apurtikas
Dypen
Mollan
Modirn-logen
RFQ
100 70 060
5981 91 1110
511 (0- 153.0
3503-00 700-57
I0Q
205500
439.31
2341.30 96.30
3922-40
422.30
OpeOCC
1668
7404 10-4-335.
7106,0
AborDoc
3306-41
S60.0
122.42 u2 313.0
-40001 536/0
2388 32 733-00
Days
(T6/21 161.01
301.0
3931.18
E.1 Hyperparameters
SMAC. All Hyperparameter settings for Oryx were kept fixed across SMAC datasets except for the ICQ policy temperature, which we found was sensitive to the mean episode return datasets. Thus, we used policy temperature that ranged between 0.9 and 0.1, depending the quality (mean episode return) of the respective datasets. For OG-MARL and OMIGA datasets, the policy temperature was as follows: Good 0.9, Medium 0.3 and Poor 0.1. For CFCQL datasets, the policy temperature was set as follows: Medium -> 0.3, Med. Replay -> 0.1, Expert -> 0.9 and Mixed -> 0.8.
RWARE. All Hyperparameter settings for Oryx were kept fixed across RWARE datasets.
MAMuJoCo. All Hyperparameter settings for Oryx were kept fixed across MAMuloCo datasets.
Table 14: Hyperparameters for Oryx on
Table 15: Hyperparameters for Oryx RWARE scenarios.
SMAC scenarios.
Parameter
Value
Parameter
Value
Sample sequence length
20
Sample sequence length
20
Sample batch size
64
Sample batch size
64
Learning rate
3e-4
Learning rate
3e-4
ICQ Value temperature
1000
ICQ Value temperature
10000
ICQ Policy temperature
E (0.1, 0.9)
ICQ Policy temperature
0.1
Model embedding dimension
64
Model embedding dimension
64
Number retention heads
1
Number retention heads
I
Number retention blocks
1
Number retention blocks
Retention heads scaling parameter
0.9
Retention heads scaling parameter
0.8
Table 16: Hyperparameters for Oryx on MAMu-
JoCo scenarios.
Parameter
Value
Sample sequence length
20
Sample batch size
64
Learning rate
3e-4
ICQ Value temperature
100000
ICQ Policy temperature
10
Model embedding dimension
64
Number retention heads
1
Number retention blocks
3
Retention heads k scaling parameter
0.9
32


Reviews:
[{"review_id": "Soc19im9KH", "reviewer": "Reviewer_MxbN", "summary": "This paper introduces Oryx, a novel algorithm designed for offline multi-agent reinforcement learning (MARL), targeting the key challenges of extrapolation error and miscoordination that arise when learning from fixed datasets in multi-agent environments. Oryx builds on Implicit Constraint Q-learning (ICQ) and incorporates the Advantage Decomposition Theorem to design a sequential policy updating scheme, where agents update their policies one at a time, holding others fixed. The method also employs a modified version of the Sable network architecture to better handle the joint action-value function and improve credit assignments. \n\nEmpirical evaluations are conducted on diverse and challenging benchmarks, including SMAC, RWARE, Multi-Agent MuJoCo. The results demonstrate that Oryx achieves state-of-the-art performance, particularly in settings with high agent density and complex coordination demands. The authors also released new large-scale offline MARL datasets to support future research.", "questions": "1. While sequential updates may help coordination, do they scale well in environments with hundreds of agents or continuous control tasks? Could you discuss whether this design limits parallelization or increases training time significantly? I believe it is worth mentioning and discussing the trade-offs and limitations.  \n\n2. In the sequential update scheme, how sensitive is the overall policy quality to the order in which agents are updated? Have you tried optimizing the order, and does it affect stability or final performance? \n\n3. In sequential updates, each agent optimizes assuming the policies of the other agents are fixed. Does this introduce error accumulation across the agent chain? Is there any correction mechanism or re-evaluation strategy to mitigate this? \n\n4. In your implementation of Implicit Constraint Q-learning for each agent, how is the constraint enforced during policy updates? Are constraints handled through soft penalties or something else? Additional clarity would help.", "limitations": "No. While the authors mention some future work, they do not explicitly discuss the limitations or trade-offs of the proposed method. Specifically, the paper would benefit from a discussion of the scalability bottlenecks or other trade-offs introduced by sequential policy updates.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths: \n \n + The proposed method combines Implicit Constraint Q-learning and the Advantage Decomposition Theorem to address critical challenges in offline MARL, such as extrapolation error and agent miscoordination. The use of sequential policy updates is well-motivated. \n\n + Oryx demonstrates state-of-the-art results across multiple offline MARL benchmarks, including high-agent-density tasks like Connector and SMAC, showcasing the method’s scalability and effectiveness. \n\n + The release of open-source implementations and offline datasets in complex multi-agent environments provides a valuable resource for the community. \n\n## Weaknesses: \n\n - The paper lacks ablation studies to isolate the contribution of each component (e.g., ICQ, Sable architecture, sequential updates). It is unclear which aspects of the method drive performance gains. \n\n - While the sequential update strategy improves coordination, it may become computationally expensive or impractical in very large-scale or real-time settings. The paper does not fully address the trade-offs or potential bottlenecks.", "was_revised": true, "final_justification": "The authors have provided thoughtful responses to my concerns. They clarified the scalability of the sequential update scheme and explained how constraints are handled in ICQ. They also pointed out their T-Maze ablation study, which isolates the role of each component. While the primary contribution is a well-engineered integration of known ideas, the empirical performance and release of datasets make it a good addition to the offline MARL literature.", "rebuttal": "We thank the reviewer for their positive review of our work and their constructive questions. By addressing them, we feel we can substantially improve the clarity of our paper's message. Here we address each of your questions in turn:\n\n* *While sequential updates may help coordination, do they scale well in environments with hundreds of agents or continuous control tasks?*\n    - Similar to our response to the reviewer vYPS, since Oryx uses a modified version of the Sable network architecture, it inherits many of its scaling properties. In particular, the Sable paper showed superior scaling trends to MAT [1] and competitive trends compared to fully independent actors for up to 1000 agents. This suggests that Oryx will scale to even more than 50 agents. The main bottleneck we encountered with scaling beyond 50 agents was handling the very large datasets. Generating datasets for systems with so many agents became quite an engineering challenge to handle. We stopped at 50 for this paper because we felt it struck a good balance between pushing the frontier of research while still being manageable for other future researchers to use without too much engineering overhead. In the future, building a better data pipeline should allow us to test Oryx at even greater scale.\n    - Oryx works very well in continuous control settings. All of the Multi-Agent MuJoCo tasks are continuous control problems, and Oryx significantly outperformed other algorithms on most of those datasets.\n* *In the sequential update scheme, how sensitive is the overall policy quality to the order in which agents are updated?*\n    - In Oryx, we randomly shuffle the order of agent updates at each learning step, similar to MAT [1] and HAPPO [2]. We agree that if you had some informed prior on which order of agent updates would be best, this may help speed up learning. However, we did not experiment with this because we wanted to keep our method as general as possible. And developing a method to automatically learn the best order of updates is out of the scope of this work, but it is an interesting direction for future work.\n* *In sequential updates, each agent optimizes assuming the policies of the other agents are fixed. Does this introduce error accumulation across the agent chain?* \n    - Indeed, the accumulation of error as agent policies are updated one after another is a central challenge in offline MARL. However, since Oryx uses an autoregressive policy scheme (like MAT [1]) where agents condition on the actions of agents that came before them, the error will be less than fully independent policies. This fact gets at the central benefit of using a model like Oryx.\n* *In your implementation of Implicit Constraint Q-learning for each agent, how is the constraint enforced during policy updates?*\n    - The ICQ constraint is enforced in two ways. First, during the critical learning step (Q-learning), one strictly only uses the actions observed in the dataset for computing the target value for the next state. This avoids extrapolation error due to out-of-distribution actions in the value function. Then, at the policy learning step, the policy distribution is updated with respect to in-distribution actions only (i.e. actions in the dataset), which implicitly avoids out-of-distribution actions. Together, these constraints mitigate the accumulation of extrapolation error. More details on the ICQ loss can be found in the original paper [3]. Central to our contribution is the sequential and autoregressive variant of the ICQ loss.\n\n**On the Ablation Study**\n\nFinally, with regard to the reviewer's comment about the lack of ablation studies, we specifically designed the TMAZE experiments to isolate which components of Oryx were important. Our main takeaway from the TMAZE ablation study was that removing any one component from Oryx (autoregressive actions, memory, or ICQ) resulted in failure on the task. Highlighting that only by combining all three ingredients together did the algorithm succeed. We conducted the ablation in our custom-designed TMAZE environment because we felt its simplicity made the results more interpretable. If, however, the reviewer feels the ablation study would be strengthened by conducting it over a more complex, but less interpretable, environment (e.g. RWARE), we would happily consider adding that in for the camera-ready version.\n\nIn conclusion, thank you once again for your very constructive questions. They have helped us sharpen our messaging in the paper. In particular, we will add a discussion on scaling to the revised version, along with more detail on how exactly the autoregressive policy updates work in practice. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.\n\n[1] Wen, Muning, et al. “Multi-Agent Reinforcement Learning is a Sequence Modeling Problem.” NeurIPS ‘22\n\n[2] Zhong,Yifan, et al. “Heterogeneous-agent reinforcement learning.” Journal of Machine Learning Research, January ‘24\n\n[3] Yang, Yiqin, et al. “Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning” NeurIPS ‘21"}, {"review_id": "bbD97no4RW", "reviewer": "Reviewer_vYPS", "summary": "Oryx proposes a novel offline multi-agent reinforcement learning (MARL) algorithm addressing two core challenges: extrapolation error and miscoordination in many-agent settings. It integrates:\n1. Sable's retention-based architecture for long-context sequence modeling.\n2. Sequential Implicit Constraint Q-learning (ICQ) with auto-regressive policy updates.\n3. Counterfactual advantage for low-variance policy gradients.\n\nEvaluated across various datasets (SMAC, RWARE, Multi-Agent MuJoCo), Oryx achieves SOTA in >80% of tasks, demonstrating superior scalability and coordination.", "questions": "1. How does inference memory scale with agent count (e.g., in 50-agents)?\n\n2. Can sequential updates handle highly heterogeneous agents?", "limitations": "None", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Novel fusion of Sable (retention mechanism) and sequential ICQ enables stable many-agent coordination. Sequential policy updates conditioned on prior agents' actions explicitly mitigate miscoordination.\n\n2. Extensive benchmarks spanning discrete/continuous control, agent scales, and horizon lengths. Ablations validate core components (e.g., T-Maze proves necessity of auto-regression, memory, and ICQ).\n   \n3. Dominates in dense-agent settings (e.g., 50-agent Connector), where prior methods (MAICQ) fail. Introduces new large-scale datasets for community use.\n\nWeaknesses\n\n1. Retention blocks and sequential decoding likely increase training costs vs. non-sequential methods (not quantified).", "was_revised": true, "final_justification": "My questions have been solved.", "rebuttal": "We thank the reviewer for their constructive comments on our paper. We appreciate the reviewer highlighting the novelty of our algorithm and acknowledging that the new benchmark datasets we provide are valuable contributions. Concerning the author's questions, we now address each in detail:\n\n* *How does inference memory scale with agent count (e.g., in 50-agents)?*\n    - The Sable [1] paper showed the scaling properties of the network for agents up to 1000. The memory scaling properties are significantly better than MAT (which is quadratic in the number of agents) and competitive with algorithms that use independent actors. This suggests that Oryx will scale to even more than 50 agents. The main bottleneck we encountered with scaling beyond 50 agents was handling the very large datasets. Generating datasets for systems with so many agents became quite an engineering challenge to handle. We stopped at 50 for this paper because we felt it struck a good balance between pushing the frontier of research while still being manageable for other future researchers to use without too much engineering overhead. In the future, building a better data pipeline should allow us to test Oryx at even greater scale.\n* *Retention blocks and sequential decoding likely increase training costs vs. non-sequential methods (not quantified).*\n    - Yes, it is true that sequential decoding increases computational complexity in the number of agents. In MAT, the complexity scales quadratically in the number of agents, which can be prohibitive for large numbers of agents. Sable, on the other hand, has far superior scaling properties thanks to the retentive network architecture. While not as good as non-sequential (independent) action selection methods, it is very competitive up to 1000 agents. Sable (and by extension Oryx) therefore strikes a good balance between superior performance and competitive scaling. It is for this reason that we chose to use the Sable network as a starting point for Oryx rather than the MAT network. \n* *Can sequential updates handle highly heterogeneous agents?*\n    - Yes, it can! Although how to achieve this depends on what exactly you mean by heterogeneous agents. If you mean the agents assume different roles in the environment, but their observation and action spaces are the same, then Oryx handles this gracefully right out of the box. If, on the other hand, agents have different-sized observation spaces and action spaces, you can still apply Oryx, but first need to pad the spaces to the same size. This is relatively easy to do and has worked well when we have tried it. Moreover, we are confident about Oryx’s ability to handle heterogeneous agents because it builds on the foundations set by the Heterogeneous Agents RL [3] line of work, which first introduced the sequential updating scheme that was then extended to Sequence Modelling by MAT [2], and finally Sable. \n\nOnce again, we thank the reviewer for their comments. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.\n\n[1] Mahjoub, Omayma, et al. “Sable: a Performant, Efficient and Scalable Sequence Model for MARL.” ICML ‘25\n\n[2] Wen, Muning, et al. “Multi-Agent Reinforcement Learning is a Sequence Modeling Problem.” NeurIPS ‘22\n\n[3] Zhong,Yifan, et al. “Heterogeneous-agent reinforcement learning.” Journal of Machine Learning Research, January ‘24"}, {"review_id": "qEwha2a35P", "reviewer": "Reviewer_wEyu", "summary": "This work proposes Oryx, which combines (1) Sable for sequential modeling, (2) ICQ for mitigating value over-estimation and (3) COMA in Offline MARL. Oryx is compared with other baselines on MA-MuJoCo, SMACv1 and RWARE as well as Connector and T-Maze.", "questions": "Please address each point raised in Strengths and Weaknesses.", "limitations": "yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The major weakness is in the use of autoregressive policies, while baselines use independent policies in the experiments. This choice is not about the different algorithmic choices (e.g. sequential updates as in HAPPO), but about the policy structure itself. This policy structure should be the same for all algorithms for a fair comparison. I believe the authors are confusing autoregressive policy structures $\\pi(a|s) = \\pi_1(a_1|s) \\pi_2(a_2|s, a_1) \\cdots$ on the one hand and sequential policy updates where each agent learns a best response to the other agents’ current policies. If Oryx is the only algorithm which uses an autoregressive policy structure, then its an unfair comparison with other baselines which use an independent policy structure i.e. $\\pi(a|s) = \\pi_1(a_1|s) \\pi_2(a_2|s) \\cdots$.\n\nSecond, as far as I can decipher from the architecture in Figure 1, Oryx requires joint observations in the encoder during both training and testing, which means it is not Centralized Training Decentralized Execution (CTDE) but rather Centralized Training Centralized Execution (CTCE). It is a little strange that Oryx was compared to different CTDE algorithms while not compared with MAT [1] which is CTCE. \n \nFinally, Oryx is simply a combination of three pre-existing algorithms, namely Sable, ICQ and COMA. \n\nIn terms of strengths, the set of benchmarks and baselines considered is comprehensive. However, the experimental setup is not correct as aforementioned, with a lack of clarity between CTCE vs CTDE and autoregressive policy structures.\n\n[1] Multi-Agent Reinforcement Learning is a Sequence Modeling Problem (Wen, et al. NeurIPS 2022)", "was_revised": true, "final_justification": "The authors have added additional experimental results which partially resolved my concerns regarding its comparison to MAT which is another CTCE algorithm. \nHowever, I believe the decision to mix CTCE and CTDE in the experimental setup makes the claims in the paper much weaker and inconclusive.\nSimply put, the policies used in Oryx have more information (joint observations + joint actions) compared to baselines (local observations + local actions). It is not an apples to apples comparison.\n\nFurthermore, as the other reviewers have also pointed out, the algorithmic contributions are minimal. \nAs a combination of advantage decomposition and ICQ, the theoretical contributions are not significant. The current form of the paper is lacking some key insight which explains why this combination is non-trivial (algorithmically) and why it is better than other methods, especially CTCE methods.\n\nI recommend making this distinction clear and using a fair setup if the authors still want to emphasize algorithmic contributions. Otherwise, the dataset itself can be a contribution in a more appropriate venue.\n\nMy overall impression of the paper has improved, but I don't find it ready for publication in its current form.\nThus, I've raised my score on originality and significance from 1 to 2 and the final score from 2 to 3.", "rebuttal": "We thank the reviewer for the time taken to review our paper. We appreciate the reviewer highlighting that our “set of benchmarks and baselines considered is comprehensive.” \n\n**Comparing Oryx to MAT**\n\nWe acknowledge that it may not be immediately clear that our algorithm is not a CTDE algorithm and thank the reviewer for pointing this out; by no means did we intend to obscure this fact. We thought it was clear that by building on the MAT and Sable line of work, the reader would be aware that Oryx is a centralised approach. In hindsight, this assumption was misguided. \n\nOur first iteration of the Oryx algorithm was built by extending the MAT network architecture (rather than the Sable one), but ultimately, we settled on the variant with the Sable network architecture because of its superior performance and scaling properties. But since Sable had already been shown to be better than MAT (at least in the online case), we thought it was not worth repeating here. We therefore only reported our performance compared to algorithms from the existing offline literature, and to date, all but MADIFF are CTDE algorithms. \n\nHaving now considered your review, we see that this was a mistake and can easily lead to confusion. To address this, we will add in our experiments comparing Oryx to the variant which used the MAT network architecture, and be clear about their difference to CTDE algorithms. Below are the results across all 43 SMAC datasets and all 6 RWARE datasets. On each dataset, we ran each algorithm over 10 random seeds. \n\nHowever, converting MAT, an online and on-policy algorithm, to an offline algorithm was itself a non-trivial task, which involved \n* Replacing the learning from a queue of online samples with learning from sampled batches of experience from a replay buffer (dataset).\n* Replacing the PPO loss  (which only works with online samples) with a loss function that works on off-policy samples.\n* Modifying the MAT network architecture to accommodate the new loss.\n\nIn particular, we replaced the PPO-style loss in MAT with our novel Autoregressive version of the ICQ loss that we derived (and also use in Oryx). The MAT network modifications required for this were similar to the ones outlined in the Oryx paper. \n\n**Aggregated Results: Oryx vs. MAT**\n\nTo compare the MAT variant and Oryx in a robust way, we applied the proposed statistical methodology from MARL-eval [1]. Namely, we provide 4 different aggregated metrics: the mean, median, interquartile mean, and optimality gap, with stratified bootstrap confidence intervals [in square braces]. To facilitate aggregation, the results for each dataset were normalised against the highest performance on that dataset before computing the aggregated performance metrics. From the tables below, it is clear to see that Oryx substantially outperforms the offline MAT+ICQ on both SMAC and RWARE datasets, with better scores across all metrics (for optimality gap, a lower score is better). We will add these results, along with a clarification of the difference between these methods and the CTDE methods, to our revised paper. \n\n**SMAC**\n|                             | MAT+ICQ            | ORYX                   |\n|------------------------|--------------------------|-------------------------|\n| Median                | 0.71 [0.62, 0.74]    | 0.91 [0.88, 0.92]   |\n| IQM                     | 0.67 [0.66, 0.69]    | 0.87 [0.86, 0.88]   |\n| Mean                   | 0.63 [0.62, 0.64]    | 0.77 [0.76, 0.79]   |\n| Optimality Gap    |  0.38 [0.36, 0.39]    | 0.23 [0.22, 0.25]   |\n\n**RWARE**\n|                             | MAT+ICQ            | ORYX                 |\n|------------------------|--------------------------|------------------------|\n| Median                | 0.85 [0.84, 0.86]    | 0.89 [0.88, 0.9]    |\n| IQM                     | 0.85 [0.84, 0.86]    | 0.89 [0.89, 0.9]    |\n| Mean                   | 0.84 [0.83, 0.84]    | 0.89 [0.88, 0.9]    |\n| Optimality Gap    | 0.16 [0.16, 0.17]    | 0.11 [0.1, 0.12]    |\n\n**Raw Tabulated Results**\n\nWe will include the raw tabulated results for these two baselines alongside the others in the appendix of the camera-ready version of the paper. We could not include them here because the tables used too many characters, given the character limit. From the results, one can see that MAT+ICQ is also a very strong algorithm when compared to the CTDE algorithms from the literature. Which, we agree, is to be expected given their Autoregressive action selection at evaluation time. To us, this result highlights the benefit of using this class of algorithms whenever possible, i.e., when fully decentralised execution is not required (more on this towards the end of our rebuttal). Our work is also the first to demonstrate the applicability of autoregressive action selection in the offline MARL setting.\n\n\n**On the Significance of our Contribution**\n\nWe believe that developing Oryx involved several intricate steps and novel insights that extend beyond a simple integration of existing ideas. We would like to highlight the following key aspects:\n* **Novel Sequential Autoregressive ICQ Loss**: We developed and formally derived a novel sequential autoregressive version of the ICQ loss, which we believe is a significant theoretical contribution.\n* **Offline Training of Online Algorithms**: A core challenge we addressed was modifying online (on-policy) algorithms like Sable and MAT to enable their effective offline training from off-policy samples from a replay buffer (dataset). This adaptation presented non-trivial technical hurdles.\n* **Network Architecture Modifications**: We carefully considered and implemented modifications to the original Sable (and MAT) network architecture to incorporate the dual Q-value and Policy distribution heads required for the ICQ loss.\n* **Temporal Memory Handling**: Appropriately handling the resetting of temporal memory in the retention-based memory of the Sable backbone, particularly when training from offline sequences, required careful attention to detail.\n* **Empirical Performance**: Our work demonstrates substantially superior performance, not just marginal, across multiple benchmarks (SMAC, RWARE, Multi-Agent MuJoCo) and new challenging datasets featuring up to 50 agents. Having addressed the issue raised by the reviewer regarding comparison to MAT, we are confident our robust empirical investigation differentiates our work within the existing literature.\n* **Open-Sourcing and Reproducibility**: To further contribute to the field and facilitate future research, we have already open-sourced both our implementation and the novel datasets, along with clear instructions for reproducing our experiments. We are confident that this will pave the way for further innovation in models derived from Oryx, and we are excited about the potential for others to build upon this direction.\n\nMoreover, we believe that classifying our contribution as merely \"combining existing ideas\" might overlook a common aspect of scientific advancement. Many significant breakthroughs often emerge from novel syntheses of existing concepts, and Oryx exemplifies this by integrating diverse ideas from the MARL field to achieve a result that is greater than the sum of its individual components. While an incremental improvement on a narrow set of benchmarks might warrant such a conclusion, our work demonstrates a significant performance gain over existing methods, supported by extensive experimentation. Furthermore, we have expanded the scope of research by developing and sharing a challenging set of benchmark datasets with up to 50 agents. Having addressed the experimental oversight highlighted by the reviewer, we are confident that our method represents a novel and significant contribution.\n\n**Final Note on CTDE vs Oryx (and MAT/Sable)**\n\nIndeed, our algorithm is not a CTDE algorithm; it belongs to the same class of algorithms as MAT and Sable. Thus, it is not directly applicable in settings where decentralised execution is required (although it's always possible to apply a decentralised policy distillation step at the end of training, if required [2,3]). Having said that, algorithms like MAT, Sable, and Oryx are nonetheless still practically very valuable as they lie somewhere in between the two extremes of fully independent actors (which scales well but may converge on a sub-optimal joint policy) and fully centralised learners (which scale poorly but have better guarantees to converge on an optimal policy in the limit) [4]. Thus, in problems with a large joint action space, which can naturally be factorised, Oryx (as well as MAT and Sable) provides a solution that trades off scalability on the one hand and quality of the joint policy on the other. For this reason, Oryx remains a highly relevant algorithm for many practical industrial applications.  \n\n**In conclusion**, we once again thank the reviewer for their valuable input on our paper. Having addressed the main concern we feel the work is significantly strengthened. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.\n\n[1] Gorsane, Rihab, et al. “Towards a Standardised Performance Evaluation Protocol for Cooperative MARL.” NeurIPS ‘22\n\n[2] Zhu, Zhengbang, et al. “MADiff: Offline Multi-agent Learning with Diffusion Models,” \nNeurIPS ‘24\n\n[3] Li, Yueheng, et al. “Multi-Agent Guided Policy Optimization.” Arxiv 2025\n\n[4] de Kock, Ruan, et al. “Is an Exponentially Growing Action Space Really That Bad? Validating a Core Assumption for using Multi-Agent RL.” AAMAS ‘25"}, {"review_id": "vTHqdYsW9z", "reviewer": "Reviewer_KGQ8", "summary": "The main methodological contribution of this work lies in integrating existing approaches—specifically, the combination of Sable (a retention-based sequence model) and MAICQ (implicit constraint Q-learning). The main claims made in the abstract and introduction accurately reflect the scope of this work. The paper clearly states that Oryx is a combination of existing methods (Sable and MAICQ) for scalable offline multi-agent reinforcement learning. While the work does not offer fundamentally new algorithmic insights, it provides a comprehensive and systematic empirical evaluation demonstrating the effectiveness and scalability of the combined approach across a wide range of challenging benchmarks.", "questions": "Is there any theoretical guarantee for the Oryx which is a combination of existing methods (Sable and MAICQ) for scalable offline multi-agent reinforcement learning?\nThe related work part can be further extended.", "limitations": "The limitations are discussed explicitly in the conclusion. In addition to acknowledging that large-scale real-world validation remains future work, the paper implicitly recognizes that the core contribution is a thoughtful integration of existing techniques rather than a novel algorithmic breakthrough.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "S. The discussion is candid about the scope and boundaries of the work.\nW. The paper does not introduce fundamentally new algorithmic principles, but instead systematically demonstrates that this combination yields significant practical benefits in challenging offline MARL scenarios. The claims, experiments, and discussions throughout the paper accurately reflect this.", "was_revised": true, "final_justification": "Thank you for the detailed rebuttal. I now better appreciate the non-trivial engineering contributions, particularly the novel autoregressive ICQ loss and the offline adaptation of Sable. The strong empirical results across diverse benchmarks, along with open-sourcing efforts, further strengthen the paper’s practical value.\n\nWhile theoretical guarantees remain limited, I agree that the demonstrated performance and scalability justify sharing this work with the community. I have accordingly raised my score from 3 to 4. I encourage the authors to clarify the novelty and theoretical positioning more explicitly in future revisions.", "rebuttal": "We thank the reviewer for their time taken to review our paper. With the following response we hope to address the reviewers' concern regarding the novelty of our work as well as the question about theoretical scaling guarantees. \n\n**On the novelty and significance of Oryx**\n\nWe understand that at a high level Oryx may seem like a simple combination of existing approaches; however, we would argue that this overlooks the many practical challenges we had to overcome in order to implement this algorithm successfully. Moreover, we believe that many significant and impactful contributions to science over the years have come from novel combinations of existing ideas (potentially from different subfields). Of course, if the results for Oryx were incremental at best, then one could justifiably lean towards rejecting it. But since we showed extensively that the combination of ideas in Oryx results in a substantial improvement over the state-of-the-art across a broad range of tasks, we believe that it's an important finding that should be shared with the research community. Particularly because scalable Offline MARL has huge practical potential for a range of problem areas. \n\nFurthermore, we think that overly optimising for novelty at scientific conferences has actually resulted in some bad incentives that have led to some dubious research. Take, for example, the recent Offline MARL benchmarking paper [1], which showed that many of the novel ideas introduced to the field in recent years as conference papers turned out not to perform as well as relatively simple algorithms that “simply” combined existing ideas from the literature. Thus, we think that penalising us for not appearing to be novel enough is unfair, particularly in this case where we have provided substantial empirical evidence that the method works far better than anything else in the field.\n\n**More details on Oryx contributions**\nFurthermore, would like to draw attention to some of the key aspects of our contribution, which we believe make it significant:\n* **Novel Sequential Autoregressive ICQ Loss**: We developed and formally derived a novel sequential autoregressive version of the ICQ loss, which we believe is a significant theoretical contribution.\n* **Offline Training of Online Algorithms**: A core challenge we addressed was modifying online (on-policy) algorithms like Sable and MAT to enable their effective offline training from off-policy samples from a replay buffer (dataset). This adaptation presented non-trivial technical hurdles.\n* **Network Architecture Modifications**: We carefully considered and implemented modifications to the original Sable (and MAT) network architecture to incorporate the dual Q-value and Policy distribution heads required for the ICQ loss.\n* **Temporal Memory Handling**: Appropriately handling the resetting of temporal memory in the retention-based memory of the Sable backbone, particularly when training from offline sequences, required careful attention to detail.\n* **Empirical Performance**: Our work demonstrates substantially superior performance, not just marginal, across multiple benchmarks (SMAC, RWARE, Multi-Agent MuJoCo) and new challenging datasets featuring up to 50 agents. Having addressed the issue raised by the reviewer regarding comparison to MAT, we are confident our robust empirical investigation differentiates our work within the existing literature.\n* **Open-Sourcing and Reproducibility**: To ensure full transparency and foster continued research, we have already made our implementation and the novel datasets openly available, complete with detailed instructions for experiment replication. We are confident this initiative will greatly facilitate further advancements stemming from Oryx-based models, and we eagerly anticipate the community's contributions building upon this foundation.\n\n**On Oryx’s theoretical guarantees**\nSable, on which Oryx is built, has theoretical guarantees when using the PPO training objective. This stems from the work by Kuba et al. (2022) [2]. In particular, HAPPO (multi-agent PPO with autoregressive updates) was shown to be an instance of their multi-agent mirror learning framework and to be theoretically sound. To obtain an instance of mirror learning requires only defining a valid drift functional, neighbourhood operator and sampling distribution. In Sable, these design choices are exactly as they are for HAPPO, and therefore, Sable inherits the same theoretical monotonic improvement and convergence guarantees as HAPPO. However, although Oryx shares a network architecture that is similar in spirit as Sable, it is fundamentally different as an offline algorithm and uses a different autoregressive objective. This makes it non-trivial to place Oryx within the same mirror learning framework and thus claim similar theoretical guarantees. We consider this an interesting avenue for future work. \n\nWhile Oryx's theoretical guarantees remain an exciting avenue for future research, the significant empirical results and novel algorithmic contributions presented in this work already establish a strong foundation, demonstrating the considerable practical value and advancement this paper brings to the community. \n\n**In conclusion**, we trust we have addressed the main concerns raised by the reviewer. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.\n\n[1] Formanek, Claude, et al. “Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation.” NeurIPS ‘24\n\n[2] Kuba, Jakub Grudzien, et al. \"Heterogeneous-agent mirror learning: A continuum of solutions to cooperative marl.\" arXiv preprint arXiv:2208.01682 (2022)."}]

Author rebuttal:

Meta-review: This work proposes Oryx, an offline MARL algorithm, targeting the extrapolation error and miscoordination that arise when learning from fixed datasets. It combines Sable and a sequential form of implicit constraint Q-Learning (ICQ). It theoretically proves that the multi-agent joint-policy under ICQ regularisation can be optimised sequentially for an auto-regressive model. Through experiments, the authors show that their method performs better than multiple methods on multiple benchmarks.

The strengths of this work are summarized as follows.
1. It combines ICQ and the Advantage Decomposition Theorem to address the extrapolation error and miscoordination problem. Such an approach is novel. 
2. Oryx demonstrates state-of-the-art results across multiple offline MARL benchmarks.
3. Introducing new large-scale datasets for community use.

The weaknesses of this work are summarized as follows. 
1. While the primary contribution is a well-engineered integration of known ideas, the empirical performance and release of datasets make it a good addition to the offline MARL literature.
2. Oryx is a CTCE method with autoregressive policies, while many baselines are CTDE with independent policies. Oryx has more information (joint observations + joint actions) compared to baselines (except MADIFF).
Decision: Accept (poster)