Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Authors: Omayma Mahjoub
Sasha Abramowitz
Ruan John de Kock
Wiem Khlifi
Simon Verster Du Toit
Jemma Daniel
Louay Ben Nessir
Louise Beyers
Juan Claude Formanek
Liam Clark
Arnu Pretorius
Venue: ICML 2025 poster (2025)
Keywords: Multi-agent reinforcement learning
Reinforcement Learning
Sequence modeling
linear recurrent models
Retentive Networks
Primary area: reinforcement_learning->multiagent
TL;DR: We develop a state-of-the-art, memory efficient and scalable cooperative multi-agent reinforcement learning algorithm that leverages retention.

Abstract: As multi-agent reinforcement learning (MARL) progresses towards solving larger and more complex problems, it becomes increasingly important that algorithms exhibit the key properties of (1) strong performance, (2) memory efficiency, and (3) scalability. In this work, we introduce Sable, a performant, memory-efficient, and scalable sequence modelling approach to MARL. Sable works by adapting the retention mechanism in Retentive Networks (Sun et al., 2023) to achieve computationally efficient processing of multi-agent observations with long context memory for temporal reasoning. Through extensive evaluations across six diverse environments, we demonstrate how **Sable is able to significantly outperform existing state-of-the-art methods in a large number of diverse tasks (34 out of 45 tested)**. Furthermore, Sable maintains performance as we scale the number of agents, handling environments with more than a thousand agents while exhibiting a linear increase in memory usage. Finally, we conduct ablation studies to isolate the source of Sable's performance gains and confirm its efficient computational memory usage. **All experimental data, hyperparameters, and code for a frozen version of Sable used in this paper are available on our website:** https://sites.google.com/view/sable-marl. **An improved and maintained version of Sable is available in Mava:** https://github.com/instadeepai/Mava.

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Omayma Mahjoub"
Sasha Abramowitz
Ruan de Kock"
Wiem Khlifi
Simon du Toit' Jemma Daniel Arnu Pretorius
Louay Ben Nessir
Louise Beyers'
Claude Formanek
Liam Clark
Abstract
Mull-Agant Transformer
Sabli
As multi-agent reinforcement learning (MARL) progresses towards solving larger and more com- plex problems, it becomes increasingly impor- tant that algorithms exhibit the key properties of (1) strong performance, (2) memory efficiency, and (3) scalability. In this work, we introduce Sable, performant, memory-efficient, and scal- able sequence modelling approach to MARL.. Sable works by adapting the retention mecha- nism in Retentive Networks (Sun et al., 2023) to achieve computationally efficient processing of multi-agent observations with long context mem- ory for temporal reasoning. Through extensive evaluations across six diverse environments, we demonstrate how Sable is able to significantly outperform existing state-of-the-art methods in large number of diverse tasks (34 out of 45 tested). Furthermore, Sable maintains perfor- mance as we scale the number of agents, handling environments with more than a thousand agents while exhibiting a linear increase in memory us- age. Finally, we conduct ablation studies to iso- late the source of Sable's performance gains and confirm its efficient computational memory us- age. All experimental data, hyperparameters, and code for a frozen version of Sable used in this paper are available on our website. An improved and maintained version of Sable available in Mava.
35
11X
80
6.5X
14
200
32
7X
Number times ranked best
Throughput (SPS) 1
GPU Memory
(GB)
Figure I. Performance, memory, and scaling properties of Sable compared to the Multi-Agent Transformer (MAT) (Wen et al., 2022), the previous state-of-the-arr, aggregated over 45 coop- erative MARL tasks. Left: Sable ranks best in 34 out of 45 tasks, outperforming other MARL algorithms tested across envi- ronments: RWARE, LBF, MABrax, SMAX, Connector, and MPE. MAT ranked best of 3/45. Middle: Sable exhibits superior through- put, processing up 6.5 times steps per second compared MAT as we scale to 512 agents. Right: Sable scales efficiently to thousands of agents, maintaining stable performance, while using GPU memory significantly more efficiently than MAT.
require decision-making systems that have strong perfor- mance for accurate control, are memory efficient to maxi- mize compute hardware and are able scale to large number of agents representing the different components of a com- plex system.
Although many existing MARL approaches exhibit or two of the above properties, solution effectively encom- passing all three remains elusive.
1. Introduction
Large-scale practical applications of mullti-aget reinforce- ment learning (MARL), such as autonomous driving (Lian & Deshmukh, 2006; Zhou al., 2021; Li al., 2022) and electricity grid control (Kamboj al., 2011; 1; Li et al., 2016),
To elaborate, we briefly consider the current spectrum of MARL, from fully decentralised training, or independent learning systems, to fully centralised learning. We consider these paradigms in terms of their potential exhibit:
'Equal contribution 'InstaDeep. Correspondence to: Ruan de Kock, Arnu Pretorius <(ndekkock, a.pretorius) @instadecp.co>>.
1. Strong performance: the general ability to solve tasks at a moderate scale.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Memory efficiency: the hardware memory require-

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
ments to perform joint policy inference execution time.
coordination capabilities when deployed at larger scale (Long et al., 2020; Christianos al., 2021; Guresti & Ure, 2021).
3. Scalability: the ability to maintain good performance as the number of agents in the system grows large.
Another line of work in CTDE has focused on develop- ing more theoretically principled algorithms. In particular, Kuba et al. (2022a) develop trust region learning methods which are subsequently extended by Kuba et al. (2022b) into a mirror learning framework for MARL, culminat- ing in the Fundamental Theorem of Heterogeneous-Aggen Mirror Leurning (Theorem 1). The theorem states that for specifically designed methods that utilise a particular heterogeneous-agen update scheme during policy optimi- sation, monotonic improvement and convergence are guar- anteed. Stemming from this work, a class of heterogeneous agent RL algorithms has been proposed including HATRPO, HAPPO, HAA2C, HADDPG, and HASAC (Zhong et al., 2024; Liu et al., 2023b). Although theoretically sound, these algorithms generally suffer from the same drawbacks as con- ventional CTDE methods in terms of practical performance at scale, for similar reasons (Guo et al., 2024).
Independent learning (IL) memory efficient but not performant or scalable. On the one end of the spectrum lies II., or decentralised methods, where agents act and learn independently. As intuitively expected, the carliest work in MARL uses this approach (Tan, 1993; 1997). However, even early on, clear limitations were highlighted due to non-stationarity from the perspective of each learning agent (Claus& & Boutilier, 1998), failing to solve even simple tasks. When deep neural networks were introduced into more mod- ern MARI. algorithms, these algorithms also followed the IL paradigm (Tampuu et al., 2017; Witt et al., 2020), with reasonable results. II. algorithms demonstrate proficiency in handling many agents memory- -fffcceeeffiiint way by typi- cally using shared parameters and conditioning on an agent identifier. However, at scale, the performance of IL remains suboptimal compared to more centralised approaches (Pa- poudakis et al., 2021; Yu et al., 2022; Wen et al., 2022).
Centralised learning (CL) - strong performance but not yel memory efficient or scalable. On the other end of the spectrum lie centralised algorithms. These include classi- cal RL algorithms that treat MARL as a single-agent prob- lem with an expanded action space, as well as approaches that condition on global information during execution, e.g. graph- communication methods (Zhu et al., 2022). A particularly interesting line of work in this setting has been to re- re-frame MARL as (typically offline) sequence model- ing problem (Chen et al., 2021; Meng et al., 2021; Tseng et al., 2022; Zhang et al., 2022; Liu et al., 2023a; Forsberg et al., 2024) and employ the use of transformers ( (Vaswani et al., 2017; Hu et al., 2021; Gallici et al., 2023; Yang et al., 2024) to learn effective control policies. One such approach for online learning is the Multi-Agent Transformer (MAT) (Wen et al., 2022) which is currently the state-of-the-ar for cooperative MARL. Although MAT is highly performant, it has limitations: (1) it lacks the ability to scale to truly large multi- systems due to the inherent memory limitations of the attention mechanism (Katharopoulos et al., 2020), and (2) it is not able to condition on observation histories.
Centralised training with decentralised execution (CTDE) - performant and memory efficient but not yet scalable. As a solution to the failings of independent learn- ing and lying between decentralised and centralised meth- ods, is CTDE (Kraemer& & Banerjee, 2016). Here, cen- tralisation improves learning by removing non-stationarity. while decentralised execution maintains memory-efficient deployment. CTDE follows two main branches: value- based and actor-critic. In value-based methods, centralisa- tion is achieved through a joint value function used during training which has a factorisation structure adhering to the sovriidullgglbba---a
(IGM) principle. This means that if each agent acts greedily at execution time it is equiva- lent to the team acting greedily according to the joint value function. Seminal work along this line includes VDN (Sune- hag et al., 2017) and QMIX (Rashid et al., 2018), with many follow-up works investigating value decomposition in MARL (Son al., 2019; Rashid et al., 2020b; Wang et al., 2020a; Son et al., 2020; Yang et al., 2020; Rashid et al., 2020a). In actor-critic methods, centralised critic is used during training and at execution time policies are deployed independently. Many popular single-agent actor- critic algorithms have multi-agent CTDE versions including MADDPG (Lowe et al., 2017), MAA2C (Papoudakis et al., 2020), and MAPPO (Yu et al., 2022), and have been com- bined with factorised crities (Wang al., 2020b; Peng et al., 2021). Although CTDE helps during training to achieve bet- ter performance at exccution time, centralised training may remain prohibitively expensive, especially if the size of the global state is agent dependent. Furthermore, independent policies, even when trained jointly, are often limited in their
Our work - sttate-ofthhe-ar performance, memory efficient and scalable. We seek to develop an approach capable of strong performance while being memory efficient and able to scale to many agents. To achieve this, we take inspiration from the centralised MAT architecture and recent work in linear recurrent models in RL (Lu et al., 2024; Morad et al., 2024) to develop an efficient online sequence modelling approach to MARL. Our method is centralised algorithm, yet is scalable and memory efficient. Our key innovation to obtain strong performance and scalability is to replace the attention mechanism in MAT with an RL-adapted version
2

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
of relention, from Retentive Networks (RetNets) (Sun al., 2023). We call our approach Sable.
2. Background
Problem Formulation We model cooperative MARL as Dec-POMDP (Kaelbling et al., 1998) specified by the tuple (W,S, (Aitee, P, R, [Ri ( (ON 7). At each timestep t, the system is in state 8 S. Each agent E N selects an action of E A,, based on its observation of E Ω4 contributing to a joint action a E A= A= ITEN A, Executing at in the environment gives a shared reward Te = R(st, at), transitions the system to a new state St+1 ~ P(- at), and provides each agent with new observation o1+1
Sable is able to handle settings with up to thousand agents and can process entire episode sequences as stateful memory, crucial for learning in partially observable settings. Through comprehensive benchmarks across 45 different tasks, we empirically verify that Sable significantly outperforms the current state-of-the-art in a large number of tasks (34 out of 45). This includes outperforming MAT (see Figure 1), the current best sequence model for MARL, while having memory efficiency comparable to independent PPO, which is fully decentralised, achieving the best of both ends of the MARL spectrum. In other words, Sable is a centralised al- gorithm with strong performance without the drawbucks of a centralised design, which is typically far less memory effi- cient and scalable when compared to fully decentralised and CTDE methods. Although concurrent work has explored using linear recurrent sequence models as world models in RL (Cohen et al., 2024; Wang et al., 2024), Sable is the first demonstration of successfully using RetNets for learning control policies. We summarise our contributions below:
O,(-|s++1, az). To support the full spectrum of MARL, our observation function O is defined to be general in the following sense: it is function that maps from the underlying global state and joint action to the agent's probability distribution over the power set of con- catenated observations, given as O : S x A + ((P()))) where P(2,) is the power set of the observation space S2 and A (P((s))) represents the set of all probability distribu- tions over P(2.). This means that for independent obser- vations, the probabilities are only non-zero over singleton sets (i.e. single observations) and for centralised observa- tions, it has full support (i.e. includes probability mass on all possible combinations). For more detail, see Appendix A. The goal is to learn a joint policy π <{alo), over an episode horizon T, that maximises the expected sum of discounted rewards, ((r) E,, E.
RetNet innovations for RL We propose the first RetNet-based architecture suitable for online RL. We develop a novel encoder-decoder RetNet with a cross- retention mechanism and adapt the decay matrix with resettable hidden states over temporal sequences. This is to ensure correct information handling from expe- rience gathered across episode boundaries, unique architectural change required for successful RL train- ing in RetNets.
Retention Retention as used in Retentive networks (Ret- Nets) introduced by (Sun et al., 2023), climinates the softmax operator from attention and instead incorporates a time-decaying causal mask (decay matrix) with GroupNorm (Wu & He, 2018) and swish gate (Hendrycks & Gimpel, 2016; Ramachandran et al., 2017) to retain non-linca This reformulation allows for the same computation to be expressed in three distinct but equivalent forms:
A new sequence model for MARL We combine the above innovations into novel MARL architecture which achieves state-of-the-art performance, can rea- son over multiple timesteps, is memory efficient, and scales to over a thousand agents.
1. Recurrent which operates on a single input token at a time via a hidden state
Comprehensive benchmarking results Our evalu- ations across 45 tasks from 6 diverse environments provide nearly double the experimental data for com- parison dedicated benchmarking work (Papoudakis et al., 2020; Yu al., 2022; Bettini et al., 2024), gives the first set of comprehensive results on newly pro- posed IAX-based environments (Bonnet et al., 2023; Rutherford et al., 2023) and is substantially more than what has been historically provided in recently pub- lished work (Gorsane et al., 2022).
h. = khh-11 + K.V. Ret(x,) = Qaher s= L......
(1)
where Q., K,, V, are por-token query, key, and value ma- trices, respectively, Each of these is computed by applying learned projection matrices Wq. WK. and Wy on the em- bedded input sequence æ. The decay factor x E (0, 1) determines the rate which information from earlier parts of the sequence is retained.
2. Parallel which operates on a batch akin to attention, given as
of tokens in parallel,
Data and code We publicly release all our experi- mental data and code, with scripts to reproduce all our findings. We also hope this will make it easy for the research community to build on this work.
if 2 m
Ret(2= = (QKT ④ D)V, Dam =
0, if s m,
(2)
3

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
MLP
Gecoder
SWIGLD
Secoder
Recurrent Retention K
SWIGZZ
C Chankwuise Retention
Recurrent
Rid Si
Figure Sable architecture and execution. The encoder receives all agent observations of, of from the current timestep along with a hidden state h representing past timesteps and produces encoded observations 6) .i.....0 oi observation-valee v())), v(,, and a new hidden state htne, decoder performs recurrent retention over current action o; mn-1 followed by cross attention with the encoded observations, producing the next action af". The initial hidden states for recurrence over agents in the decoder at the current timestep are deen m and by the end of the decoding process, it generates the updated hidden states (haee) A Adees).
where D is referred the decay matrix.
poral dependencies across entire episodes. We explain how Sable operates during both training and execution, how adapt retention to work in MARL, and provide different strategies for scaling depending on the problem setting.
3. Chunkwise which is a hybrid between the parallel and recurrent forms and allows for efficient sequeee mod- eling. The approach involves splitting the sequence into smaller chunks, each length B, and can be written as:
Execution Sable interacts with the environment for de- fined rollout length, L, before each training phase. During this interaction, the encoder (left in Figure 2) uses chunk- wise representation, processing the observation all agents at each timestep in parallel. hidden state hame maintains memory of past observations and is reset the end of each episode. During execution, the decay matrix, D, is set to all ones, allowing for full self-retention over all agents' observations within the same timestep. These adjustments to retention, particularly the absence of decay across agents and the resetting of memory at episode termination, result in the following encoder formulation during execution:
Qu BBB((B- Ki K BBB-1):BB Vig VB h4 = KT (V() ② () + h;-1:
Gik= B-i-1 A E (1....B
Ret(z)g) = (Q K o DV + (Q(3)h--1) O G xt v E (1...B) B)- (3)
The above equivalent representations enable two key advan- tages over transformers: (1) it allows for constant memory usage during inference while still leveraging modern hard- ware accelerators for parallel training, and (2) it facilitates efficient handling of long sequences by using the chunkwise representation during training, which can re-expressed in recurrent form during inference.
Ret(o:) = Qche one.
(4...,11.
where hone = 8(Khim + KJ V1),
(4)
0, the episode has ended
8 = 1, if the episode ongoing
3. Method
In this section, we introduce Sable, our approach to MARL as sequence modelling using a modified version of retention suitable for RL. Sable enables parallel training and memory- efficient execution scale, with the ability to capture tem-
where Qt, Kr, Ve are query, key, and value matrices of all agents' observations at timestep within the rollout that started at 1th timestep and o, is the embedded observation sequence.

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Algorithm Sable
Require: rollout longth (L), updates (U), agents epochs
epochs (K)) minibatches (M)
Jooint + hard hard 0
Initialize hidden state for encoder and decoder zeros
2: for Update = = .,,,..., U do
"jont Pjnint
Store initial hidden states for training
for 1,
5: de, vr, Aace + encoder..hunkwiiseeee Monc)
fori=11 1, do
7:
of. (2 get decoder.
A-Auto-regressively decode actions
for Step enviromment using a, and store (Ot, (de, de, Yt, Tota ((c(o)) in buffer
10: if episode terminates then Ajoins O1 clse hjjint Achyiinn
11:
end for
12:
Use GAE compute advantage estimates Á and value targets 0
13:
for epoch = 1.2,.... , K
do
14:
Sample trajectories
(ob..: Abb:: do Toua Motal Abo. (0oz.)) from B, shuffling along the agent dimension
15:
for minibatch 1.2..... ... M
16:
Generate Denc, Dine given Equations and 10, and day.2 Ō Vou.. & encoder. chunkwise Nome" Dene)
17:
T (as... (bs,..) ) decader chunkwl se(as) dL: Atrain Ddec)
19:
8 8 V Loo Ana ora, ara)
20:
Φ ee- Φ Vo IMSE( Youay Mouse )oys))
21:
end for
22: end for
23:
end for
The decoder (right in Figure 2) operates recurrently over both agents and timesteps, decoding actions auto- regressively per timestep as follows:
of algorithms that lie on the continuum of Heterogencous Agent Mirror Learning (HAML), with proven convergence guarantees (Kuha al., 2022b).
Ret(a)) = Qhh,,
Sable uses the chunkwise representation for both encoding and decoding during training, allowing it to process entire trajectories in parallel while using a hidden state to maintain the memory of previous trajectories. Pseudocode for the training phase can be found in Lines 12-22 of Algorithm 1.
(5)
where hs = h1-1 + (K)) i 1
with h1 = hdes + VZ and h ((shx). Here, Qi. K2. Vt are query, key and value matrices and at the embedded action of the jth agent at timestep t. The hidden state dene is carried across timesteps, decaying at the end of each timestep and resetting to zero when an episode ends. Within each timestep, the intermediate variable h; is sequentially passed from one agent the next and is used exclusively in the retention calculation. Pseudocode for the execution phase can be found in Lines 4-11 of Algorithm 1.
Sable's chunkwise formulation processes trajectories т, of length L, containing N agents, by dividing them into smaller segments of length C. It applies decay factor over time and resets the memory using matrices D, and at the end of each episode, resulting in chunkwise training equation for each chunk i.
Training During training, Sable samples entire trajecto- ries 7 from an on-policy buffer and randomly permutes the order of agents within timesteps. The encoder takes as input sequence of flattened agent-timestep observations from an entire trajectory: lot of o o.4... representing sequence of observations that start at timestep 1. The decoder takes a similar sequence of actions instead of obser- vations as input. The encoder and decoder outputs are then used to update the network via the clipped PPO objective function (Schulman et al., 2017). This objective, combined with auto-regressive action selection, puts Sable in the class
QTT Q0(--1):s
K(T) Ko(--1)Ci
V(r.) Vo(-1)ios
h; Kita (Vrr oc) + 8n SL/ h
(6)
Ç D [N/C,,NNN
Ret(2)).) (QIT)KEE o DD VmTI + (0mm)/))) CE
where E = (II/NNHH
if S Ntdo if > Nldo
0,
5

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
The floor operator in ç. [ ensures that all agents from the same timestep share the same decay values. The input [[r] is the sequence of observations from T for the encoder and the sequence of actions for the decoder. We represent the index of the first terminal timestep in 2i(r] as ldg and use h to denote the hidden state at the end of the current trajectory т. Finally, the matrix
tened agent-timestep observation along the time axis dur- ing training with the constraint that agents from the same timestep must always be in the same chunk. This allows Sable to process chunks of rollouts several factors smaller than the entire rollout length while maintaining a memory of the full sequence during processing.
a modified version of the decay matrix from standard retention, with dimensions (NI, NL), which define in more detail below.
Code Our implementation of Sable is in JAX (Bradbury et al., 2023). All code for the version of Sable used in this paper is available on our website, while an improved and maintained version of Sable is available in Mava.
In practice, rather than computing /t during training, we reuse the hidden states from the final step of the previous execution trajectory Tprevs which means that we replace hypre with hf in the case of the encoder and h in the case of the decoder.
4. Experiments
We validate the performance, memory efficiency and scala- bility of Sable by comparing it against several SOTA base- line algorithms from the literature. These baselines can broadly be divided into two groups. The first group consists of heterogeneous agent algorithms that leverage the advan- tage decomposition theorem. To the best of our knowledge, the Multi-Agent Transformer (MAT) (Wen et al., 2022) represents the current SOTA for cooperative MARL on dis- crete environments, and Heterogeneous Agent Soft Actor- Critic (HASAC) (Liu et al., 2023b) the current SOTA on continuous environments. The second group includes well- established baseline algorithms. including IPPO (Witt et al., 2020), MAPPO (Yu et al., 2022), QMIX (Rashid al., 2020a) and MASAC. For all baselines, we use the JAX- based MARL library Mava (de Kock et al., 2023).
Adapting the decay matrix for MARL In order for Ret- Nets to work in RI, we make three key adaptations to the decay matrix used during training. First, we ensure that each agent's observations are decayed by the same amount within each timestep. Second, We ensure that the decay matrix accounts for episode termination so that information is not allowed to flow over episode boundaries. Third, we construct an agent block -wiee decay matrix for the encoder to ensure that there is full self-retention over agents within each timestep. A more detailed discussion on the construc- tion of the decay matrices as well as an illustrative example are given in Appendix E.
Scaling and efficient memory usage In practical appli- cations, there might be different axes of interest in terms of memory usage. For example, scaling the number of agents in the system, or efficiently handling the sequence length that can be processed at training time, i.e. the number of timesteps per update. We propose slightly different ap- proaches for efficient memory use and scaling across each axis.
Evaluation protocol We train each algorithm for 10 in- dependent trials for each task. Each training run is allowed 20 million environment timesteps with 122 evenly spaced evaluation intervals. At each evaluation, we record the mean episode return over 32 episodes and, where relevant, any additional environment specific metrics (e.g. win rates). In line with the recommendations of Gorsane et al. (2022), we also record the absolute performance. For task-level aggre- gation, we report the mean with 95% confidence intervals while for aggregations over entire environment suites, we re- port the min-max normalised inter-quartile mean with 95% stratified bootstrap confidence intervals. Following from Agarwal et al. (2021), we consider algorithm X to have a significant improvement over algorithm Y if the probabil- ity of improvement score and all its associated confidence interval values are greater than 0.5. All our evaluation ag- gregations, metric calculations, and plotting leverage the MARL-eval library from Gorsane et al. (2022).
Scaling the number of agents. Scaling to thousands of agents requires a significant amount of memory. Therefore, in this setting, we use MAT-style single-timestep sequences to optimise memory usage and reserve chunking to be ap- plied across agents. This requires only changing the encoder during execution, as the decoder is already recurrent over both agents and timesteps. However, this change to the en- coder makes it unable to perform full self-retention across agents, as it cannot be applied across chunks. During train- ing, the process mirrors that of execution but is applied to both the encoder and decoder.
Scaling the trujectory context length. Since the training sequence will grow proportional to N in the case where Sable maintains memory over trajectories, training could become computationally infeasible for tasks requiring long rollouts. In order to accommodate this, we chunk the flat-
Environments We evaluate Sable on several JAX-based benchmark environments including Robotic Warehouse (RWARE) (Papoudakis et al., 2021), Level-based forag- ing (LBF) (Christianos et al., 2020), Connector (Bonnet et al., 2023), The StarCraft Multi-Agent Challenge in
6

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Mean apisade return
1.0-
NAT
Mean episode return
1.0
NASAC
Mean episode return
1.0
WAPRO
0.8
35
0.6
0.6
0.0
0.4
0.4
DMX
2P0
525
0.0
2.5
75
16 125
0.0
0.0
2.5
17.5
20
Timesteps [Millions]
Timesteps [Millions]
Timesteps [Millions]
(a) RWARE
(b) MABrax
(c) SMAX
Mean episode return
Mean spisode return
1.0
Mcan episode return
1.0
0.8-
0.81
0.6
0.6
0.6
19P0
0.2
KAPP
0.0
0.0
0.01
25
17.5
7.5 10 12.5 Timesteps [Millions]
1255 Timesteps [Millions]
175 20
Timesteps [Millions]
(d) Connector
(e) LBF
(f) MPE
Figure 3. Sample efficiency curves and probability of improvement scores aggnegated per environment suite. For each environment, results are aggregated over all tasks and the min-max normalized inter-quartile mean with 95% stratified bootstrap confidence intervals are shown. Inset plots indicate the overall aggregated probability of improvement for Sable compared to other baselines for that specific environment. A score of more than 0.5 where confidence intervals also greater than 0.5 indicates statistically significant improvement over a baseline for a given environment (Agarwal et al., 2021).
Table environment episode return. Inter-quartile mean of the absolute episode returns with 95% stratified bootstrap confidence intervals. Bold values indicate the highest score per environment, and an asterisk indicates that a score overlaps with the highest score within one confidence interval.
Enviromment
Sable (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
RWARE
0.81 (o. 70.0.83
(0.600.7,0771)
0.51 (0.47,055)
(1..
MABrax
0,56 53,0.88;
(044(0.42)0447)
(.600.570064)
0.5 49,00.22
0.82 (0.7..07).
0.8 (3.760083)
SMAX
0.94 92,0.983
0.92
(.9..).))) (0.,0.))))
0.86 84,00.888
Connector
0.95 05.0.883
0.88 88,0 80)
0.91 (0.1,.00)
0.93
LBF
1.0 0.0.1.0
0.99 (0.98.0.9))
1.0 1.0.1.0..0)
0.99 0.99 (0.99,1.0)
MPE
0.95 94.00..66)
0.69 (0.68.07777
0.81 0.81 (0.50..11
(0.7.0.)).
0.76
(.79...0...22)
JAX (SMAX) (Rutherford et al., 2023), Multi-agent Brax (MABrax) (Peng et al., 2021) and the Multi-agent Particle Environment (MPE) (Lowe et al., 2017). All environments have discrete action spaces with dense rewards, except for MABrax and MPE, which have continuous action spaces and RWARE which has sparse rewards. Furthermore, we compare to HASAC and MASAC only on continuous tasks given their superiority in this setting, and QMIX only on SMAX as has been shown to perform suboptimally in other discrete environments (most notably in sparse reward settings such RWARE) (Papoudakis et al., 2020). Fi- nally, highlight that our evaluation suite comprised of 45 tasks represents nearly double the amount of tasks used by prior benchmarking work (Papoudakis et al., 2020) and substantially more than conventional research work recently
published MARL (Gorsane et al., 2022).
Hyperparameters All baseline algorithms as well as Sable were tuned on each task with a tuning budget of 40 trials using the Tree-structured Parzen Estimator (TPE) Bayesian optimisation algorithm from the Optuna library (Akiba et al., 2019). For discussion on how to access all task hyperparameters and for all tuning search spaces, we refer the reader to Appendix D.
4.1. Performance
In Figure 1, we report the amount of times that an algorithm had a significant probability of improvement over all other algorithms on a given task. Furthermore. we present the per environment aggregated sample efficiency curves, prob-
7

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
ability of improvement scores and episode returns in Figure 3 and Table 1. Our experimental evidence shows Sable achieving state-of-the-arl performance across wide range of tasks. Specifically, Sable exceeds baseline performance on 34 out of 45 tasks. The only environment where this is not the case is on MABrax. For continuous robotic control tasks SAC a particularly strong baseline, typically outper- forming on- on-policy methods such as PPO (Haarnoja et al., 2018; Huang et al., 2024; Freeman et al., 2021a). a). Given that Sable uses the PPO objective for training, this performance is unsurprising. However, Sable still manages to achieve the best performance in the continuous control tasks on MPE. We note that previous benchmarking and evaluation work (Papoudakis et al., 2020; Gorsane et al., 2022) has recom- mended training off-policy algorithms for factor of 10 less environment interactions than on-policy algorithms due to more gradient updates for the same number of environment interactions. In our case, we find that off-poliig systems do roughly 15 times more gradient updates for the same amount of environment interactions. If we had done this, the performance of HASAC, MASAC and QMIX would have been less performant than reported here. Additional tabular results, task and environment level aggregated plots given in Appendix C.
A new environment for testing agent scalability in coop- erative MARL A task in Neom is characterised by peri- odic, discretised I I-dimensional pattern that is repeated for a given number of agents. Each agent observes whether it is in the correct position and the previous actions it has taken. Agents receive a shared reward which is calculated as the Manhattan distance between the team's current pattern and the underlying task pattern. We design three task patterns:
(1) s simple-simee
{0.5, 0.7, 0.8, 0.7, 0.5, 0.3, 0.2, 0.3), (1, (3) quick-flie ip: (0.5.0, 0.5, 0}. For more details, see Appendix B.7.
(2) half-1-half-0
Experimental setup We evaluate the performance of Sable, MAT, and IPPO on the LBF and Neom environments. For LBF, experiments involve tasks with 32, 64, and 128 agents, while for Neom we include tasks with 32, 512 and 1024 agents. To measure the memory usage efficiency on both LBF and Neom environments on the agents' axis, we select 32 the fixed chunk size, which corresponds the smallest number of agents used in experiments.
Results From the results in Figure 4, we observe the fol- lowing, First, Sable and IPPO exhibit similar memory scal- ing trends, with IPPO's lower memory usage attributable to its much smaller networks. However, while IPPO scales well from a memory perspective, it is unable to learn effec- tively with many agents. achieves high initial rewards, but its performance degrades as training progresses. Second, we find that Sable can consistently outperform MAT. Although the margin is small, this performance is achieved while main- taining comparable computational memory usage to IPPO, whereas MAT scales poorly and requires more than the maximum available GPU memory (80GB) on Neom tasks with 1024 agents. Notably, for Neom with 1024 agents, Sable sustains a stable mean episode return of around 25, indicating that approximately 40% to 50% of the agents successfully reached the target location. This result is sig- nificant given the shared reward structure, which poses a difficult coordination challenge for such a large population of agents.
4.2. Memory usage and scalability
We assess Sable's 's ability to efficiently utilise computational memory, focusing primarily on scaling across the agent axis.
Challenges in testing scalability using standard envi- ronments Testing scalability and memory efficiency in standard MARL environments poses challenges, as many en- vironments such as SMAX, MPE and Connector, expand the observation space as the number of agents grows. MABrax has uniquely assigned roles per agent making it difficult to scale up and RWARE does not have a straightforward way to ensure task difficulty as the number of agents increases. For these reasons, the above environments are difficult use when testing algorithmic scalability without significantly modifying the original environment code. Among these, LBF is unique because it is casier to adjust by reducing agents' field of view (FOV) while maintaining a reasonable state size and offering faster training. However, other than the FOV, still requires modifications to ensure a fixed observation size, see Appendix B.3 for more details. De- spite these adjustments. LBF could not fully demonstrate Sable's scaling capability, as it could not scale past 1 128 agents due to becoming prohibitively slow. Therefore, to explore scaling up to a thousand agents, we introduce Neom, a fully cooperative environment specifically designed to test algorithms on larger numbers of agents.
4.3. Ablations
We aim to better understand the source of Sable's perfor- mance gains compared to MAT. There are two specific im- plementation details that Sable inherits from the retention mechanism that can easily be transferred to attention, and therefore to MAT. The first is using root mean square nor- malization (RMSNorm) (Zhang & Sennrich, 2019) instead of layer normalization (Lei Ba et al., 2016) and the second is using SwiGLU layers (Shazeer, 2020; Ramachandran et al., 2017) instead of feed forward layers. Other than implementation details, the difference between MAT and Sable is that Sable uses retention instead of attention and it conditions on histories instead of on single timesteps. To
8

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
GPU Memoory (G0)
101
Sable Jount]
Mean espeesde return
Sandor Aguet
eprodde
94
agentt
37 73 10 150
375 267
Timestops [Milions]
id0 12.5 120 200
36 73 TEL 157 W/5 100
[Milions]
(a) LBF memory
(b) LBF 32 agents
(c) LBF 64 agents
(d) LBF - 128 agents
Meren eppende reterdeereeuumm retuum
Mean eppieciie return
E. t 24 0 x
32-
CPU Neroory
63
Number of agents
Timestops [Millians)
Timestops [Millons]
Timestops [Milions)
(c) Ncom memory
(I) Neom 32 agents
(g) Ncom 512 agents
(h) Neom 1024 agents
Figure 4. Memory usage and agent scalability. When scaling to many agents, Sable is able to achieve superior converged performance
while maintaining memory efficiency. MAT memory on Neom 1024 agents and thus it's curve is omitted from (h).
Mcan optcode return
Nonn apisode return
Sade tount
Maatt uphonde wht
3.
bun
03
Timestops [Millonss]
25 108
150 175 233
5.0 150 12.5 150 175 20 Timestaps [Milions]
Chunk Size
80
5. 15 125 263
Timestops [Nilions]
(a) Ablations Implementation details
(b) Ablations Memmrry vs chunk size
Figure 5. Ablation studies on RWARE and SMAX. (a) Comparing Sable with MAT with modifications from Sable's implementation details. (b) Showing the relationship between chunk size, performance and memory usage on RWARE.
determine the reason for the performance difference be- tween Sable and MAT, we adapt MAT to use the above implementation changes, both independently and simultane- ously and we compare MAT to a version of Sable with no memory that only conditions on the current timestep. We test all three variants of MAT and the Sable variant RWARE tasks (tiny-4ag and medi um 4ag)) and two SMAX tasks (3s5z z. and ama cv2.5 S.nuniit ts) and compare them with the original implementation. We tune all methods using the same protocol as the main results.
ple efficiency of MAT, although not to the level of Sable's performance. When we compared MAT, Sable, and Sable without memory, we discovered that Sable's performance improvement stems from its ability to use temporal memory, rather than from the retention mechanism itself. This evident because Sable without memory (which performs similarly to MAT) differs from Sable only in how the input sequence is structured.
In Figure 5b, we see that even when dividing the rollout trajectories into chunks that are up to a factor of 16 smaller than the full rollout length, Sable's performance remains consistent, while its memory usage decreases.
In Figure 5a, we see that the above implementation details do make difference to MAT's performance. In RWARE, MAT's variants slightly increase in both performance and sample efficiency. However, Sable still achieves signifi- cantly higher performance while maintaining similar sam- ple efficiency. The same cannot be said for Sable without memory which performs similarly to the default MAT and significantly worse than MAT with the implementation im- provements. In SMAX, we observe marked increase in sample efficiency for MAT equaling the sample efficiency of Sable and outperforming Sable's 's sample efficiency with- out memory, but no increase in overall performance. This is likely due to the fact that both MAT and Sable already achieve close to the maximum performance in these SMAX environments. In summary, we find that these implementa- tion details do matter and improve the performance and sam-
5. Conclusion
In this work, introduced Sable, novel cooperative MARI. algorithm that achieves significant advancements in memory efficiency, agent scalability and performance. Sable's ability to condition on entire episodes provides with enhanced temporal processing, leading to strong perfor- mance. This is evidenced by our extensive evaluation, where Sable significantly outperforms other leading approaches. Morcover, Sable's memory efficiency complements its per- formance by addressing the significant challenge of scaling MARL algorithms as it is able to maintain stable perfor- mance even when scaled to over a thousand agents.

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Acknowledgements
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-150977, 2021.
The authors would like to thank Andries Petrus Smit for use- ful discussions on RetNets, Nacef Labidi and the MLOps team for developing our model training and experiment or- chestration platform Alchor, and the rest of the InstaDeep decision-making research team for valuable discussions dur- ing the preparation of this work. We would like to thank the Python ( (Van Rossum et al., 1995; Oliphant, 2007; Hunter, 2007; Harris et al., 2020) and JAX (Bradbury et al., 2023; DeepMind et al., 2020; Heek et al., 2024; Bonnet et al., 2023; de Kock al., 2023; Rutherford al., 2023; Toledo et al., 2023) communities for developing tools that made this work possible. We also thank the Google TPU Re- search Cloud (TRC) for supporting this research with access to cloud TPUs. Finally, the authors would like to thank the anonymous ICML reviewers for useful feedback which helped to improve the quality of our manuscript.
Cho, K. Learning phrase representations using mn encoder- decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Christianos, F., Schäfer, L., and Albrecht, S. V. Shared expe- rience actor-critic for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Christianos, F., Papoudakis, G., Rahman, M. A., and Al- brecht, S. V. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Con- ference on Machine Learning. pp. 1989-1998. PMLR, 2021.
Impact Statement
Our aim in this work is to contribute to the advancement of reinforcement learning algorithms that are specifically suit- able for practical applications, such as control large-scale multi-agent systems. Although it is possible to enumerate many potential societal impacts, due its generality, we feel no specific aspect need be highlighted here.
Claus, C. and Boutilier, C. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998(746-752):2, 1998.
Cohen, I.,, Wang, K., Kang, B., and Mannor, S. Improv- ing token-based world models with parallel observation prediction. In Forty-first International Conference on Machine Learning, 2024.
References
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural in- formation processing systems, 044:9304--93200 2021.
de Kock, R., Mahjoub, O., Abramowitz, S., Khlifi, W., Tilbury, C. R., Formanek, C., Smit, A. P., and Preto- rius, A. Mava: a research library for distributed multi- agent reinforcement learning in jax. arXiv preprint arXiv:2107.014660, 2023. URL https:/ //arsv///xvvv.. oxg/pdf/2107.. 01460. 14466.pp.... pdf.
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
DeepMind, Babuschkin, I., Baumli, K., Bell, A., Bhupati- raju, S., Bruce, J., Buchlovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Dedieu, A., Fantacci, C., Godwin, J., Jones, C., Hemsley, R., Hennigan, T., Hes- sel, M., Hou, S., Kapturowski, S., Keck, T., Kemaev, I. King, M., Kunesch, M., Martens, L., Merzic, H., Mikulik, V., Norman, T., Papamakarios, G., Quan, J., Ring, R., Ruiz, F., Sanchez, A., Sartran, L.., Schnei- der, R., Sezener, E., Spencer, S., Srinivasan, S., Stano- jević, M., Stokowiec, W., Wang. L., Zhou, G., and Vi- ola, F. The DeepMind JAX Ecosystem, 2020. URL http: http:///gttttt com/ gowgge-
Bettini, M., Prorok, A., and Moens, V. Benchmarl: Bench- marking aulti-agent reinforcement learning. Journal of Machine Learning Research, 25(217):1-10, 2024.
Bonnet, C., Luo, D., Byme, D., Surana, S., Abramowitz, S., Duckworth, P., Coyette, V., Midgley, L.. I., Tegegn, E., Kalloniatis, T., et al. Jumanji: a diverse suite of scal- able reinforcement learning environments in jax. arXiv preprint XrXiv:2306. 09884, 2023.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurrn, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman- S., and Zhang, Q. JAX: composable transformations of Python+NumPy programs, 2023. URL http::/gh thub.. com/ www./oogll/oooooooooooooooooooo. google/ / jax.
Forsberg, A. L., Nikou, A., Feljan, A. V., and Tumova, J. Multi-agent transformer-acceleraaee for satisfaction of stl specifications. arXiv preprint 24X3:2403.15111 2024.
10

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Freeman, C. D., Frey, E.,, Raichuk, A., Girgin, S., Mordatch, I,, and Bachem, O. Brax differentiable physics gine for large scale rigid body simulation, 2021a. URL https:/ / /arxivv....//////a///a//////aav. org/ abs/ 2106.11066.0 13281.
Huang, S., Gallouédec, Q.. Felten, F., Raffin, A., Dossa, R. F. J., Zhao, Y., Sullivan, R., Makoviychuk, V., Makovi-
ichuk, D., Danesh, M. H., Roumégous, C., Weng, J., Chen, C., Rahman, M. M., M. Araújo, J. G., Quan, G., Tan, D., Klein, T., Charakorn, R., Towers, M., Berth- elot, Y., Mchta, K., Chakraborty, D., KG, A., Charraut, V., Ye, C., Liu, Z., Alegre, L. N., Nikulin, A., Hu, X., Liu, T., Choi, J., and Yi, B. Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning, arXiv preprini wwx.v:2402.03000 2024. URL https:/ www.aaivvvoog.....o htwpaa///222222 020.0.0..334.
Freeman, C. D., Frey, E.,, Raichuk, A., Girgin, S., Mordatch, I., and Bachem, O. Brax a differentiable physics engine for large scale rigid body simulation, 2021b. URL http: //github com/ google/bres
Gallici, M., Martin, M., and Masmitja, Transfqmix: Transformers for leveraging the graph structure of multi- agent reinforcement learning problems. arXiv preprint arXiv:2301.05334 2023.
Hunter, J. D. Matplotii:: A 2d graphics environment. Com- puting science & engineering, 9(03):90-99) 95, 2007.
Gorsane, R., Mahjoub, O., de Kock, R. J., Dubb, R., Singh, S., and Pretorius, A. Towards standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems, 35:5510-5521 2022.
Kaelbling, L. P., Littman, M. L.,, and Cassandra, A. R. Plan- ning and acting in partially observable stochastic domains. Artificial intelligence, 1998.
Kamboj. S., Kempton, W., and Decker, K. Deploying power grid-integrated electric vehicles as a multi-agent system. volume 1, pp. 13 13-20, 01 2011.
Guo, X., Shi, D., Yu, and Fan, W. Heterogeneous multi- agent reinforcement learning for zero-shot scalable col- laboration. arXiv preprint arXiv:2404.03869 2024.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnss: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5166 1165. PMLR, 2020.
Guresti, B. and Ure, N. K. Evaluating generalization and transfer capacity of multi-agent reinforcement learn- ing across variable number of agents. arXiv preprint arXiv:2111.141177 2021.
Kraemer, L. and Banerjee, Multi-agent reinforcement learning rehearsal for decentralized planning. Neuro- computing, 190:82-10 94, 2016.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Confer- ence on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URLhttpss http:///pooceeeeeeeee mlr. press v80 2a2000/aaarojjja 8b. html.
Kuba, J. G., Chen, R., Wen, M., Wen, Y., Sun, F,, Wang, J., and Yang, Y. Trust region policy optimisation in multi- agent reinforcement learning. In International Confer- ence on Learning Representattons, 2022a. URL https: / //opnrevvee.... net conttfforumee
Kuba, J. G., Feng, X., Ding, S., Dong, H., Wang. J., and Yang, Y. Heterogeneous- mirror learning: A con- tinuum of solutions to cooperative marl. arXiv preprint orXhv:2208.0166) 2022b.
Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., al. Array programming with 2020.
numpy. Nature, 585(7825))357-384
Lei Ba, Kiros, R., and Hinton, G. E. Layer normaliza- tion. ArXiv e-prints, pp. arXiv-1607, 2016.
Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., and van Zee, M. Flax: A neural network library and ecosystem for JAX, 2024. http:t/ github. com/ google/flax..
Li, W., Logenthiran, T., Phan, V.-T., and Woo, W. L. In- telligent multi-agent system for power grid communica- tion. In 2016 IEEE Region 10 Conference (TENCON), pp. 3386-3388 2016. 7848681.
doi: 10.1109/TENCON
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.088: 0666600845, 2016.
Li, Y., Ma, D., An, Z., Wang. Z., Zhong, Y., Chen, S., and Feng, C. V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. JEEE Robotics and Automation Letters. 7(4):09114-0022,, 2022. doi: 10.1 1..1109/L.... R..9/..A..022.319999 2022.11280..
Hu, S., Zhu, F., Chang, X., and Liang, X. Updet: Universal multi-agent reinforcement learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001, 2021.

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Lian, Z. and Deshmukh, A. Performance prediction of unmanned airborn vehicle multi-agent system. European Journal of Operational Research, 172:680-695, 07 2006. doi: 10.
Parisotto, E., Song, F., Rae, J., Pascanu, R., Gulcehre, C., Jayakumar, S., Jaderberg. M., Kaufman, R. L.., Clark, A., Noury, S., et al. Stabilizing transformers for reinforce- ment learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Liu, J., Zhang, Y., Li, C., Yang, C.. Yang. Y., Liu, Y., and Ouyang, W. Masked pretraining for multi-agent decision making. arXiv preprint arXiv:2310. 1846, 2023a.
Peng, B., Rashid, T., Schroeder de Witt, C., Kamienny, P.-.,, Torr, P., Böhmer, W., and Whiteson, S. Facmac: Factored multi-agent centralised policy gradients. Ad- vances in Neural Information Processing Systems, 34: 12208-1222) 2021.
Liu, J., Zhong, Y.. Hu, S., Fu, H., Fu, Q. Chang, X., and Yang, Y. Maximum entropy heterogeneeuu--geent mirror learning. arXiv preprint arXiv:2306. 10715, 2023b.
Ramachandran, P., Zoph, B., and Le, Q. V. Searching for activation functions, 2017. URL https: wwtph:t//axiiiittte //arxiv.. org/ attpaa//...1p.//1 05941.
Long, Q. Zhou, Z.,, Gupta, A., Fang, F,, Wu, Y., and Wang, X. Evolutionary population curriculum for seal- ing multi-agent reinforcement learning. arXiv preprint arXiv:2003. 2020.
Rashid, T., Samvelyan, M., Witt, C., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. arxiv. arXiv preprint arXiv: 1485, 2018.
Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, Multi-agent actor-critic for mixed cooperative-compeiiiee environments. Advances in neu- ral information processing systems, 30, 2017.
Rashid. T., Samvelyan, M.. De Witt, C. S., Farquhar, G.,
Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning, Journal of Machine Learning Research, 21(178):1-51, 2020a.
Lu, C., Schroecker, Y., Gu, A., Parisotto, E., Foerster, J., Singh, S., and Behbahani, F. Structured state space mod- els for in-context reinforcement learning. Advunces in Neural Information Processing Systems, 36, 2024.
Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020b.
Meng, L., Wen, M., Yang, Y., Le, C., Li, X., Zhang, W., Wen, Y., Zhang, H., Wang, J., and Xu, B. Offline pre- trained multi-agent decision transformer: One big se- quence model tackles all smac tasks. arXiv preprint 1rXiv:2112.028844 2021.
Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Khan, A., de Witt, C. S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R. T. Whiteson, S., Lacerda, B., Hawes, N., Rocktaschel, T., Lu, C., and Foerster, J. N. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint 2023.
Morad, S., Lu, C., Kortvelesy, R., Liwicki, S., Foer- ster, J., and Prorok, Revisiting recurrent reinforce- ment learning with memory monoids. arXiv preprint arXiv:2402.09990, 2024.
Oliehock, F. A., Amato, C., al. concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., and Whiteson, S. The StarCraft Multi-Agent Challenge. CoRR, abs/ abss/199..04443 2019.
Oliphant, T. E. Python for scientific computing. Computing in science engineering, 9(3): (33):10-00, 2007.
Papoudakis, G., Christianos, F., Schäfer, L., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint N0wv:2006.07869, 2020.
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High Figh-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
Papoudakis, G., Christianos, F., Schäfer, L., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. URL /ttp://arxivv org/ abs /abs//200 2006 07869.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06344 2017.
Shazeer, N. Glu variants improve transformer. arXiv
preprint 0052200000000..0000222, 2020.
12

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Inter- national conference on machine learning, pp. 5887-5896. PMLR, 2019.
Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ neurips. cc cc/ oc/ppppee_iiiles/pprrr//////// 3 3fae2235477ae0111000530000444442 pdf.
Son, K., Ahn, S., Reyes. R. D., Shin, J., and Yi, Y. Qtran++: improved value transformation for coopera- tive multi-agent reinforcement learning, arXiv preprint arXiv:2006. 12010, 2020.
Wang, J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.00 2000::0008.0106,, 2020a.
Wang. Y., Han, B., Wang, T., Dong, H., and Zhang, Dop: Off Off-policy multi-agent decomposed policy gradi- ents. In International conference learning representa- tions, 2020b.
Sun, Y., Dong, L., Huang, S., Ma. S., Xia, Y., Xue, J., Wang J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023.
Wang. Z., Yue, D., Long, J., and Zhang, Y. Parallelizing model-based reinforcement learning over the sequence length. In The Thirty-eighth Annual Conference Neu- ral Information Processing Systems, 2024.
Sunehag, P., Lever, G., Gruslys, A., Czamecki, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.0553 0566..996..099,,... 2017.
Wen, M., Kuba, J., Lin, R., Zhang, W., Wen, Y., Wang. J., and Yang, Y. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Infor- mation Processing Systems, 35: 65509-1661,,,, 2022.
Tampuu, A., Matiisen, T., Kodelja, D., Kuzovkin, I,, Korjus, K., Aru, J., Aru, J., and Vicente, R. Multiagent cooper- ation and competition with deep reinforcement learning. PloS one, 12(4):e0172395, 2017.
Witt, C. S. D., Gupta, T., Makoviichuk, D., Makoviy- chuk, V., Torr, H. S., Sun, M.. and Whiteson, S. Is independent learning all you need in the star- craft multi-agent challenge? ArXiv, abs/2011.09553 2020. URL https: //ppi. semant cscholar.
Tan, M. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth inter- national conference on machine learning, pp. 330-337, 1993.
Tan, M. Multi-agent reinforcement learning: Independent versus cooperative agents. In International Confer- ence Machine Learning, 1997. URL https: //api. www.sssenanccchoo.e...... oet.aant.cccccol.....ooooooopppp org/CorpusI:: 272885126.
Wu, Y. and He, K. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Yang, M., Yang, Y., Lu, Z., Zhou, W., and Li, H. Hierar- chical multi-agent skill discovery. Advances in Neural Information Processing Systems, 36, 2024.
Toledo, E., Midgley, L., Byme, D., Tilbury, C. R., Macfar- lane, M., Courtot, C., and Laterre, A. Flashbax: Stream- lining experience replay buffers for reinforcement learn- ing with jax, 2023. URL https: //gpt/ub///ttu///// com/ inst anstadepaa i / f1 ashbax/.
Yang. Y., Hao, J., Liao, B., Shao, K., Chen, G., Liu, W., and Tang. Qatten: general framework for cooper- ative multiagent reinforcement learning. arXiv preprint 2020.
arXiv:
Tseng, W.-C., Wang, T.-H. J., Lin, Y.-C., and Isola, P. Of- fline multi-agent reinforeement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226-233 237, 2022.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooper- ative multi-agent games. Advances in Neural Information Processing Systems, 15:24411122466 2022.
Van Rossum, G., Drake, F. L., et al. Python reference man- ual, volume 111. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan. S., and Garnett, R. (eds.), Advances Neural Information
Zhang, F., Jia, C., Li, Y-C.,, Yuan, L., Yu, Y., and Zhang, Z. Discovering generalizable multi-agent coordination skills from multi-task offline data. In The Eleventh Inter- national Conference on Learning Representations, 2022.
13

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Zhong, Y., Kuba, J. G., Feng, X., Hu, S., Ji, J., and Yang. Y. Heterogeneous-agent reinforcement learning. Journal of Machine Learning Research, 25(1-67):1, 2024.
Zhou, M., Luo, J., Villella, J., Yang, Y., Rusu, D., Miao, J. Zhang, W., Alban, M. FADAKAR, I., Chen, Z., Huang, C., Wen, Y., Hassanzadeh, K., Graves, D., Zhu, Z., Ni, Y., NguNeu,,,, N., Elsayed, M., Ammar, H., Cowen- Rivers, A., Ahilan, S., Tian, Z., Palenicek, D., Rezaee, K. Yadmellat, P. Shao, K., chen, d., Zhang, B., Zhang, H., Hao, J., Liu, W., and Wang. J. Smarts: An open- source scalable multi-agent rl training school for au- tonomous driving. Kober. J., Ramos, F., and Tom- lin, C. (eds.), Proceedings of the 2020 Conference Robot Learning, volume 155 of Proceedings of Machine Learning Research, pp. 264-285. PMLR, 16-18 Nov 2021. URL https: http//occeeengss.cp// www.mmmp..... press// html.
v1
Zhu, C., Dastani, M. and Wang, S. survey of multi- agent reinforcement learning with communication. arXiv preprint arXiv:2203.08975 1, 2022.
14

---

Sable: Performant, Efficient and Scalable Sequence Model for MARL
Appendix
A. Background
A.1. A general Dec-POMDP observation function to simultancously support II, CTDE and CL
For specific agent i, let S2; be its set of possible atomic observations. These atomic observations o; Q2 could themselves be complex entities (e.g., embedded and/or concatenated observations from shared information across different agents the environment state). The actual observation perceived by agent i is wi C Q4 meaning w; E PP2)), where P(22;) is the power set of Ω2, The observation function for agent i, denoted by O, maps from the global state and the joint action to probability distribution over the power set of agent i's atomic observations, P(22). This can be written as:
: A
where ((P()))) represents the set probability distributions over P():). The probability that agent observes specific set wi E P(),) given the global state and joint action a is denoted by (s, This formulation allows for handling different learning and execution setups in MARL:
Independent observations (for IL and CTDE during execution): In this case, the probabilities Oi(wi]s, a) are non-zero only for singleton sets (i.e., when agent observes exactly one atomic observation, which is its local observation). Formally, for any s E S and a E A:
if Of s a) > O, then |وس| =
This means that w; {o:} for 0; Ωz Consequently, for any w, P(2:) such that [يm] * 1, O, a) 0.
Centralised observations (for CTDE during training and CL): In this case, the observation function O has full support over P())). This implies that the probability mass can be distributed across all possible subsets of Ω2 Formally,
You E P())), is possible that Ow 8, a) >
This allows an agent potentially observe single piece (local) information, or multiple pieces of information obtained via information sharing between agents and/or the environment state.
Note that although we support all possible settings in our work, including centralised training and execution, our problem setting remains partially observable and aligned with the definition of Dec-POMDP (Olichock et al., 2016). The general problem setting we consider individual agent observations. That is, even at execution the agents can condition on other agents' observations through attention/retention for CL. this does not reconstruct the full state, and therefore remains a partial (but aggregated) observation.
cooperative task with shared rewards where the global state is not factorised across
As concrete example, consider two-agent grid world where agents receive joint reward when they simultaneously reach goal G.
#
#
A2
#
A.1
#
A1 then has a partial observation of the grid, which can be given as
A1
#
while A2 has partial observation
#
A2
#
15

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
An aggregation over these observations does not reconstruct the true global state which implies that the problem remains partially observable and cooperative due to the shared reward.
Finally, we point out that for CL,, it is of course true that the observation function provides more information per agent compared to CTDE and IL methods during execution time. However, this is at the cost of increased inference time requirements (time and compute). It is exactly this drawback of centralised learning that we address our work by introducing Sable.
B. Environment details
B.1. Robot Warchouse
Figure 6. Environment rendering for Robot Warchouse. Task name: tiny-2ag.
The Robot Warehouse (RWARE) environment simulates warehouse where robots autonomously navigate, fetching and delivering requested goods from specific shelves to workstations and then returning them. Inspired by real-world autonomous delivery depots, the goal in RWARE is for team of robots to deliver as many randomly placed items as possible within a given time budget.
The version used in this paper is LAX-based implementation of the original RWARE environment (Papoudakis et al., 2021) from the Jumanji environment suite (Bonnet al., 2023). For this reason, there a minor difference in how collisions handled. The original implementation has some logic to resolve collisions, whereas the Jumanji implementation simply ends an episode if two agents collide.
Naming convention The tasks in the RWARE environment are named according to the following convention:
<size> <num agent agents> a>accifff>
Each field in the naming convention has specific options:
<size>: Represents the size the Warehouse which defines the number of rows and columns of groups of shelves within the warehouse (e.g. tiny, small, medium, large).
<num_agents>: Indicates the number of agents.
<diff>: Optional field indicating the difficulty of the task, where "easy" and 'hard' imply 2N and N/2 requests (shelves deliver) respectively, with N being the number of agents. The default to have N requests.
In this environment, we introduced an extra grid size named "xlarge" which expands the default "large" size. Specifically, increases the number of rows in groups of shelves from three to four, while maintaining the same number of columns.
Observation space In this environment, observations are limited to partial visibility where agents can only perceive their surroundings within 3x3 square grid centred on their position. Within this area, agents have access to detailed information
16

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
including their position and orientation, as well as the positions and orientations of other agents. Additionally, they can observe shelves and determine whether these shelves contain package for delivery.
Action space The action space is discrete and consists of five total actions that allow for navigation within the grid and delivering the requested shelves. These actions include no operation (stop), turning left, turning right, moving forward, and either loading or unloading a shelf.
Reward Agents receive a reward of I for each successful delivery of a requested shelf, coloured in green in Figure 6, to a designated goal (in black) and 0 otherwise. Achieving this reward demands a sequence successful actions, making notably sparse.
B.2. SMAX
Step 30
30
15
Figure 7, Environment rendering for SMAX. Task name: 2s3z.
SMAX, introduced by Rutherford et al. (2023), is re-implermentation of the StarCraft Multi-agent Challenge (SMAC) (Samvelyan et al., 2019) environment using JAX for improved computational efficiency. This redesign eliminates the necessity of running the StarCraft II game engine, thus results on this environment are not directly comparable to results on the original SMAC. In this environment, agents collaborate in teams composed of diverse units to win the real-time strategy game StarCraft. For an in-depth understanding of the environment's mechanics, we refer the reader to the original paper (Samvelyan et al., 2019).
Observation space Each agent observes all allies and enemies within its field of view, including itself. The observed attributes include position, health, unit type, weapon cooldown, and previous action.
Action space Discrete action space that includes 5 movement actions: four cardinal directions, a stop action, and a shoot action for each visible enemy.
Reward In SMAX, unlike SMAC, the reward system is designed to equally incentivise tactical combat and overall victory. Agents earn 50% of their total return from hitting enemies and the other 50% from winning the episode which ensures that immediate actions and ultimate success are equally important.
B.3. Level Based Foraging
In the Level-Based Foraging environment (LBF), agents are assigned different levels and navigate a grid world where the goal is to collect food items by cooperating with other agents if required. Agents only consume food if the combined level of the agents adjacent to given item of food exceeds the level of the food item. Agents are awarded points when food is collected.
The version used in the paper is JAX-based implementation of the original LBF environment (Christianos et al., 2020) from the Jumanji environment suite (Bonnet et al., 2023). To the best of our knowledge, there are no differences between Jumanji's 's implementation and the original implementation.
17

---

Sable: Performant, Efficient and Scalable Sequence Model for MARL
Figure 8. Environment rendering for Level-Based Foraging. Task name: 2s 8 2p-2f..
Naming convention
The tasks in the LBF environment are named according to the following convention:
<obs>- cobs==xksieexxxyiiie---aconotoo
Each field the naming convention has specific options:
<oba>: Denotes the field view (FOV) for all agents. not specified, the agents can see the full grid.
<x_size>: Size of the grid along the horizontal axis.
<y size>; Size of the grid along the vertical axis.
<naggents>; Number of agents.
food>: Number of food items.
<force_c>; Optional field indicating forced cooperative task. In this mode, the levels of all the food items are intentionally set equal to the sum of the levels of all the agents involved. This implies that the successful acquisition of food item requires high degree of cooperation between the agents since agent is able collect food item by itself.
Observation space As shown in Figure the 8x8 grid includes 2 agents and foods. In this case, the agent has a limited FOV labelled "2s", indicating 5x5 grid centred on itself where it can only observe the positions and levels of the items in its sight range.
Action space The action space in the LBF is discrete, comprising six actions: no-operation (stop), picking up food item (apple), and movements in the four cardinal directions (left, right, up, down).
Reward The reward is equal to the sum of the levels of collected food divided by the level of the agents that collected them.
Adapting the LBF environment for scalability experiments In the original Level-Basee Foraging (LBF) implementation, agents processed complete grid information, including items outside their FOV, managed by masking non-visible items with the placeholder (-1.-1,0 0), where each element of this triplet stands for (x, y, level). To improve computational efficiency, revised the implementation completely remove non-visible elements from the observation data, significantly reducing the observation size and ensuring agents process only relevant information within their FOV.
For instance, with standardised FOV of 2, as illustrated in Figure 8, an agent sees 5x5 grid centred around itself. Non-visible items are now excluded from the observation array which makes it easy to convert the agent's 's vector observation
with level one from [1, 2, 3, -1, -1, C, 2, 2, 1, -1, -1, 0] to [ 1, 2, 3, 2, 2, 1]. However, tasks with numerous interacting agents, where the dynamics of visible items consistently change, fixed array sizes are
18

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
required. We address this by defining the maximum number of visible items (2 FOV + 1)2, filling any excess with masked triplets to keep uniform array dimensions.
We designed three scenarios to test scalability on LBF using the standardised FOV of 2, ensuring a maximum of 25 visible items within the 5x5 grid, including each agent's information. To manage agent density as the environment scales, we created the following configurations for our experiments: 2s 2s-32x32-32p-11 6f, 2s- 5x455 64p-32f, 32,,, and 2s 64x64 128p- f.
B.4. Connector
Figure Environment rendering Connector. Task name: con-10x10 0-10a.
The Connector environment consists of multiple agents spawned randomly into grid world, with each agent representing start and end position that needs to be connected. The goal of the environment few steps possible. However, when an agent moves, it leaves behind path that is impassable by all agents. Thus, agents need cooperate to allow the team to connect to their targets without blocking other agents.
connect each start and end position in as
Naming convention In our work, we follow this naming convention for the Connector tasks:
con-<x (nn.ryy.snnee-ttn..eeeeeeeet....
Each field in the naming convention means:
<x size>: Size of the grid along the horizontal axis.
<y-size>:
Size of the grid along the vertical axis.
<num agents>: Indicates the number of agents.
Observation space All agents view an X square centred around their current location, within their field of view, they can see trails left by other agents along with the target locations of all agents. They also observe their current (I,,) position and their target's (z, y) position.
Action space The action space is discrete, consisting of five movement actions within the grid world: up, down, left, right, and no-operation (stop).
Reward Agents receive +1 on the step where they connect and -0.03 otherwise. No reward is given after connecting.
B.5. Mutli-Agent Particle Environments
The Multi-Agent Particle Environments (MPE) comprises physics- based environments within 2D world, where particles (agents) move. interact with fixed landmarks, and communicate. We focus exclusively on the "simple-spread" tasks, the only non-communication, non-adversarial setting the suite, where agents cooperate instead. In this setting, agents aim to
19

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Stap:
1.5
1.0
0.5
0.0
-0.5
2.0
-1.5
-1.0
-0.5
0.0
055
1.0
155
2.0
Figure 10. Environment rendering for Multi-Agent Particle. Task name: simple.spread..2er
cover landmarks to gain positive rewards and avoid collisions, which result in penalties. We employ JAX-based clone of the original environment from the JaxMARL suite (Rutherford al., 2023).
Contrary to the suggestions made by Gorsane et al. (2022), only experiment the reasons, thus, we beyond only the recommended version of a simple-spreas dimmleesppeeddeee and also test on 5 and 10 agents variants.
for the previous
Naming convention our case, the si simple-sprees tasks used in the MPE environment are named according to the following convention:
s
<num agents>: Indicates the number of agents in the simple spread task, where set the number of landmarks equal to the number of agents.
Observation space positions.
Agents observe their own position and velocity as well as other agents' positions and landmark
Action space Continuous action space with actions. Each action represents the velocity in all cardinal directions.
Reward Agents are rewarded based on how far the closest agent each landmark and receive a penalty if they collide with other agents.
B.6. MaBRAX
MaBrax (Rutherford et al., 2023) an implementation the MaMuJoCo environment (Peng et al., 2021) in JAX, from the JAXMARL repository. The difference that it uses BRAX (Freeman al., 2021b) as the underlying physics engine instead
20

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
of MuJoCo. Both MaMuJoCo and MaBrax continuous control robotic environments, where the robots split up so that certain joints controlled by different agents. For example, in ant 4x2, each agent controls different leg of the ant. The splitting of joints is the same in both MaBrax and MaMuJoCo.
The goal is to move the agent forward as far and as fast as possible. The reward is based on how far the agent moved and how much energy it took for the agent to move forward.
Observations are separated into local and global observations. Globally, all agents observe the position and velocity of the root body. Locally, agents observe the position and velocity of their joints as well as the position and velocity of their neighboring joints.
Observation space
Action space A continuous space where each agent controls some number of joints Each of the actions is bounded in the range [] and the value controls the torque applied to a corresponding joint.
Reward Agents receive the reward from the single-agent version of the enviromment. Positive reward is given if the agent moves forward and negative reward is given when energy is used to move the joints. Thus, agents are incentivised to move forward as efficiently as possible.
B.7. Neom
Year wnwioosed Stop
Figure 11. A random initialization of the Neom environment.
Beyar laves (iea00)
Figure 12. solved version of the neom environment.
Neom tasks require agents to match a periodic, discretised 1D pattern that is repeated across the given number of agents. These tasks are specifically designed to assess the agents' ability to synchronise and reproduce specified patterns in a coordinated manner and limited time frame.
21

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Naming convention
The tasks in the Neom environment are named according to the following convention:
<pattern-type> <num agents>ag
Each field in the naming convention has specific options:
peettern-typex Represents the selected pattern for the agents to create ( ("simple-sine", "half-l-half-D" and "quick-flip").
<num.agents>: Indicates the number of agents.
The observation space consists of a binary indicator showing whether the agent is in the correct position, concatenated with the agent's previous actions.
Observation space
Action space The action space consists of unique elements in the pattern, with each element defining actions:
s simplessine {0.2, 0.3, 0.5, 0.8)
half-1-half-0: (1, 0}
quick-flip: {0.5, -0.5}
Reward The reward function is calculated using the mean Manhattan distance between the team's 's current pattern and the target pattern. The reward ranges from for a perfect match to -1 for the maximum difference, with normalization applied. Additionally, if the pattern is correct, the agents receive a bonus that starts at a maximum value 9.0 and gradually decreases as the episode progresses, based on how much time has passed.
C. Further experimental results
C.1. Additional per-task and per-environment results
In Figure 13, give all task-level aggregated results. In all cases, report the mean with 95% bootstrap confidence intervals over 10 independent runs. In Figure 14, we give the performance profiles for all environment suites.
C.2. Additional tabular results
When reporting tabular results, it can be challenging to represent information from entire training run single value for given independent trial. For this reason we give all tabular results for different aggregations. In all cases here, the aggregation method we refer to is the method that was used to aggregate a given training run into a point estimate. For aggregation over these point estimates, we always compute the mean over independent trials along with the 95% bootstrap confidence interval.
C.2.1. MEAN OVER THE FULL TIMESERIES
22

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Sable (ours)
MAT
MAPPO
IPPO
QMIX
MASAC
HASAC
velood recur recure
171 114
106 104
178
10.
Threetps 214
4p
Timestess 1P8 (ISG)
TsTesiess Fersi
Timesees: Milans]
Tessiip Fons)
ar. A02
hoppor 301
Trresadi 24 Bind
Terettppe Birss
Thranaps landl
Temessopp
Trriatass
(Nil ora)
Irnestes (M Fens)
instess (Nfion)
Timesoas (P0lors)
Tanaalapo IM
Timolagl Mona
TirRo.ss (Nllss)
Sty-4op
Bhy 4op hard
sirddd pattuu mm
abrge
1.50
xbage-4ag hard
2032
Timstape and
Tmattops [01e
12 Timeoags (N3lons)
Timooasi
30tr
In ve
1SE 383
123
178 202
10 TMierps/Mildngg
Figure 13. Mean episode return with 95% bootstrap confidence intervals on all tasks.
23

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Sable (ours)
MAT
MAPPO
IPPO
QMIX
MASAC
HASAC
1.00
1.00
1.00
Fraction of rums with score >
Fraction of runs with score >
Fraction of runs with score
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
0.0
0.2
0,4
0.6
0.8
1.0
0.8
Mean episode return (T)
Mean episode return (t)
Mean episode return (t)
(a) RWARE
(b) MABrax
(c) SMAX
+1.00
1.00
1.00
Fraction of runs with score
Fraction of runs with scare
Fraction of runs with score
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.00
0.00
0.00
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
1.0
Mean episode return (T)
Mean episode return (T)
Mean episode return (T)
(d) Connector
(c) LBF
(f) MPE
Figure 14. Per environment performance protiles.
24

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Table Mean episode return training with 95% bootstrap confidence intervals for all tasks. Bold values indicate the highest per task and an asterisk indicates that a score overlaps with the highest score within one confidence interval.
Task
Sable (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
tiny Zag
15.33 oo 15.00)
a1.32/ a0.
4.01:a200.00)
2ag--2ag-had
12.63
8.30 (4.0010
4,28 74..7.00
dag
29.95
22. 22.93
16, 15
9.93ga
smalll-fag dug
11,70 14.11.)
6. (815.57:3433)
2.57
somill-dage Aag-had
7.10
5.77
(8
1.27
medium
7,71
3.74ga
1.27
mediann--ag hard
onu
on or
1.41 54.0
large 4ag
3,50 3.
3.48
1. 1.26,0000
1.19 16.1
large- hard
1.54
1,20
O 0.000
0.01 0.00....0...
slango-aa
2.0222.1..200
16 333333
p.34
0.00
cap-to-ap-tted
0.40 oo (1.009))
medium-fuu bag
8,72
8.65 8.65
6 (200020)))))
2.29
large Nag
mus
10.13
(k..3)
lurge Sag
4.75 205.4)
hopper
1421.60 (22.60011n.... unn 14.568554
1394.73, 1544 19.1449.11)
1358 1358.66
uone (o)
1553 56 cass 171s Bo"
1556.21; 1 2.58.222.00.....000....
Mallrax
halfcheztab. 6xc
2092 89
1899.52 22,22
2335.33
33/2253
2272 98 p11s
2833
3229, 46
walkerrdd
663.1 foyoo oi, 743.08)
623 (4.50 (ar)
1448 pasan 23.2044777
15
2390 74)
1564 24..
2138.08 17.1s-.7...00 27.223
2998 59 casas stsss as)
26 200M an)
humanuid_99-
2066 41
300
463. (43.744412.55.855..
455 (a5.77
4029.0 0l
3095. (2200.03,32244.35
263k
1.96,
164, coL1.00)
93 (.932.022:33
2.741.1..
1.80 T1.1.30
(19.199)
0611.22)
1.812..9011.
1.68
Du.vvs.5 .ir
1,85 (6.1900)
1,64g. (4.1477
66 (7)2247)
1.51
1.68, (66.1.97
Ob, vs Miz
1.92
1,93
1.74g
1,70
1.53
1,38
1.17 1.30)
1.58
1.35
10m0,15,
1.6lly.4.. mg)
3.30g 20.3
1.1 (9.2200
1.3% 20.3
365x_j44 Tafoe
1,62
I. S6y
1.38 38.88..5500 (.2524444
1.58
1.42
27tn, 1i, 30hm
1.93
161 963.65)
(67
1.71:
1.28
J.niisssss
1662 (1.1000
( 62y
1.55 .01
1.50 (0.3333
smacv2.110,ue
( (0))
1,42 en.1
1,48 (4:1119)
1.33
1.30) 1331133
smoov22 20, uniss
1.3
1.23
1.22 S1.)10
0
0.85 0.85
Sa553a
0,85
0.67 o0uuet
0.8 (us,,.0))
0.8]: a01
mou
0.73 710
0.74 mu. 79)
con 10c10.104
21.000)
0.38 1s.0
0.49
oon- 1io-115ti 15x15x230
0.64
0.13 10.-0
0.24
3x8-2p-25-eory
(001000)
0.95 (0250))))
0.97/
2e8-2g-21 24-cmp
1,00 a co01.00
97 (0L
(.970.562247)
0.96
the 30-33-3
0.99 (0000)))
0.98
0.95)
Zo- 10x 10-100030 10-3p:3
0,98
(.
0.93
15x15-33p-5 3p-5t
0.73 mun
0.73 (03-29)
45.60
15x15 31
0.97 (a (.0))
0.93
0.92:a.1.0000
13x15-4p-57
(.92
0.34
0.80
0.82
simple sprend_3ag
4.81;-4.00...10 -m.))
(9,89 10.16 (10000
.943 -1.31)
11.18 11.30. -10311
-6.21 (1.95
sumple speend 5ag
14 18:44)
229.75 avas 9000)
15,
56.-66611 16-43)
simplle aapeeddee_d__lg
-00.06
-57 64
-2.55
30)
-54.09; 34411P.-31.42--88777
-52.08
25

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
C.2.2. MAX OVER FULL TIMESERIES
Table 3. Maximum episode return over training with 95% bootstrap confidence intervals for all tasks. Bold values indicate the highest
score per task and asterisk indicates that score overlaps with the highest within confidence interval.
Task
Sable (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
Eany-2ag
22,11
17.94
13.49 (3..610))))
uay hard
16.81 20
14.14
niny-dag
50.69 20..0.3..0
tay--ag-hard
33,89
22 (s
22,17
10.57
smalll-dag
17.98
(7.7877.37.427777
19.49 20.28.777
1..9
4 29 st:-at)
smalll-ag--ard
13.2%
10.43 (00 70.
medium--ag
12/5..2,1n
8,78 27..6.44)
311 s0.3.73)
molium-kag-hurd
7.37 (1433,5
0.04g
3.1 Tuuass
06
lurge fag
92
5 5.38gs 11.0090
3.23
large- dag-
5.18 2677)
10 co.0.33)
slarge Aag
4.03 ps
3. 3.98
0.04g (6/0.94)
storge-4ag-hat
0.04
O 02
3,50 20.5-913
large-Rag
22.68 12.222222
16,40 07.0.43)
9.3% 06.1
10.24 10.24 100331
9.84
3. 3.88
4.10gz
hopper, 3xll
2210 28 oo)
1965 98
2045 15
1684-45
2250 56 2405-44)
2423.96
Malres
halffcheetal, fxcl
sets 64)
2017 ar)
yrut 2000 Th)
2790.42
79,797
2e3 11)
3725.47 ast
3,2
1078 64 18,13899,111
1301 76 76:2000.00,000
3658.69 2000 3003
1093 (17
2642: 20 12.2501337
2238 20r.
amt.
3697.15
2622.8 87
38 06 56 sizza
$200.00 05
5454 61 (ooss
5797 08:
humanoid,9--- 9
3795.55,
434. 35
35/42.700,440 477
551 2.,6
$56
6257.01 (280..8..522..0)
59500 4046.16,5412.44
2630
2,00 2,002 on
00.2.00
2,00,, 00)
00.2.003
20000 00
3450
2,00 2,000:200,2 2.0022222220
22 2.06
2,000
2,00,22
00.2..0)
2.00, 2.00 20012
3s. St.
2,00 2,0002.00,2
2,00/2
2. 1.00
2,00,
2,00, 2,00
6h,
2,00 2.0022
2.00 00.2.00
2. .000
2.00, 00.2.00)
1.87
L. 00)
(.72 1103
2,00,, 00.3-003
1.87 000
[tlm.os_.110 11m
1.88 1.7,2.000
1.92
I. 1.49
1,58
2,00
3552. va 3602
2,00
1.99
1.99%
3 1.988
1.91
2.0022200...00 (M)
( 50.3.99
2.000"
1.822,7701 10000
sunnev2.5 units
1.96
1.502 10.3
3.93 19.1
1.00g s0.3900
1.85: 29.1
1mmev2_110_mii
1.70 1.70.
1 R0
1,90, 1,90
18
1.79
sinnev2 units
1,70
1400g
(.69)
1.29 27.3.32)
1,28
Connector
0.89 (.
seuoues)
con-7&7xk
0.85 (8855-5 si.
80
0,8 88 20.
8. esu 0.94)
con- 10c l0s 10s
0.79
14
0. S8
O 664
15xt 5x23a
0,
0,34
0.43 42.0.44)
8x8-2p-2F-coop
2,001
(0.3.00)
00.3.003
2x 23-8x8-21 Sx8-0p-222 20-00op
1,00
1,00 00/1.00
..00 LooL
1,00, co.2.003
1.80
25 10x10
1ts 10-33--5
1.00
1.00-20011
1,00 00.1.00
1.00
1,00
1.00
1.00,,
1,00,
35x15-3p-55
1,00 2.000200110000
10 07.0.99)
0.56 04L-072
15xd1 4p-31
1.00
1.00 00/1043
1.00, 1.00
1.00,, 00.0.0)
15x15-4p-57
1.00
1,00 (0.3.00
1.00;
1.00,
simple spread Sag
4.30
5 3,85 -5.70
96, --47--550
7.35 -6.-0))
4.61 -.61 -450)
4 4.56 Lak,
simple sprend Sag
291
-0 70-37)
-18. 74,
-16.53 18 18.1811119)
simple spread 10ug
-35.32
25.43,
48.12
33..94
-39.55 -
49.39
47.76 13.0
26

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
C.2.3. FINAL VALUE OF THE TIMESERIES
Table 4. Final episode return over training with 95% bootstrap confidence intervals for all tasks. Bold values indicate the highest score per task and an asterisk indicates that score overlaps with the highest score within confidence interval.
Task
Sable (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
tiny 2ng
20.92
17.26 17.26,
y2.01
7.65 (.05..05..
tiny- Zag-hard
16. 20122443)
13 13.77
6,07
tny dag
4366 sn a0.00)
28 28
26.00 54.90 54.94-54
17.70 00090000
tiny lag hard
30.63
19.
small-Aag
17. 17.08
18 (7.0..0000
10. 72..07.11..000
3.80c 20.64-33
small hard
12,47 (k.00)
9,35 1007))
medium
13,05 u2 06.13.79
7,40ps 20, 8.301
2.5%) 20.351
medium-le medium-4fg-horl
6 7.43)
2.
large dag
6,30
4.49,
3.07 c1
1.82
hurge lag hard
3,4% 6.3a
2.37
0.00
0.0Mosoa
slargo-dig
starge 4ag hard
0,74
0.27 (o L103)
0.000p 0un-333
0.000)
medium fag
12,4 3eu anuss"
13.12 15.05.....00
12. 12,54qu
3.6221.71.50
hegg- Sog
10.72.m any
15.08 (429.1000
8.43g
large Sug
9,52 win
9.2 (6.)))))))
3,58 o0u--9
3.43 (QL4.29)
hopper,
2090.31 p222.9922466 29
1891. (1203 11111133
1869
Mo 690
1493 (1 25244.777)
Maßrax
balichettah fxc
23H 03 me0..9.2711
2552.6 26..6
2014 90
2779 89 p9aa.
5272.00 5272.96 26
3633.09 porm
1006 (7000.01118
1128. 18
00,229.00
2506.84,
2244,40,,000
407 477471777)
1564 60, 120...22....
1265 74 1000.32).
32:25222.0 205750)
p000.7, 3002 mg
43.4000.517
4821.98, c2r7.2..
5353 Say
LuTT csms.ss""
hunsancid_ 9
3368 (2000-22,1977 T3)
385 (1700022000000
521 (oum) (5.3313)))
521 544.322747777
50446.19 00316.22,.90
39233 (20 4816.347
2432
2.00
1 1.9% (.6..100
1.99; 2.000
1,99 (999
2.00 2,00, (00
3652
1.99)
2,00
2,00
2,00
1.99
35
1.98
1.99 00:0000
2,00 22 2..0000
2,00: 2000000
199 s 194 ony
1.59
5m, 15. dets
1,25
L 1001-00
1.00
1.89 y sa
1.74g
10m VS 11m
L.84
(57
1.29/)
1.73
1
on 1.171
1.92g 2013.000 1.77
1.68 p.
27th vs. 30m
1.91 shu no0
1,86g in
(1 sou
3.42 30. 160
smapx2. 5 units
1.80
1.7%g Lan)
1.70g 0641-9)
(,73 on 1-03-
1.70 1.70
smacx2, 10 uniis
55% 1.09)
1664
1.77
1.67
1.58
units
1
1 no.
Sxda3a con 7xchida
0,80 mn
0.75 (.75
0,76 90.0.99
0.77 0.77 (00
con 10s Ox 10u
0.73 12.0.75
0.24,.2300.0
0.77
0,56
15x15x23a
4,14 -1.0000000000
BxB- 2F-crop
1..0.00
1.0,1.000000
00.1.00
1.00 L.0.200)
24-8x 202-2-2-oooo
1.00 00.1.20
1.00: 0000000
(.0.2.00
10c10:30 3p-36
1,0%;
a.00
1,0%
1.00
Ze- O0l O0-1-33
1,0%
1,00
0, 2.99
Se 3p-35-
15x15- Ap-36
1.00; 10.0.00
100
0.99 1.00)
0.99 2.99
15x 15-55---55
0.9%
0.97 (156..00)))
0.97 a.97:a
0.97
SIPE
simple speead_
-5.05
4.78
4.85
.8.49
-6.06 -181
simpl..qeeel.. Sag
-12.61,
221000
-22 22-566-2210--2220
221.79-22.27: -2.000
simple.qpendd 10ug
-36.75
449.44 -9.44...uu ou th. 360
-41. 41.00,-11uc (3100-1100
-47.4 -41.40) -9.00
-52.8 -52.6, -1.17.11100
27

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
C.2.4. ABSOLUTE METRIC
Table 5. Absolute episode return over training with 95% bootstrap confidence intervals for all tasks. Bold values indicate the highest score per task and asterisk indicates that overlaps with the highest score within one confidence interval.
Task
Sable (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
Ean
21.17/90.0
17.06
12,28
tay Zag- hard
15.43 s0
13.40
6,15
tiny-dag
24.20 26.27)2)
16.9%
tay-dag-hard
30,97
20.54
19.01 (o
smalll-ag
16.47 ac.0,a0
18.27 02.02.28777
10.
smalll-ag-hard
12.02 26.11222
2.27/3/0x.4.0)
mediumm-agg
12.74
(s 17.0
2. 08.841))
moofum-kag-hard
6.79; 5.79
164,2 6.00))
3,00
I. (.77
lurge fag
6.22: 1o)
4.6 L6 6.93
L84
large dag-
2.224. an
2.28 3.40)
0.05 20000
slarge Aag
3.76 30,5
4.7k on
3.73 no.
0:00:
storge-4ag-hit
0.70 a.70
0159
01.00
6:00)p.0....00
12.97
13.32 ca
(6444) 12.33
3:77:) 49.5
large-Rag
4.87:3.10x.200
9.22 gu332
9.00%
3,58
3.03
hopper, 3xll
2058 29 ory
1901 (2,
1933 p3 eo.2305.04)
1608 52 $2,15 (0l3mm4
2253.3 (224.35/
2459.92,
Mallesx
halffcheetal fxcl
2717.5 po00 seon 28)
un
2912 pm 14,30es 18)
2784.17
51)
s7u30er any"
3739,89 usnse reo
3,2
1051.1 Fess 199.00922228.
1177.
1385.00
2017 17.2006.1
1086. 30
0L3137.33)
2636,95 2azz
2017 14)
2310-41: 2677..4,22553000
amt.
3185.77
2428.0 97
3307 23re
1366 81 (
4751 1.03/423
5069 32.
humanoid,9-- 9-
3449.77
397
76.42.66.40..0 (31)
515.45 520.39;
$12 65
6302.42,
oz)
498
2630
2,0002200.0000 on
2.00)
1.000
2,00, L50.2.00
3450
2,000
2,00
2.000.200022000
2.00 210.2.00
3s. St.
1988,
198
I. 1.99:
1.95,
6h,
2,00 2.0022
1.00)
2,000 L.002.000
1.09...0.20000
1.85 L.A.2.00
(.47 a0)
600g do 1443
1,05 no. 200)
1.35gL GGL
[1lm.os_.110 11m
1.80g 20
1.25
1. 2. 1.00
1.9%, LOOLU.59)
3552. vi 360z
1,94
1.93g
1,500
L889
1.7%
2.000.00...00
ma.3
1.91, (4.1000
sunnev2.5 units
1.72
1.77 17.1.79)
1,700 usc 71)
1.69 oo.
1.66% u0.
1mmev2_10_mmii
1.52
1 6,
1.70 (.003.71)
1. J..4
1.80% L.563.3))
sinner2 20 units
1.47
1,49 46.3.00
1
1
1.3 1.3
Connector
0.85 gustu od)
con-7x7k3
0.79 1P
7. ma. T8)
76.6 71.0
con- 10c l0s 10s
0.74 Auruu
O 66
1.700
0.71 (.71
15xt 5x23a
0, 0.70
25 18.0.333
0,67 0.67: (.00)
268-2p-25-coop
(003.00)
1.00 (.000100))
2x 25-8x8-21 Sx88-p--222 20-00op
1,00
1,00 50.1000
20..1000
1.88
25 10x
1ts 10-33--5
0.99
2000
0.00
0 0.97
1,00
1,00
1,00
33x15-3p-55
0.91 soouas)
15xd1 4p-31
1000
1.00
15x15-4p-57
0.99
0.97
simple.spre spread Sag
4.92
6.59 F6.-444
-.72 (6.66663))
72
8.35 -.81
5.27
6.29(-644-514
simple sprend Sag
-12.5
is 1o.0-1175 ouy
25.
-a -94 19)
84; 73 so mny
-2.97
18 -9
-1.89%
-1735
10
simple spread 10ug
33.93
30.07
41.83 15,
42.08
51.01
-1.01 0433-9990-9000
49.71, -9.39 54
28

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
C.2.5.
MEAN OVER TIMESERIES
Table 6. Inter-quartile mean episode return over training with 95% bootstrap confidence intervals for all tasks, Bold values indicate the highest score per task and an asterisk indicates that a score overlaps with the highest score within one confidence interval.
Tack
Suble (Ours)
MAT
MAPPO
IPPO
MASAC
HASAC
QMIX
liny-2ag
17.68
12.67 14.
3084)
hang-2ag-hund
12667
9,40
4.99 mn
tiny-kag
32.81
25 25.00 01
17 2...s.2a.
tiny-4ag-hand
23.12
14.17/7
15.71 pnusutes
6.53
small dog
10,75
13.30
6.75
2,83g
small 4ag hard
7.64
1.39
Roare
medium- lag
1.28 To ann
medlimm-lag-hunt
3,092.0 A.33)
2,05 oeLT
68 7
largn-dag
4.28 2.52..30
3.91
I. LTALL
lange dag hard
1.27
0.00 0.00uccouu
D.00u
alarge-og
2.24 e0)
langs-dag--und
0.00 Co0 coutaa))
mediant-ag
9.58 1100000.0
9.44,,
6.72
(1
large Sag
9.06 9.06
11.26
5.84
3.18
large Sag hard
6.76
5.
1.27 (l0.a)
1.52 any
hopper_2s1
1460 050.077.......77
p38.......... es;"
1536 110,43333333)
1410.71 62)
1613 LL 07.3052.11"
Malbro
balfchectahh.Sl
2034.55 59 prsss 2294.00)
(88566:2200 63 10,2022 X1)
2493. 2303 29.00447
2030.4 11.1
3376.90 snon sw..o..000
walkerdd.203 2x3
668.60 68
766.66 6 1172-a1
1204 59 (160,42.1400
571 58 p8
1470.59; rsss
1220 26
am,4-2
2087.44 42
5585 95 (4/4 an 2780.00)
21 96 26,244.0022244411
$210 2000
3828. 13 (asun
4150. 155..12.mm
47.747.44007777
hamsnoka 9
2200
390. 25: 200.42)
471 Luuuurramm eN
u00.76o......... my
3255 3255.00 (4.5007.4)
2x3x
2,00 2.000.00))
(.00 61)
20000.co..00
(.99
(
3e5t
1.000,2....00 2.20..000
1.3 96)
1.99 00.2.00)
2.002200,2000
1,96 1.07.1.0
(.97..53.0000
1.75
2,000
1.99 (PP.0.0..00 50)
1999 (.8.1000
1.04
1.62
a.26
1.39 puuss. 45)
1.75 (4,1300)
1,59 aomm
10mm so.1
1.73, 50.3
31 (y 10.00.1 303
1. 191 1o. 169
72k oo 1711
1,50g an aor"""
345 Jofr.
1.81
76
1.52g
1,52
27t1,19, 19, 50hl
1,59
7.55g 800
1.73
1.83g 75. 900
F
1,420 20)
units
1.70
667 66)
1,53g ca: wa)
1,60 ce an
smazv2 units
1.40
149 (.4k
1.59 (.55
1.38 (45---6620000 -20
smacv2 units
1.25 26.1.00)
0.87 (as
con-5x5x5a
0.55
0.75 pu
01.53
0.53 550
con-7673 7k757
0.79
0.72
0.75
1.76
10c101102
0.43
0.52 a0
15235
0.69 m.00
0.38 17/0.00
8x8 1p 28 coop
1.00 1.00..000
1.00 1-00. o0))
1,00 (.0.10000
1.00
2s Rx8-2p-222 1a88-2-2f-ccopp
1.00
19.9 puasup
1,00 L.20LL000
10kc 10o11--30 3p-3
1,00 2100.1 00)
1,000 L000)100)
10x 10-3p--3
0.99 (.2.0.20.
0:95 Oifip.anu s0)
35-55
0.77 (177mmm 7a. my
11 17 160
1.73 m1a
15x15-4p-3C
1.00) 196)
1,00
1.00)
1.00
15x15 4p 54
0.96
0.91
0.58
0.91
simple.spread_3ng
5,80 -88
8.22 -7.0)
100
-5.6,-1.2--.00
5.71
simple.spreadee
15 10-42)
65 aaus0
23:47,22 22 14-200
24 -11---8--
11a5 -51.13
simple..preut_1100 100g
38.81 0-.81116..4.---11 -1.344
-57. 15:-55590--66
-42. 25: 42.00
52; -2.022222
-53 -1.311-500--51000
D. Hyperparameters
We make all hyperparameters as well as instructions for rerunning all benchmarks available along with the code provided the following link: https://site..oogee hotpaeesgooggee.ooo///e///////ae effective vectorised environments. For HASAC and MASAC we use 64 vectorised environments, while for QMIX we use 32 vectorised environments. We leverage the design architecture of Mava which can distribute the end-to-end RL. training loop over multiple devices using the pmap JAX transformation and also vectorise using the vmap JAX transformation. For IPPO, MAPPO and Sable we train systems with and without memory. For IPPO and MAPPO this means that networks include Gated Recurrent Unit (GRU) (Cho, 2014) layer for memory and for Sable this means training over full episode trajectories at time only one timestep at time. For MASAC and HASAC (Liu et al., 2023b) we only train policies using MLPs and for QMIX (Rashid et al., 2020a) we only train system with memory due to the original implementations of these algorithms doing so.
For all on-policy algorithms on all tasks, we always use 128
In cases where systems are trained with and without memory, we report results for the version of the system that performs the best on a given task. In all hyperparameter tables, a parameter marked with an asterisk "*" implics that it is only relevant for the memory version of given algorithm.
D.I. Hyperparameter Optimisation
We use the same default parameters and parameter search spaces for a given algorithm on all tasks. All algorithms are tuned for 40 trials on each task using the Tree-structured Parzen Estimator (TPE) Bayesian optimisation algorithm from the
29

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Table 7. Default hyperparameters for Sable.
Parameter
Value
Activation function
GeLU
Normalise Aavantage
True
Value function coefficient
0.5
Discount y
0.99
GAE A
0.9
Rollout length
128
Add one-hot agent ID
True
Table 8. Default hyperparameters for MAT.
Parameter
Value
Activation function
GeLU
Normalise advantage
True
Value function coefficient
0.5
Discount y
0.99
GAE A
0.9
Rollout length
128
Add onc-hot agent ID
True
Optuna library (Akiba et al., 2019).
D.1.1. DEFAULT PARAMETERS
For all algorithms, we use the default parameters:
1.1.2. SEARCH SPACES
We always discrete search spaces and search the following parameters per algorithm
Table 9.
Default hyperparameters for MAPPO and IPPO.
Parameter
Value
Critic network layer sizes
[128, 128]
Policy network layer sizes
[128, 128]
Number of recurrent layers"
1
Size of recurrent layer"
128
Activation Function
ReLU
Normalise advantage
True
Value function coefficient
0.5
Discount
0.99
GAE
0.9
Rollout length
128
Add one-hot agent ID
True
30

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Table 10. Default hyperparameterss for MASAC and HASAC.
Parameter
Value
Q-network layer sizes
[128, 128]
Policy network layer sizes
[128, 128]
Activation function
ReLU
Replay buffer size
100000
Rollout length
8
Maximum gradient norm
10
Add One-hot Agent ID
True
Table 11. Default hyperparameters for QMIX.
Parameter
Value
Q-network layer sizes
[128, 128]
Number of recurrent layers*
Size of recurrent layer' Activation function Maximum gradient norm
256
ReLU
10
Add one-hot hot agent ID
True
Sample sequence length
20
Hard target update
False
Polyak averaging coefficient r
0.01
Minimum exploration value
0.05
Exploration value decay rate
0.00001
Rollout length
Epochs Add one-hot agent ID
True
Table 12. Hyperparameter Search Space for Sable.
Parameter
Value
PPO epochs
{2, 5, 10, 15}
Number of minibatches
21, 4, 8)
Entropy coefficient
{0.1, 0.01, 0.001, 1}
Clipping
{0.05, 0.1, 0.2]
Maximum gradient norm
(0.5. 10}
Learning rate
{1e-3, 5e-4, 2.5e-4, le-4, le-5)
Model embedding dimension
{32, 64, 128}
Number retention heads
{1, 2, 4]
Number retention blocks
{1, 2, 3}
Retention heads scaling parameter
(0.3, 0.5, 0.8. 1}
Table 13. Hyperparameter Search Space for MAT.
Parameter
Value
PPO epochs
{2, 5, 10, 15}
Number of minibatches
{1, 8}
Entropy coefficient
(0.1, 0.01, 0.001, 1)
Clipping
(0.05, 0.1, 0.2)
Maximum gradient norm Learning rate
(0.5, 10}
{le-3, 5e-4, 2.5e-4, le-4, le-5) (32, 64, 128) {1, 2,
Model embedding dimension
Number transformer heads
Number transformer blocks
(1,2,3)
31

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
Table 14. Hyperparameter Search Space for MAPPO and IPPO.
Parameter
Value
PPO epochs
[2, 4, 8
Number minibatches
{2. 4, 8)
Entropy coefficient
{0, 0.01, 0.00001}
Clipping
{0.05, 0.1, 0.2}
Maximum gradient norm
(0.5, 10} {1e-4, 2.5e-4, 5e-4)
Critic learning rate
Policy learning rate
(1e-4, 2.5e-4, 5e-4)
Recurrent chunk size
{8, 16, 32, 64, 128)
Table 15. Hyperparameter Search Space for MASAC and HASAC.
Parameter
Value
Epochs
{32, 64, 128}
Batch size
(32, 64, 128)
Policy update delay
{1, 2, 4}
Policy learning rate
(1e-3, 3e-4, 5e-4)
Q-network learning rate
{le-3, 3e-4, 5c-4)
Alpha learning rate
(1e-3, 3e-4, 5e-4)
Polyak averaging coefficient
[0.001, 0.005}
Discount factor y
(0.99, 0.95)
Autotune alpha
(True, False)
Target entropy scale
5, 10)
Initial alpha
{0.0005, 0.005, 0.1}
Shuffle agents (HASAC only)
{True, False)
Table 16. Hyperparameter Search Space for QMIX.
Parameter
Value
Batch size
{16, 32, 64, 128
Q-network learning rate
(3e-3, 3e-4, 3e-5, 3e-6}
Replay buffer size
{2000, 4000, 8000)
Target network update period
{100, 200, 400, 800)
Mixer network embedding dimension
{32, 64)
32

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
D.2. Computational resources
Experiments were run using various machines that either had NVIDIA Quadro RTX 4000 (8GB), Tesla V100 (32GB) or A100 (80GB) GPUs as well on TPU v4-8 and v3-8 devices.
E. Sable Implementation Details
E.1. Pseudocode
The following are some useful notes when reading Sable's pseudocode (Algorithm 1).
Bold inputs represent that the item is joint, in other words, it applies all agents. For example: a the joint action and the value of all agents.
The clipped PPO policy objective can be given as:
Lyro (0, A, or, a.) = min (ci(0))..cip((r()))) c)A,) where r0 Or. a.) = xp (a. ot))
(7)
Toodd (arlo:)
The encoder is optimised using the mean squared error:
LMSE (o, v, 0) (vo(or) -,)
(8)
where 0, is the value target computed 6, = rt + yo(o(+1). We always compute the advantage estimate and value targets using generalised advantage estimation (GAE) (Schulman et al., 2015).
We denote de as binary flag indicating whether the current episode has ended or not, with de = 1 signifying episode termination and de indicating continuation.
E.2. Additional Implementation Insights
E.2.1. RETENTIVE RECODER-DECODER ARCHITECTURE
RetNet is decoder-only architecture designed with only causal language modelling in mind. This is illustrated by the assumption in Equations 1, 2 and 3 that the key, query and walue inputs are identical, as they are all represented by the single value x. However, MAT uses an encoder-decoder model to encode observations and decode actions. In order to extend retention to support the cross-retention used decoder, Sable takes a key, query and value as input to a retention block in place of X. Additionally, uses the key proxy for X in the swish gate used in multi-scale retention (Sun et al., 2023) and the query as a proxy for X for the skip connection in the RetNet block (Sun al., 2023).
E.2.2. ADAPTING THE DECAY MATRIX FOR MARL
Since the decay matrix represents the importance of past observations, it is critical to construct it correctly during training, taking into account multiple agents and episode terminations, so that no agent is "favoured" and memory doesn't 't flow over episode boundaries. To achieve this, we make three modifications to the original decay matrix formula.
First, cooperative MARL, a joint action is formed such that all agents act simultaneously from the perspective of the environment. This means the memory of past observations within the same timestep should be weighted equally between agents; therefore, Sable uses equal decay values for these observations.
Second, unlike self-supe learning, RL requires algorithms to handle episode termination. During acting this is trivial for Sable: hidden states must be reset to zero on the first step of every episode as seen in Equations 4 and 5. However, during training, resetting needs to be performed over the full trajectory т in parallel using the decay matrix. If there is termination on timestep las then the decay matrix should be reset from index (Nta, Nt)). Combining these two modifications, we obtain the following equation, given set of terminal timesteps Ta, then Vta E Ta:
o
if if Nta>j Dij
[K1i--1/N1 iii j
Dij = Múj 2 D.j, Mij -
(9)
otherwise
0,
if
33

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
This updated equation makes sure that Sable does prioritise certain agents' past observations because of their arbitrary ordering and ensures that all observations before the terminal timestep are forgotten.
The final decay matrix modification is only required in the encoder to allow for full self- over all agents' observations in single timestep, to match the full self-attention used in MAT's encoder. Since RetNet is decoder-only model where the decay matrix acts as a causal mask, we had to adjust Sable's architecture to be able to perform self-retention. We do this by creating N X N blocks within the decay matrix, where each block represents a timestep of N agents. This leads to the final modification of the decay matrix specifically for the encoder:
[1160111/11,
if Li/N] 2 Li/N] otherwise
Dig = Mij O Dij.
Dij 10,
(10)
(0,
In this case, the floor operator in the first condition creates the blocks that enable full self-retention. Examples of both the encoder and decoder decay matrices used during training can be found in the appendix E.3.3.
E.2.3. POSITIONAL. ENCODING
Positional encodings are crucial for Sable to obtain a notion of time. Previous works in single-agent reinforcement learning (RL) that utilize transformers or similar architectures (Parisotto et al. (2020); Lu et al. (2024)) have demonstrated the importance of positional encoding. Empirical results showed that the best method for Sable is absolute positional encoding, as introduced by Vaswani et al. (2017). In this method, the agent's 's timestep is encoded and added to the key, query, and value during processing. Importantly, we discovered that providing all agents within the same timestep with the same positional encoding is pivotal for performance. When agents were assigned sequential indices (e.g., agent i at step receiving index N & i), is commonly done with tokens in NLP tasks, the performance was significantly worse.
E.2.4. HANDLING CONTINUOUS AND DISCRETE ACTION SPACES
Similar to PPO, Sable can effectively manage both continuous and discrete action spaces, through the use of different network heads depending on the action space. For discrete actions, Sable outputs logits that parameterize a Categorical distribution. In the case of continuous actions, Sable outputs mean values and employs a shared log standard deviation parameter to define Gaussian distribution, from which actions are sampled. This approach aligns with PPO's methodology for handling both types of action spaces.
F.3. Algorithmic Walkthrough: Concrete Example
Sable processes full episodes as sequences, chunking them into segments of defined rollout lengths. During each training phase, processes a chunk and retains hidden states, which are passed forward to subsequent updates. In this section, we provide concrete example to illustrate how the algorithm works in practice.
1. EXAMPLE SETUP
To illustrate Sable's chunkwise processing, consider simple environment with 3 agents, and rollout length (L) of 4 timesteps. The goal of this setup is to demonstrate how Sable processes the execution and training phases of a trajectory starting at timestep I. For this example, we will assume that an episode termination happened at I + 1, which means at the second step of the trajectory.
E.3.2. EXECUTION PHASE
During this phase, Sable interacts with the environment to build the trajectory т. Each timestep t E (l,...(++3) is processed sequentially. At each timestep I, the encoder takes input the observation of all the agents at I, along with the hidden state from the previous timestep, hemç. If = I, indicating the beginning of the current execution phase, the encoder uses h The encoder then computes the current observation representations ô, observation values U,, and updates the hidden state for the next timestep, hene, These observation representations are passed to the decoder alongside 1 If / = I, the decoder initialises with Adlr similar to the encoder. The decoder processes the input recurrently for each agent, generating actions one by one based on the previous agent's 's action. For the first agent, a sfart-of-sequence token is sent to the decoder, signaling that this is the initial agent and prompting the decoder to begin reasoning for its action independently of prior agents. As each agent's s action is decoded, an intermediate hidden state h is updated, and once the decoder iterates over all
34

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
agents, generates he for the timestep l. At the end of each timestep, both the encoder and decoder hidden states decayed by the factor K, reducing the influence of past timesteps. If the episode ends at timestep f (in at I 1), the hidden states are reset to zero; otherwise, they continue to propagate to the next timestep with the decay applied.
E.3.3. TRAINING PHASE
Once the trajectory r is collected, the observations from all agents and timesteps are concatenated to form a full trajectory sequence. this case, the resulting sequence is [o) .... o.+3 [-4] Both encoder and decoder compute retention over this sequence using Equation 6. However, rather than recalculating or initialising Hygeeu the encoder and decoder take as input the hidden states from the final timestep of the previous execution phase, hi and h respectively, Within the retention mechanism used during training, the decay matrix D and & control the decaying and resetting of information.
The decay matrix, D, has shape of (NL, NL). which in this case results in a (12,12) matrix, where each element calculates how much one token retains the information of another token. For example, D5,3 shows how much information of retains from of:
N
0
R
Denc
O
0 0
Ddcc
As shown in this example the decay matrix D, agents within the same timestep share identical decay values, ensuring consistent retention for all agents within given timestep. Additionally, the decay matrix resets for agents once an episode ends. For instance, after the termination at timestep ( + L, subsequent timesteps (D712.1:12) no longer retain information from the prior episode as can be seen from the zeros at D7.12, 1:6- The encoder decay matrix, Dene enables full self-retention over all agents in the same timestep as can be seen through the blocks of equal values within the decay matrix. In contrast, the decoder decay matrix, Ddec: only allows information to flow backwards time, so agents can only view the actions of previous agents they make decisions sequentially.
The second key variable in the retention mechanism is &, which controls how much each token in the sequence retains information from the hidden state of the previous chunk (hi=g for the encoder and hdeg for the decoder). The & matrix represents the contribution of past hidden states to the current timestep, ensuring continuity across chunk boundaries. For this case, & is structured as follows:
As shown, & ensures that after the termination at timestep 1 + 1, the tokens in subsequent timesteps no longer retain information from the hidden states of the prior episode.
35

---

Sable: a Performant, Efficient and Scalable Sequence Model for MARL
E.3.4. HANDLING LONG TIMESTEP SEQUENCES
Consider trajectory batch r' consisting of agents and 512 timesteps. This results in sequence length of 512 x 1536. Given the potential memory limitations of the computational resources, handling such large sequence may pose challenges when applying Equation 6, which assumes the entire sequence is processed input.
To handle these long sequences, we divide the trajectory into i smaller chunks. However, it's essential to maintain the condition that each chunk must be organized by timesteps, meaning all agents from the same timestep must belong to the same chunk. The first chunk will use the hidden state as described earlier (hg=g for the encoder and h for the decoder). For subsequent chunks, when chunking the trajectory into smaller chunks of size 4, the input hidden state is recalculated using the following equation:
h = K/ K/. (V4 O ( + 5k hiprous
D12,1:12
Suppose that a chunk starts at timestep and there is a termination at + 1. In this case, the decay matrix for B would be structured similarly to the one used in the example of Section E.3.3. And given that, Ç will be equal the following:
(0000000'
This structure ensures that only tokens from the current episode contribute to the hidden state, while any information from the previous episode is ignored. Additionally, the term òx H is set to zero, ensuring that once an episode ends, the associated hidden state does not carry over to the next chunk.
36


Reviews:
[{"review_id": "NSoIrmAou1", "reviewer": "Reviewer_pmaV", "summary": "The work proposes a novel sequence model architecture for multi-agent reinforcement learning (MARL) and conducts a large empirical evaluation to validate the efficacy of the new approach. The architecture is based on retention networks and optimises the sequence model architecture similar to the prior multi-agent transformer (MAT) in a central learning fashion. However, in evaluations across 45 tasks, the novel architecture is found to outperform standard MARL baselines and MAT by significant margins, and to be significantly more efficient in terms of memory requirements, model inference speed, and to be more scalable to tasks with many agents. To verify the importance of different components of the proposed approach, ablation studies are provided in two tasks that verify the importance of each novel component.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, especially your comments on our positioning within the context of different MARL algorithms, and questions on implementation and experimental details, which helped us improve the paper. We provide detailed responses below.\n\n> Confusion from the introduction's contrast between Sable and prior MARL approaches.\n\nWe acknowledge that the introduction to Sable can be improved and will make it clear that Sable is a CL method. Our narrative was that Sable breaks the typical CL mold by being both performant, memory efficient and scalable, unlike other CL methods. We will update the introduction to convey this more clearly.\n\n> Include the training objective in Section 3 and add Figure 13 to Section 3 to visualize the network architecture.\n\nDue to the page limit constraint, we moved some additional details and the architecture figure to the appendix. We will add these back into the main text of the updated version.\n\n> How is the Sable network and policy adjusted for continuous vs discrete action spaces, and does the optimization objective differ?\n\nSable's policy network uses different output heads for discrete and continuous actions, but the architecture and PPO optimisation objective remain the same. For discrete actions, the policy head outputs action logits per agent, which are used to sample actions and train the policy. For continuous actions, the policy outputs mean values and a shared log standard deviation parameter, to sample actions from a Gaussian distribution. We will add this and more details to the appendix.\n\n> in what way the retentive network architecture applied in Sable differs from Structured state space models for in-context reinforcement learning?\n\nWhile both approaches share the core idea of replacing attention with a more memory-efficient mechanism to enable scalable sequence processing, they differ in their underlying architecture, and research goals. Sable relies on the cross-retention mechanism which is an extension we added to RetNets. There is no obvious analogue for this in S5. Additionally, the focus of S5 is on single-agent RL, in-context learning and meta-learning, while our work focusses on computationally efficient long context memory in MARL. If interested, we refer the reviewer to Section 2.4 of the RetNet paper [[Sun (2023)](http://bit.ly/4ldsu7i)] where the difference between S4 and RetNets are discussed in detail.\n\n> Why can't the multi-agent transformer architecture use observation history as inputs?\n\nMAT’s architecture lacks a recurrent formulation and handling temporal memory with transformers is challenging [[Parisotto (2019)](https://bit.ly/449KLfB), [Meng (2022)](https://bit.ly/4lbz6TJ)].  Although it is possible to maintain a cache at inference time for memory over the sequence, it is less scalable due to high memory requirements from maintaining a cache. Our RetNet for RL is advantageous in that it only requires a hidden state and constant memory at inference time.         \n\n> Why does IPPO's performance degrade during training in the 128-agent LBF task in Figure 4?\n\n\n(*) Sharing parameters makes distinguishing agents harder, and partial observability leads to non-stationarity. Sable and MAT overcome this with auto-regressive action selection, which aids coordination and non-stationarity. Hyperparameters shouldn't be the issue, as they were tuned. For additional information, we refer the reviewer to Appendix A3, Lines 808-824.\n\n> Would the authors be able to provide information on the training and inference cost of Sable in comparison to MAT, IPPO and MAPPO?\n\nBelow we show training and inference SPS in Neom. \n\n\nTable 1: _Training SPS_\n| Task Name       |   IPPO  |  Sable  |   MAT     |\n|--------------------- |-----------|-----------|-------------|\n| Neom-512-ag   |  ~24k   | 410    |  63      |\n| Neom-128-ag   | ~59k    | 2542| 1505 |\n| Neom-32-ag     | ~180k  | 11111 | 10391 |\n\nTable 2: _Inference SPS_\n| Task Name       |   IPPO  |  Sable  |   MAT     |         \n|--------------------- |-----------|-----------|-------------|\n| Neom-512-ag   |  4600  | 759    |  234    |\n| Neom-128-ag   | 3735  | 1590| 1503 |\n| Neom-32-ag     | 5022  | 3229 | 3198 |\n\n\nSable is significantly faster than MAT but slower than IPPO. This is expected as Sable and MAT both use larger transformer style networks, while IPPO uses a smaller MLP. Additionally, we believe this is a more fair comparison as MAT is also a centralised learning method and the previous SOTA in MARL, while IPPO is independent and has significantly worse performance than Sable.\n\n> Would I be correct in understanding that a smaller chunk size comes at a cost of reduced training speed due to lower degrees of parallelisation?\n\nYes, exactly. Decreasing chunk size reduces training speed. Larger chunk sizes allow more parallel computation and faster training.", "claims_and_evidence": "Overall, I find the claims made in this work to be clearly presented and well supported by clarifications and empirical evidence.\n\nThe only minor point that I found confusing is in the introduction, in which the authors contrast their approach to three categories of MARL: independent learning, centralised training with decentralised execution, and centralised learning. However, as I understand, Sable represents a centralised learning in the same way as the multi-agent transformer is centralised learning. This makes the contrast to the categories somewhat confusing, and I would suggest to clarify the relationship of Sable to such prior work to avoid confusion.", "methods_and_evaluation_criteria": "The methodology appears sound and is largely well presented in Section 3 of the work. However, the work omits several details that are only mentioned in the Appendix and should at least be briefly stated in the main part of the work:\n1. The training objective is only defined in Appendix D and should be included in Section 3.\n2. The network architecture is somewhat hard to follow from Section 3 without visualisation. Figure 13 is very well presented but unfortunately only shown in the Appendix of this work.\n3. Experiments include tasks with continuous and discrete action spaces but the work does not clarify how the Sable network is adapted to adjust for these differences. Would the authors be able to clarify how the policy is adjusted for these settings and whether the optimisation objective differs across these settings?", "theoretical_claims": "The work does not provide any theoretical claims and proofs.", "experimental_designs_or_analyses": "I verified the details provided about the conducted experiments. I find the evaluation to be well presented and very detailed. I commend the authors for following suggestions of recent work on evaluation practises in RL, and for providing plenty of details in the supplementary material.\n\nBelow are some clarification questions and further comparison points that have not been presented in this work and would benefit the contextualisation of this work:\n1. The work compares heavily to the multi-agent transformer approach and states that MAT is \"not able to condition on observation histories\". Would the authors be able to elaborate on this statement? Given the MAT architecture is based on a transformer, it would seem plausible to add longer context based on the observation history of agents, similar to in Sable, even if this has not been done in the original work.\n2. In Figure 4, the performance of IPPO is shown to degrade throughout training for a LBF task with 128 agents which is unexpected. Would it be possible that this degradation is the result of suboptimal hyperparameter tuning? How do the authors explain that performance of IPPO becomes worse as the algorithm continues to train?\n3. The work discusses the achieved returns and memory efficiency of Sable and MARL baselines but does not discuss their training cost. Would the authors be able to provide the cost of training Sable in comparison to MAT, IPPO and MAPPO? Related, Figure 5 (b) shows how the memory cost can be reduced by using smaller chunk sizes without deteriorating performance. Would I be correct in assuming that reduced chunk size comes at a cost of reduced training speed?\n4. Figure 1 compares the throughput of Sable to MAT in terms of steps per second. Would the authors be able to provide a similar comparison to IPPO and MAPPO and clarify how exactly these numbers were obtained?", "relation_to_broader_scientific_literature": "The authors state that their work takes inspiration from recent work in linear recurrent models that considered e.g. the application of state space models in RL [1]. Would the authors be able to elaborate in what way the retentive network architecture applied in Sable differs from this work?\n\n[1] Lu, Chris, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. \"Structured state space models for in-context reinforcement learning.\" Advances in Neural Information Processing Systems 36 (2023): 47016-47031.", "essential_references_not_discussed": "I am not aware of any essential references that are not discussed.", "other_strengths_and_weaknesses": "I would like to commend the authors on a strong empirical evaluation that provides depth and breadth and answers focused questions. I further would like to emphasise that the work releases all experimental data and code.", "other_comments_or_suggestions": "No further suggestions.", "questions_for_authors": "1. Would the authors be able to elaborate why the multi-agent transformer architecture is stated to not be able to use observation history as inputs?\n2. In Figure 4, the performance of IPPO is shown to degrade throughout training for a LBF task with 128 agents which is unexpected. Would the authors be able to elaborate on why this might occur?\n3. Would the authors be able to provide information on the training and inference cost of Sable in comparison to MAT, IPPO and MAPPO? \n4. Figure 5 illustrates how a reduced chunk size can reduce the GPU memory cost without any cost to algorithm performance. Would I be correct in understanding that a smaller chunk size comes at a cost of reduced training speed due to lower degrees of parallelisation?\n\n**In response to the author rebuttal, I increased my score**", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2MdI9DIv7C", "reviewer": "Reviewer_qBU1", "summary": "This paper presents a novel sequence modeling approach for MARL. It adopts the retention mechanism instead of the attention mechanism in MAT to achieve computational efficiency, memory efficiency, and scalability.\n\n## update after rebuttal\nDuring the rebuttal, the authors adequately addressed most of my concerns. Although the problem settings are not clearly presented in the current version of the manuscript, the authors showed their willingness and plan to address this in the modified manuscript. In this regard, I will maintain my score toward acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, especially the close attention paid to our equations/notation. We provide detailed responses below.\n\n>(Q1) Do all baseline methods are trained in via centralized training?\n\nIn addition to answering the above question, we also wish to clarify a misunderstanding evidenced by the following comment:\n\n> “However, to my understanding, some of the baselines utilize partial information (in the default setting), unlike the proposed method or MAT. This information gap makes direct comparisons unfair.”\n\nWhile MAT and Sable process information from all agents using a single network, they only use local observations during training and not the global state, unlike CTDE methods, e.g. MAPPO, QMIX. All methods use the same observations at inference time. We will clarify the definition of CL in the introduction. Not all baselines belong to the CL paradigm; we include baselines from IL, CTDE and CL.\n\n> (Q2) Formal problem setting\n\nIn Section 2, we define the problem setting as a decentralised-POMDP.\n\n> (Q3) Random permutation of agents and performance\n\nWe didn't investigate this, believing it's more principled to randomly permute agent order each timestep. This prevents the model from relying on specific orderings, avoiding bias [[Kuba (2022)](https://bit.ly/4hTYYQU)].\n\n> (Q4) Do QMIX and MAPPO still leverage partial information during decision-making, while Sable and MAT utilize global information?\n\nPlease refer to our answer in Q1.\n\n> (Q5) Dimensions of matrices\n\nPlease refer to our answer to (C1) below and to how we intend to rewrite Equation 6 in our answer to reviewer `wrvH` marked (***).\n\n> outdated literature/baselines on value-based methods.\n\nA well-tuned QMIX has been shown to outperform various extensions [[Hu (2023)](https://bit.ly/3QUdGwx)]. For this reason, we feel that QMIX represents a sufficiently strong value-based baseline. We will also clarify this in the experiments section.\n\n>(C1) In Eq. (3), $\\nu_{ij}$ and $\\zeta_{ij}$ contain index $j$ but $j$ does not appear in their expressions. Perhaps, it could be expressed differently to avoid any confusion.\n\nSince Equation 3 does not have $\\xi$ or $\\nu$, we assume the reviewer is referring to $\\xi$ and $\\zeta$ in Equation 6. We acknowledge that our presentation of Equation 6 was imprecise. This misrepresentation was inadvertently transferred from the original RetNet Paper. Both $\\zeta$ and $\\xi$ are matrices of the same shape as the decay matrix $D$, with dimensions $C \\times C$, where $C$ is the chunk size. These matrices contain values that are constant across columns but vary across rows. We will revise Equation 6 to eliminate the ambiguity caused by the overloaded use of the index $i$, and we will explicitly define the role of $j$ to avoid confusion. We refer the reviewer to our response to reviewer `wrvH` marked (***) for an overview of how we intend to update Equation 6.\n> (C2) In training part, the corresponding loss functions and algorithm presented in Appendix should be mentioned for readers to refer to them.\n\nDue to the page limit, we did not have enough space to add this in our initial submission (moving it to the appendix), but since we have an extra page for the camera-ready version, we will add the loss function back to the main text.\n\n> (C3) Some pictorial illustration would be helpful for readers to understand Neom, a newly introduced task, if possible.\n\nThank you for the suggestion, we will add a render of a step in Neom.\n\n> major contribution is replacing the attention mechanism in MAT with the retention.\n\nIndeed, the reviewer is correct that this is our main contribution. However, we do not see it as a weakness of our work. The use of retention in Sable goes beyond a straightforward replacement of attention in MAT. To get it to work, we had to change several aspects of the original retention mechanism including:\n* Introducing a reset mechanism within the decay matrix to ensure that memory is retained within episodes and not across their termination boundaries. \n* Carefully control the decay over timesteps, which, unlike the original RetNet’s decay that  operates only over token positions, has to handle multiple tokens/observations in each timestep. \n* Developing a cross-retention mechanism, a retentive encoder and an encoder-decoder RetNet, none of which are part of the original RetNet design and are also not  straightforward implementations.\n\nTherefore, Sable as a working retention-based sequence model for RL, is a highly non-trivial algorithmic implementation. This should also be clear when comparing our code with the original implementation of RetNets and/or MAT. Additionally, retention enables Sable to attend over entire trajectories, which is impossible in MAT and is the main reason for Sable's impressive performance. We are excited about what Sable is capable of, with extensive empirical evidence giving such a strong signal for its potential use in applications.", "claims_and_evidence": "In general, yes. The authors elaborate the reasoning mathematically and prove their argument experimentally. (e.g. Memory usage comparison with baseline methods)", "methods_and_evaluation_criteria": "Yes. They compared the proposed methods in various MARL benchmark problems.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, their experiments mostly seem valid. However, to my understanding, some of the baselines utilize partial information (in the default setting), unlike the proposed method or MAT. This information gap makes direct comparisons unfair. An explicit acknowledgment of this information gap (if it exists) may be necessary to avoid misleading readers unfamiliar with the baselines and experiments. If the authors have modified their implementation to address this gap, it should be properly mentioned in the manuscript.", "relation_to_broader_scientific_literature": "The paper presents a novel approach that adopts RetNet (Retentive Networks) for MARL, achieving scalable methods applicable to very large-scale multi-agent tasks, including scenarios with thousands of agents.", "essential_references_not_discussed": "As the paper covers various MARL settings, this version reasonably includes essential literature, although it omits some state-of-the-art (SOTA) algorithms in specific test settings. For example, the paper introduces and compares somewhat outdated literature on value-based methods.", "other_strengths_and_weaknesses": "Strength\n - The paper explored the multi-agent problems in various perspectives, such as IL, CTDE, and CL.\n - The paper conducted extensive experiments to evaluate the proposed model in diverse benchmark problems.\n - The authors open-sourced their code.\n - The proposed methods are applicable to very large-scale multi-agent problems.\n\nWeakness\n - Although the paper evaluated the proposed method in diverse MARL tasks, its major contribution is replacing the attention mechanism in MAT with the retention mechanism.", "other_comments_or_suggestions": "(C1) In Eq. (3), $\\nu_{ij}$ and $\\zeta_{ij}$ contain index $j$ but $j$ does not appear in their expressions. Perhaps, it could be expressed differently to avoid any confusion.\n\n(C2) In training part, the corresponding loss functions and algorithm presented in Appendix should be mentioned for readers to refer to them.\n\n(C3) Some pictorial illustration would be helpful for readers to understand Neom, a newly introduced task, if possible.", "questions_for_authors": "(Q1) Do all baseline methods are trained in via centralized training?\n\n(Q2) The authors mentioned that their approach is classified as centralized training. Then, what is the formal formulation for the main problem setting? DecMDP? MMDP? or what? The proper formulation rather than just general problem formulation for cooperative MARL tasks should be mentioned somewhere in the manuscript.\n\n(Q3) How critical is it for performance to conduct a random permutation of the order of agents within a timestep?\n\n(Q4) Do QMIX and MAPPO still leverage partial information during decision-making, while Sable and MAT utilize global information?\n\n(Q5) It would be helpful for readers to better understand the content if the dimensions of each matrix were explicitly defined somewhere in the manuscript. For example, $\\zeta=D_{NL,1:NL}$ is a confusing. Is it different from $\\zeta=D_{NL,NL}$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pDibRh7gLb", "reviewer": "Reviewer_D22g", "summary": "The paper proposes to replace the attention mechanism in Multi-Agent Transformers with Retentive Networks and shows that such tweak (called Sable method by the paper) leads to improvement in the following three dimensions: strong performance, memory efficiency and scalability. The paper evaluates the Sable method on multiple multi-agent benchmarks to show its performance against the baseline methods spanned from independent learning, centralized training with decentralized execution, to centralized learning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your clarifying questions and feedback. We provide detailed responses below.\n\n> The claim on the scalability may be a bit untenable\n\nRetNets' chunkwise formulation allows us to process long sequences in small chunks, scaling to arbitrarily long sequences regardless of whether they consist of many agents at a single timestep or over multiple timesteps. This scalability difference is evident in Figure 4, where MAT scales much worse than Sable.\n\n> Sable’s details are vague, it is essentially the same as MAT, both are centralized methods, the claim of \"a new sequence model for MARL\" is debatable\n\nWe emphasize that Sable introduces a fundamentally different sequence modeling approach compared to MAT. While both Sable and MAT are CL methods, Sable can reason temporally, which allows it to model sequences of agents over time and capture long-term dependencies, unlike MAT, which is limited to reasoning within a single timestep. \n\nDue to the page limit, we did not have enough space to add all the method details in our initial submission but they can all be found in the appendix. We have an extra page for the camera-ready version and will move these method details into the main text to improve clarity. \n\n> What does b mean\n\nThe b subscript denotes a batch of trajectories from the buffer. We acknowledge that this notation could be made more clear, and we will update Algorithm 1 and the notation accordingly to avoid confusion.\n\n> the decoder only takes the actions as input\n\nWe are only referring to the first block of the decoder, the second block performs cross retention with the output of the first block and the encoded observations as can be seen in Figure 13.\n\n> Sable does not take the trajectories as input\n\nThis statement referred to is specifically to the scaling strategy described in Section 3 (Scaling the number of agents) for handling thousands of agents. It represents a variant of Sable optimized for extremely large agent counts, not the default implementation used in most experiments. The algorithm in Appendix D shows the full version of Sable that conditions on trajectories. \n\n> Sable does not rely on the retention\n\nSable still relies on retention. This statement referred to a limitation when using the agent-chunking scaling strategy described in Section 3. When chunking across a large number of agents, self-retention can only be applied per chunk, which means that agents in different chunks do not “retend” to each other. When the number of agents all fit into a single chunk, it is possible to perform full self-retention across all agents as they all fit within the same chunk. \n\nThis provides a design trade-off to the user when considering the scale of the problem at hand and the memory available. Furthermore, we would like to point out that in Figure 4, we show that even though Sable cannot perform full self-retention across all agents (only per chunk) at large scales, it still achieves the best performance, whereas MAT is not even able to fit into memory at the extreme end.\n\n> unsure what advantage decomposition theorem it refers to\n\n(**) We are referring to the advantage decomposition theorem (ADT) originally derived in [[Kuba (2022)](https://bit.ly/4hTYYQU)] (there called Lemma 1). This theorem underpins the Fundamental Theorem of Heterogeneous-Agent Mirror Learning (HAML) as mentioned on Line 58 Column 2 in the introduction. We will amend the text to make the link between HAML and the ADT clear. \n\n> degradation of IPPO performance looks a bit suspicious\n\nPlease refer to our answer to a similar question from reviewer `pmav` marked (*).\n\n> memory usage, is it in the training or in the inference?\n\nThe memory usage reported in Figure 5 is measured during training as this is when the bulk of the memory is used.\n\n> On the usage of memory efficient transformers\n\nMemory efficient transformers without a dual form do not maintain a hidden state, thus would need to keep a cache of per-agent/timestep observations during inference. The two most important aspects of retention in Sable for scaling is low memory requirements and the dual form for efficient inference. For a further discussion on the differences between RetNets and Transformers, we refer the reviewer to Section 2.4 in [[Sun (2023)](http://bit.ly/4ldsu7i)]. We are not against adding some of these related works into the text if the reviewer thinks it would be useful. \n\n> should be gradient descent for ϕ\n\nThank you for pointing this out, we will update it.\n\n> final configs used to produce the reported results?\n\nThe final configs, source code, and raw experimental data are available at the link at the end of Section 3. You can download the data by pressing \"Experiment Data\" at the top of the page and find optimal hyperparameters at All experiments data/Benchmark/optimal-hyperparams/… to reproduce our results. We chose not to add the configs directly to the appendix as it would add a large number of extra pages.", "claims_and_evidence": "* The claim on the scalability may be a bit untenable. My understanding of RetNet is that it was proposed to reduce the memory cost in inference time, assuming that the model is well trained and everything else would remain the same as the transformers. So, in this sense, **Sable is essentially the same as MAT and should be a centralized method**. So why Centralized Learning is deemed as not memory efficient or scalable. But Sable, as one of such kind, is scalable? Further, if the memory constraint is the main issue in MAT when scaling up to large numbers of agents, we  can then resort to some memory-efficient optimizations of transformers (E.g., SGD instead of Adam) or memory-efficient transformers? Can the authors explain why RetNet is required specifically here? \n\n* The claim of “a new sequence model for MARL” is also a bit debatable. The Sable is no different than MAT in the sequence modelling perspective. They are both centralized methods that take the whole sequence from all agents as input and output the joint actions autoregressively.", "methods_and_evaluation_criteria": "**The method part is rather vague in general and there is not much info on this in the main text**. The paper, possibly intentionally, puts the method details (implementation details) into the appendix, which I guess might imply that the algo details of Sable are pretty like MAT: they both use the PPO-like training for the actor and critic updates, as specified in Algorithm 1, and the observation sequence encodings from the encoder are fed into the decoder to produce actions. \n1. However, it’s unclear what $o_b$ is. What does $b$ mean here? \n2. In the main text, “The decoder takes a similar sequence but of actions instead of observations as input”, which implies the decoder only takes the actions as input (different from the algorithm). \n3. Again, “we use MAT-style single-timestep sequences to optimise memory usage and reserve chunking to be applied across agents.”, which implies that Sable does not take the trajectories as input (different from algorithm).  \n4. Furthermore, “this change to the encoder makes it unable to perform full self-retention across agents, as it cannot be applied across chunks” what does it mean here? The Sable does not rely on the retention then?", "theoretical_claims": "There are no theoretical proofs and claims in this paper. \n\nThe paper mentions \"It is this autoregressive action selection which leverages the advantage decomposition theorem to give Sable theoretically grounded convergence guarantees.\" But **there is no such analysis throughout the paper, including appendix**. It's even **unsure what advantage decomposition theorem it refers to (no references provided)**.", "experimental_designs_or_analyses": "In section 4.2, the degradation of IPPO performance looks a bit suspicious: any reasons why it happened? Can it be addressed by normalizing the returns or learning rate decay? Regarding the memory usage, is it in the training or in the inference?", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "there are quite a lot of papers on reducing the memory cost in training transformers, e.g., *Memory-efficient Transformers via Top-k Attention*, *Memory Efficient Continual Learning with Transformers*, and in transformer optimizations, e.g., *ZeRO: Memory Optimizations Toward Training Trillion Parameter Models*, *Full Parameter Fine-tuning for Large Language Models with Limited Resources*. The paper should discuss how these are related and should consider some of them as the baseline improvements for transformer architectures used in MAT", "other_strengths_and_weaknesses": "* Strengths: the empirical results are promising\n\n* Weakness: the paper in its current version does not present the method in a clear and convincing way; the paper also misses quite a large chunk of related work on memory-efficient transformers and the optimizations.", "other_comments_or_suggestions": "There is an error in Line 20 in algorithm 1: should be gradient descent for $\\phi$.", "questions_for_authors": "The paper provides the hyperparameter search space for Sable, MAT etc. but what are the final configs used to produce the reported results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "n/a"}}, {"review_id": "l2ga3HTKma", "reviewer": "Reviewer_wrvH", "summary": "This paper proposes to use retentive networks to process multiple agents' observations and actions in MARL. The proposed framework can scale to a large number of agents. Extensive experiments and analysis are conducted. As a result, the proposed framework shows performance improvements in 34/45 of tasks and also achieves memory efficiency compared to MAT and IPPO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback. Your comments on the retention equations and other aspects of our work have helped us identify areas for improvement. We address your questions and comments below.\n\n>wondering if agents share parameteres, espicially on Q, V, K matrices\n\nSable uses a single network for all agents. The QVK matrices are not unique per agent, but shared in the sense that each agent is treated as another element in the sequence. While longer sequences incur higher memory usage, RetNets address this by constraining computational memory to a fixed chunk size.\n\n>the effect when the chunk size and the number of agents increase jointly\n\nThe chunk size can be increased as the number of agents increases, but this will affect memory requirements. The optimal setting is to process the entire training sequence at once, but if this isn't feasible, Sable allows the chunk size to be tuned to maximize hardware utilization and enable training on arbitrarily large sequences.\n\n>which decomposition theorem are you referring to\n\nPlease see our reply to D22g marked (**).\n\n>agents' actions are dependent in a order. How would \\hat{h}_0 be?\n\nAs shown in Line 177 Column 2, $\\hat{h}\\_{0}$ is $h\\_{t-1}^{dec}$, where $h_{t-1}^{dec}$ is the decayed hidden state from the previous timestep. $\\hat{h}$ is an intermediary variable that accumulates the hidden state over agents within a single timestep. This is decayed once per timestep to produce  $h_{t}^{dec}$. Regarding the agents’ actions order dependency, we mitigate any potential bias by shuffling agent order during training.\n\n> it is unclear about the \\tau and \\tau_prev\n> using \\tau to refer to a chunk can be confused\n\n(***) Thank you for going through our paper in such detail, we will make the following changes to the paper to help with the clarity of our method. \n\nFor the chunkwise representation, we split a trajectory $\\tau$ consisting of $L$ timesteps, with $N$ agents gathered during inference into smaller chunks each of length $C$ such that the retention for chunk $i$ can be given as: \n\n$\\begin{equation}\\begin{aligned}Q_{[ \\tau_{i} ]} &= Q_{C(i-1):Ci}, K_{[\\tau_i]} = K_{C(i-1):Ci}, V_{[\\tau_i]} = V_{C(i-1):Ci} \\\\\\\\h_i &= K^T_{[\\tau_i]} \\left( V_{[\\tau_i]} \\odot \\zeta \\right) + \\delta \\kappa^{ \\lfloor L/C \\rfloor } h_{i-1}, \\zeta = D_{N \\cdot \\lfloor L/C \\rfloor, 1:N \\cdot \\lfloor L/C \\rfloor} \\\\\\\\\\text{Ret}(\\boldsymbol x_{[\\tau_i]}) &= \\left( Q_{[\\tau_i]} K^T_{[\\tau_i]} \\odot D \\right) V_{[\\tau_i]} + \\left( Q_{[\\tau_i]} h_i \\right) \\odot \\xi \\\\\\\\\\text{where } \\ & \\xi_{j} = \\begin{cases} \\kappa^{\\left\\lfloor j / N \\right\\rfloor + 1}, &  \\text{if } j\\leq Nt_{d_0}  \\\\\\\\\n0, &   \\text{if } j > Nt_{d_0} \\\\\\\\\\end{cases}.\\end{aligned}\\end{equation}$\n\nHere $h$ is an intermediary variable carrying information from one chunk to the next, $h_0$ is the hidden state at the beginning of $\\tau$ that will be used for training, $\\zeta$ is the last row of the decay matrix that is created from the data of the chunked trajectory for chunk $i$ and $\\xi$ is a duplicated column vector. Please see our answer to (C1) of reviewer `qBu1`. \n\nThis removes the confusing $\\tau$ and $\\tau_{prev}$ notation from the text and should link Equations 3 and 6 more clearly. It also removes $i$ from the definition of $\\xi$ giving clarity around $Nt_{d_0}$, as $j$ is now an index in $\\xi$.\n\n>it is unclear whether you refer to a chunk as a set of agents or a set of observations over time step\n\nSable's flexibility allows for treating either entire rollouts with multiple agents at each timestep, or just the number of agents, as the training sequence length. With E environments, L timesteps, N agents, and C chunks, the default training batch shape is $(E, NL)$, divisible into $(E, N [L / C] )$ size chunks. When using only the number of agents as the sequence length, the shape is $(EL, N)$, divisible into $(EL, N/C)$ size chunks.\n\n>in section 2 you are using continuing tasks while in equation 4 you consider epsiodes\n\nAll tasks we consider have fixed length time horizons and termination conditions and we allow for environments to automatically reset once an episode terminates. Thus, for a fixed rollout length it is possible for there to be multiple terminal timesteps. \n\n>not clear how's the decay matrix updated according to your equation\n\nIn our case, the decay matrix is blockwise lower diagonal of size  $(LN, LN)$ with block size $(N, N)$ where $N$ is the number of agents. Each element is exponentially decayed given its position in time for a given trajectory which follows Equation 2. We discuss our adaptations to the decay matrix after Equation 6 in Lines 213-240 and give an example in Appendix D.\n\n>Would it be possible to use a smaller chunk size, e..g, 8, for all tasks?\n\nThis is possible, but not practical. Smaller chunks during training use less memory at the cost of wall clock time. In practice, we train with a chunk size that is as large as our computational memory permits for the fastest training wall clock time.", "claims_and_evidence": "Regarding the memory used, I'm wondering if agents share parameteres, espicially on Q, V, K matrices, which may lead to linearly increased memory with respect to agents and time steps.", "methods_and_evaluation_criteria": "Equation 6 is unclear and confusing. Please read Questions For Authors", "theoretical_claims": "No theoretical claims provided", "experimental_designs_or_analyses": "The experiments are cndense. I'm wondering what would be the effect when the chunk size and the number of agents increase jointly.", "relation_to_broader_scientific_literature": "Due to the auto-regressive way used in generating actions, the paper may relate to extensive-form games or MARL which considers communication among agents.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "This paper provides extensive details about the algorithms, implementatio, and experiments with figures and tables.", "other_comments_or_suggestions": "- it is a bit confusing to concurrently use subscript for agent index and time step. besides, it is confusing to use i refers to both agent index and a chunk.", "questions_for_authors": "- In euqation 5, do you share K dan V for different agents and different time steps?\n- in page 4: which decomposition theorem are you referring to? In the auto-regressive way, agents' actions are dependent in a order. How would \\hat{h}_0 be?\n- it is unclear about the \\tau and \\tau_prev. The author needs to explain how does \\tau and \\tau_prev differ from the subscript i used in Equation 3. \n- it is unclear whether you refer to a chunk as a set of agents or a set of observations over time step. In line 205, you are talking about processing agents in parallel. However, in equation 6, L is actually analogous to B in euqation 3 so a chunk is a set of observations over time step. Based on this, using \\tau to refer to a chunk can be confused since \\tau refers to a trajectory. Moreover, in equation 6, i <= Nt_do is also confusing since i is agent index so only when t_do=0 you will get i > Nt_do. And, I'm not sure what j stands for. Is it an agent or a time step?\n- it is confusing that in section 2 you are using continuing tasks while in equation 4 you consider epsiodes. If you are using episodic tasks, Equation 6 will never achieve a second terminal time step (therefore t_do is the terminal time step rather than the first terminal timestep).\n- it is not clear how's the decay matrix updated according to your equation (where equation 3 explicitly incorporate exponentially updates)\n- since the GPU memories with chunk size 128 is below 3.3 and the performance of Sable seems to be insensitive to the size of chunk. Would it be possible to use a smaller chunk size, e..g, 8, for all tasks? What if the number of agents increase as well?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors adapt a RetNet architecture to the cooperative MARL setting to produce a scalable (high throughput, low memory cost) centralized learning method that beats SOTA by significant margins on a wide range of domains. The discussion period raised several constructive comments. As the authors have agreed in their rebuttal, they should carry on with the improvements suggested by the reviewers (refine the problem formulation, pull in algorithmic details from the appendix, etc).
Decision: Accept (poster)