Performant, Memory Efficient and Scalable Multi-Agent Reinforcement Learning

Authors: Omayma Mahjoub, Sasha Abramowitz, Ruan John de Kock, Wiem Khlifi, Simon Verster Du Toit, Jemma Daniel, Louay Ben Nessir, Juan Claude Formanek, Louise Beyers, Liam Clark, Arnu Pretorius
Primary area: reinforcement learning
Keywords: Multi-agent reinforcement learning, Reinforcement Learning, Decision Making, Multi-agent systems

Abstract:
As the field of multi-agent reinforcement learning (MARL) progresses towards larger and more complex environments, achieving strong performance while maintaining memory efficiency and scalability to many agents becomes increasingly important. Although recent research has led to several advanced algorithms, to date, none fully address all of these key properties simultaneously. In this work, we introduce Sable, a novel and theoretically sound algorithm that adapts the retention mechanism from Retentive Networks to MARL. Sable's retention-based sequence modelling architecture allows for computationally efficient scaling to a large number of agents, as well as maintaining a long temporal context, making it well-suited for large-scale partially observable environments. Through extensive evaluations across six diverse environments, we demonstrate how Sable is able to significantly outperform existing state-of-the-art methods in the majority of tasks (34 out of 45, roughly 75\%). Furthermore, Sable demonstrates stable performance as we scale the number of agents, handling environments with more than a thousand agents while exhibiting a linear increase in memory usage. Finally, we conduct ablation studies to isolate the source of Sable's performance gains and confirm its efficient computational memory usage. Our results highlight Sable's performance and efficiency, positioning it as a leading approach to MARL at scale.

Decision: reject
Reviewer scores: 3, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729657252134

thread_type:
official_review

<thread_object>
creator: reviewer_7eua

<content>
summary:
This paper introduces a novel and theoretically sound algorithm Sable, adapting the retention mechanism from retentive networks to MARL. The authors demonstrate that Sable ranks first in 34 out of 45 tasks (roughly 75%), outperforming SOTA MARL algorithms across 6 environments: RWARE, LBF, MABrax, SMAX, Connector and MPE. Meanwhile, the advancements of Sable in memory efficiency and agent scalability are significant.

strengths:
1. The motivation is well justified, the method is reasonable.
2. The problems are practical and relevant to the large-scale MARL community.
3. Well-written paper with comprehensive results reported.
4. Authors provide several experimental details and corresponding code on their proposed algorithm.

weaknesses:
While I lean towards acceptance, I have a few concerns. I am willing to raise my score if the authors can resolve my concerns and questions.
1. The novelty of Sable should be further clarified, stating how it builds upon, differs from, and improves upon existing methods, and so on.
2. As stated in the abstract, “In this work, we introduce Sable, a novel and theoretically sound algorithm that adapts the retention mechanism from Retentive Networks to MARL.” Where is the theoretical convergence guarantee analysis in the main text?

questions:
I have asked my questions in the Weaknesses section. I have no further questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472533651

thread_type:
official_review

<thread_object>
creator: reviewer_btd1

<content>
summary:
This paper begins by analyzing the broad categorization of current MARL approaches and addresses the issues of high computational and memory resource costs and the lack of capability to capture historical information in MAT. Inspired by Retention, the paper adapts the Retention architecture for MARL, introducing the Sable method. This method maintains high computational efficiency and low memory consumption while supporting a large number of agents, and it leverages historical knowledge to improve policy learning. The paper concludes with a comparison of SOTA models in the MARL domain across multiple environments, achieving excellent results that thoroughly validate the algorithm's performance. Overall, this paper mitigates the problems inherent in traditional MAT methods and proposes a new, efficient MARL architecture, providing a fresh direction and approach for research in this field.

strengths:
1. The introduction succinctly outlines the general classification of MARL methods and their limitations, while also identifying the issues with current SOTA methods. This leads smoothly into the presentation of the paper's method and its advantages, ensuring a coherent flow and quick understanding of the current state of MARL and the paper's contributions.
2. Building on the shortcomings of previous MAT methods, the paper innovatively proposes the Sable method by adapting the Retention architecture to RL algorithms. This adaptation results in a method with lower computational and memory costs, scalability to a large number of agents, and the ability to incorporate historical observations for better policy learning. The method also provides new research directions and theoretical foundations for the MARL domain.
3. The paper conducts experiments in several classic and challenging environments, comparing multiple SOTA models in the MARL domain. The extensive experimental results demonstrate the effectiveness of the algorithm, and additional experiments with varying numbers of agents confirm its computational and memory efficiency.

weaknesses:
1. The Related Work section lacks a thorough introduction to some classic MARL algorithms, which would help readers unfamiliar with the field understand the broader development and methodologies in MARL.
2. While the Introduction directly addresses the classification of MARL methods and their current limitations, it could benefit from more foundational background information to better contextualize the motivation and overall method design for readers.

questions:
1. Can the method adapt to changes in the number of agents or significant variations in observation while maintaining the robustness of the overall policy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624526827

thread_type:
official_review

<thread_object>
creator: reviewer_3tra

<content>
summary:
The paper introduces Sable, a novel algorithm for multi-agent reinforcement learning (MARL) that aims to achieve strong performance, memory efficiency, and scalability. Sable adapts the retention mechanism from Retentive Networks to MARL, allowing for computational efficiency and the ability to maintain long temporal contexts. The authors demonstrate that Sable outperforms several state-of-the-art methods across diverse environments.

strengths:
1. This paper introduces a novel approach by adapting retention mechanisms for multi-agent reinforcement learning (MARL), representing a creative solution to enhance memory efficiency and scalability as the number of agents increases. The experimental results are particularly impressive.

2. The study includes a comprehensive set of experiments encompassing six commonly used MARL environments and a total of 45 tasks. The results demonstrate robust performance, indicating that Sable significantly outperforms existing methodologies in 34 out of 45 tasks. Furthermore, the approach exhibits stability with over a thousand agents, which is a critical consideration for large-scale applications.

weaknesses:
The related work section is insufficiently comprehensive, primarily focusing on recurrent nets and Transformers. It would be beneficial to include more detailed discussions on specific algorithms in multi-agent reinforcement learning (MARL). In the MABrax environment, the performance of Sable is significantly worse compared to other environments. A more in-depth analysis and discussion of the reasons for this disparity would be highly valuable.

questions:
1. Table 1 is too small, affecting the overall aesthetics and clarity of the presentation. Could the structure be adjusted or some content trimmed to enhance its visual appeal and readability?

2. The ablation study fails to pinpoint the main design elements that influence Sable's performance. Although the modified MAT shows some improvement, it still falls short compared to Sable. Further analysis and experimentation would likely be beneficial for the community to effectively reuse these insights.

3. In Figures 4b, 4c, and 4d, the final reward continues to increase as the number of agents increases. I would like to understand the reason behind this because usually the performance will decrease as the number of agents increasing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650100523

thread_type:
official_review

<thread_object>
creator: reviewer_rdrb

<content>
summary:
The paper presents Sable, an approach based on a modified Retentive Network (RetNet) architecture that leverages attention mechanisms with a retention-based memory management approach. It is shown to improve the memory efficiency of MARL and scale well to large number of agents.

strengths:
+ The proposed Stable considers MARL as a sequence modeling problem and leverages Retentive Networks. It is able to maintaining a long temporal context and leads to efficient memory usage.

weaknesses:
While I like the idea presented in this paper, there are many issues with the experiments and evaluation.
- Formulating MARL as a sequence modeling problem has been an active area of research. The authors have cited (Wen et al., 2022) and  (Kuba et al., 2021). But since IPPO was considered as a main baseline, why not also consider baselines like independent decision transformers and other sequence modeling approaches? Further, there are also many decomposition based MARL algorithms that outperform QMIX (e.g., weighted-QMIX, FOP, DOP, PAC) and mean-field based MARL algorithms. These should not be ignored in the evaluation. 
- In Figure 1 - Middle, while the authors claim that Stable is more throughput efficient, but it is only 6.5x, compared to 45x by IPPO.
- In Figure 1 - Right, IPPO is 9x but the bar is much lower than Stable that is 7x. 
- In Figure 4, IPPO seems to consistently have lower memory usage than Stable. Doesn't it conflict with the main claim of the paper?
- In Table 1,  MASAC, HASAC, and QMIX have many missing return values. It is hard to believe these algorithms don't work in these environments.
- The comparison with baseline algorithms need to cover more complex MARL tasks with continuous action/state spaces, like Starcraft and MuJoCo, which are used by many of the baselines.
- Table 1 summarizes the return comparison with baselines. Since the main advantage of Stable is memory efficiency, a similar table on memory comparison should be provided. 

Further, the authors claim that Sable provides "theoretically grounded convergence guarantees". But I was not able to find any analysis or proofs in the paper. It simply cited (Kuba et al., 2021). It is unclear how it applies here.

questions:
Please address my questions in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731961144939

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author's reply to all Reviewers - Part 1

comment:
**We thank all the reviewers for their time and for reviewing our work.** We will respond to each reviewer’s questions individually to address their concerns and give suggestions for changes based on their comments. Once any updates have been made, we will specifically highlight these sections in revised versions of the paper. For overall comprehension and to make the discussion easy to initiate, **we wish to summarise positives** on what reviewers had to say about the work and **point out shared concerns that we aim to address**: 

## Shared praise

Reviewer `3TrA` writes that the introduced algorithm represents

>… a creative solution to enhance memory efficiency and scalability as the number of agents increases. 

that

> The experimental results are particularly impressive. 

and finally that the paper’s results 

>… demonstrate robust performance, indicating that Sable significantly outperforms existing methodologies in 34 out of 45 tasks. Furthermore, the approach exhibits stability with over a thousand agents, which is a critical consideration for large-scale applications.

Reviewer `BtD1` gives the work the highest possible scores for Soundness, Presentation and Contribution (4 for all) and says that the paper is written in a way that ensures a 

> …coherent flow and quick understanding of the current state of MARL and the paper's contributions

that

> …the paper innovatively proposes the Sable method by adapting the Retention architecture to RL algorithms. This adaptation results in a method with lower computational and memory costs, scalability to a large number of agents, and the ability to incorporate historical observations for better policy learning. 

and that 

> … extensive experimental results demonstrate the effectiveness of the algorithm, and additional experiments with varying numbers of agents confirm its computational and memory efficiency.

Reviewer `7eua` writes that 

> … the advancements of Sable in memory efficiency and agent scalability are significant

that the problems addressed in the paper 

> …are practical and relevant to the large-scale MARL community.

and that the work is a

> Well-written paper with comprehensive results reported

## Shared concerns

As for concerns, we wish to highlight the following points that were raised by more than one reviewer. 

**1. Missing theoretical convergence guarantees analysis**

Reviewer `rdRb` writes that

>…the authors claim that Sable provides "theoretically grounded convergence guarantees". But I was not able to find any analysis or proofs in the paper. It simply cited (Kuba et al., 2021). It is unclear how it applies here.

Reviewer `7eua` asks

>…Where is the theoretical convergence guarantee analysis in the main text?

We thank the reviewers for raising this issue, as admittedly it was not clearly outlined within the main text. 

------------------------------------------------------------------------------------------------------------------------------------------------

**Updated in revised paper**: we have added a detailed clarification of this point in the main text of the revised version of our paper which has been uploaded. We hope this will address the reviewer's concerns and we remain open to any additional feedback that could improve our manuscript.

------------------------------------------------------------------------------------------------------------------------------------------------

*(Older reply): Our claim relies on previous theoretical results proved in a series of recent papers. First, is the advantage decomposition theorem/lemma [kuba2021, Lemma 1], as highlighted in our paper. This result was subsequently used to develop trust region learning approaches for MARL with monotonic performance improvement guarantees [kuba2022a, Theorem 2]. Even though PPO-style algorithms with autoregressive updates do not strictly adhere to trust region learning theory, and therefore, do not have strict monotonic improvement, more recent work has placed PPO within a class of mirror learning algorithms that indeed enjoy monotonic improvement and theoretical convergence guarantees [kuba2022b, Theorem 3.6]. This work has since been extended to the multi-agent setting [kuba2022c, Theorem 1], and in particular, HAPPO (multi-agent PPO with heterogeneous autoregressive updates), was shown to be an instance of multi-agent mirror learning, and therefore theoretically sound. To obtain an instance of mirror learning requires defining a valid drift functional, neighbourhood operator and sampling distribution. In both MAT and Sable, these design choices are exactly as they are for HAPPO, and therefore, we claim that Sable inherits the same theoretical monotonic improvement and convergence guarantees as HAPPO, and by extension MAT.*

We will make sure to update our paper to significantly improve the text around these details, as well as recapitulate certain core theoretical results and derivations supporting our claim in the appendix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732261821016

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New revised version of our paper addressing main concerns

comment:
Dear reviewers, we want to thank you all once again for your time reviewing our work and for providing constructive feedback to improve our manuscript. **Based on your feedback and the main concerns raised, we have revised and uploaded a new version of our paper. We made the following changes**:

* We have significantly expanded the introduction with additional context, related work and more clearly explaining Sable's position and novelty within the existing literature. *[Addressing concerns raised by `rdRb` and `7eua`]*
* We have added detailed clarification in the main text of how Sable has theoretical monotonic improvement and convergence guarantees. *[Addressing concerns raised by `3TrA `, `BtD1 ` and `7eua`]. Note: We will still add additional theoretical derivations (recapitulating existing results) in the appendix to further clarify this point.* 

We also added the following to address specific concerns raised by `3TrA `:
* We expanded on and added references supporting why Sable performs worse than SAC-based algorithms on MaBrax.
* We added "Sable without memory" to the ablation study and significantly improved the discussion around the reason for Sable's impressive performance. We now fully isolate that the reason for Sable's performance is the way it structures the sequence enabling its long-term memory.
* We updated Table 1 to only have per-environment aggregations for all algorithms to make the table more readable and to improve the paper's overall aesthetics. We include the per-task tables in the Appendix and refer the reader to them in the main text for more results.
* We updated Figures 4b, 4c, and 4d to have constant agent-to-gird ratios for all tasks leading to the maximum reward per task decreasing as the number of agents increases.

We hope to have addressed all the reviewer's main concerns and that reviewers find our revised paper a satisfactory improvement. We remain open to any additional feedback that could improve our manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Sable, a MARL algorithm that adapts retention mechanisms from Retentive Networks aiming to achieve memory efficiency and scalability. While the reviewers acknowledged the potential value of improving MARL performance with limited memory usage, significant concerns were raised about both theoretical and empirical aspects of the work. The paper claims convergence guarantees but lacks rigorous proofs, merely citing previous results without establishing how they extend to Sable's specific setting. The novelty needs better positioning as sequence modeling approaches and RetNet adaptations have been explored in MARL before. The experimental results show concerning inconsistencies - the claimed memory efficiency advantages appear contradicted by figures showing IPPO with lower memory usage, and performance on MaBrax environments notably lags behind baselines. Although the authors provided detailed responses and additional experiments, these core issues around theoretical soundness, technical novelty, and empirical validation remain insufficiently addressed. 

Given these substantial limitations in the current manuscript, I recommend rejection.