Multi-Agent Decision S4: Leveraging State Space Models for Offline Multi-Agent Reinforcement Learning

Authors: Ashmita Bhattacharya, Malyaban Bal
Primary area: reinforcement learning
Keywords: offline multi-agent reinforcement learning, sequence-based offline reinforcement learning

Abstract:
Goal-conditioned sequence-based supervised learning with transformers has shown promise in offline reinforcement learning (RL) for single-agent settings. However, extending these methods to offline multi-agent RL (MARL) remains challenging. Existing transformer-based MARL approaches either train agents independently, neglecting multi-agent system dynamics, or rely on centralized transformer models, which face scalability issues. Moreover, transformers inherently struggle with long-term dependencies and computational efficiency. Building on the recent success of Structured State Space Sequence (S4) models, known for their parameter efficiency, faster inference, and superior handling of long context lengths, we propose a novel application of S4-based models to offline MARL tasks. Our method utilizes S4's efficient convolutional view for offline training and its recurrent dynamics for fast on-policy fine-tuning. To foster scalable cooperation between agents, we sequentially expand the decision-making process, allowing agents to act one after another at each time step. This design promotes bi-directional cooperation, enabling agents to share information via their S4 latent states or memory with minimal communication. Gradients also flow backward through this shared information, linking the current agent's learning to its predecessor. Experiments on challenging MARL benchmarks, including Multi-Robot Warehouse (RWARE) and StarCraft Multi-Agent Challenge (SMAC), demonstrate that our approach significantly outperforms state-of-the-art offline RL and transformer-based MARL baselines across most tasks.

Decision: reject
Reviewer scores: 3, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1729675633660

thread_type:
official_review

<thread_object>
creator: reviewer_ntod

<content>
summary:
The paper presents Multi-Agent Decision S4 (MADS4), an innovative approach to offline multi-agent reinforcement learning (MARL). The authors build on the success of Structured State Space Sequence (S4) models in single-agent tasks and extend them to MARL by leveraging their parameter efficiency, long-context handling capabilities, and faster inference compared to transformers. The paper introduces a Sequentially Expanded Markov Decision Process (SE-MDP) framework that promotes inter-agent cooperation by enabling sequential action-taking with limited communication between agents. The results on benchmark environments (Multi-Robot Warehouse and StarCraft2) demonstrate the superior performance of MADS4 over state-of-the-art baselines in offline MARL settings.

strengths:
1) The use of S4 models provides clear benefits in terms of efficiency, handling longer temporal contexts, and faster inference times. This is a critical advantage over transformers, especially for offline reinforcement learning tasks that involve long trajectories.
2) The paper provides empirical results showing that MADS4 outperforms state-of-the-art methods on RWARE and SMAC, highlighting its potential for cooperative multi-agent tasks that require coordination among agents.
3) Offline reinforcement learning in MARL is a challenging but important area, especially when online interaction with the environment is costly or risky. The focus on leveraging pre-collected data is a strength of this work, aligning it with real-world applications.

weaknesses:
1) The choice of environments (RWARE and SMAC) does not sufficiently test the generalizability of MADS4. These benchmarks are widely used but do not fully capture the complexity of real-world multi-agent tasks. Testing in environments with higher-dimensional state spaces or dynamic agent interactions (e.g., more heterogeneous teams or real-time strategy games) would provide stronger evidence of scalability and robustness.
2) The paper mainly compares MADS4 to transformer-based models. However, given the growing body of work in MARL using other architectures like graph neural networks (GNNs) or attention mechanisms that directly model agent interactions, it would be valuable to include comparisons to such baselines.
Insufficient Analysis of Sequential Dependencies: The SE-MDP framework introduces a strong assumption of sequential action-taking, which may not always be practical in environments where agents need to act simultaneously or in parallel. This limitation isn’t fully addressed, and the paper does not provide an alternative for situations where this sequential action may be a bottleneck.
3) The focus on S4 models without considering the possible limitations of these models (such as their effectiveness in high-dimensional continuous action spaces) reduces the generality of the conclusions. Additionally, the paper does not explore whether there are settings where transformers might outperform S4, such as tasks that require more flexible attention mechanisms.

questions:
1. How does the sequential dependency between agents in SE-MDP affect performance in environments that require real-time parallel actions from all agents? Would MADS4 struggle in such settings, and if so, how could it be adapted?
2. Have you considered testing MADS4 in environments with more dynamic interactions, such as continuous control tasks or real-world applications (e.g., autonomous driving, resource allocation)?
3. How does MADS4 compare to architectures like graph neural networks, which are also designed for multi-agent cooperation? It would be beneficial to see a comparison with GNN-based approaches to understand the specific advantages of S4.
4. Given that the S4 model relies on long-term dependencies, what are the trade-offs in using S4 over transformers in tasks with varying sequence lengths? Would transformers perform better in environments with shorter sequences or fewer agents?
5. Can you provide more intuition or formal analysis regarding the gradient flow between agents in SE-MDP? How does this contribute to stability, and why does it improve learning compared to other methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729785783748

thread_type:
official_review

<thread_object>
creator: reviewer_idiw

<content>
summary:
This paper proposes using the Structured State Space Sequence (S4) model for offline MARL and tests it on the Multi-Robot Warehouse (RWARE) and StarCraft Multi-Agent Challenge (SMAC).

strengths:
This work is interesting in exploring whether offline MARL could benefit from advancements in models such as S4.

weaknesses:
(1) The motivation for using S4 instead of a transformer does not fully convince me. The paper claims, “Existing transformer-based approaches for offline MARL either train agents independently, without fully considering them as a multi-agent system, or rely on a centralized transformer model, which suffers from scalability issues.” However, the proposed method also requires the global environment state, and it seems to require the global environment state even during execution (This is strong requirement!)! Then when does your method scale with respect to the number of agents? This needs further clarification.

(2) The paper introduces the setting of a Sequentially Expanded MDP, which implies a sequential structure in the decision-making process. Why limit the proposed algorithm to this setting? This requires justification. Additionally, how do you determine the order of agents? Is it done randomly, since your test cases do not have a predefined order?

(3) Another major concern is the information sharing with limited communication. This assumes that an agent can communicate with the previous agent who just made a decision. This is already a strong form of coordination and should not be assumed to come freely. Could you provide real-world examples where this is applicable? And how the agents could acheieve this?

(4) Is the comparison with other works fair? The proposed algorithm is tested with this information-sharing mechanism and sequential structure, while other algorithms are not. If other algorithms had access to similar forms of communication and coordination, they might also benefit.

questions:
(1) It is necessary to discuss and compare your approach with CFCQL (see 1’), which is the state-of-the-art CQL-based method designed for offline MARL. It performs very well in SMAC domains, so a comparison would be valuable.

(3) The claim, “However, extending these regularization techniques to multi-agent learning presents significant challenges due to the exponential growth of the joint state-action space as the number of agents increases,” is inaccurate. CFCQL has extended CQL to the multi-agent setting, and the growth is linear with the number of agents, not exponential.

(4) This works states that  "these algorithms do not provide guarantees of global-level regularizations and fail to explicitly or implicitly learn cooperative behavior." What exactly does it mean by "global-level regularizations"? What are the global regularizations in the proposed method?

(5) For the 5m_vs_6m task, why do all algorithms fail to reach the performance level of the offline good dataset?

(6) How are the the results obtained for OMAR on the SMAC task? The official repository only contains the continuous version of OMAR, with no discrete versions available. Did authors implement the descrete action version for OMAR by yourself?

1' Shao, J., Qu, Y., Chen, C., Zhang, H., & Ji, X. (2024). Counterfactual conservative Q learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594617646

thread_type:
official_review

<thread_object>
creator: reviewer_r7cj

<content>
summary:
This paper investigates using Structured State Space Sequence (S4) models for multi-agent offline reinforcement learning. The authors extend the Decision S4 model, originally designed for single-agent RL, to a multi-agent setting, following the decision transformer approach that frames offline RL as a sequence modeling problem. To enable cooperation among agents, the authors adopt a sequentially expanded MDP (SE-MDP) paradigm, dividing each decision-making step into mini-steps. Agents share information through the latent representation of S4 models. The proposed approach is evaluated on a subset of the Multi-Robot Warehouse (RWARE) and StarCraft2 Multi-Agent Challenge (SMAC) benchmarks. Experimental results demonstrate that the proposed approach outperforms several baseline methods.

strengths:
**Originality & Significance**

While there is abundant research on using S4 for reinforcement learning, there is limited work on adapting S4 in a multi-agent offline RL setting. This work is well-placed in the literature and could provide the community with valuable insights on the effectiveness of S4 in multi-agent offline RL.


**Clarity**

The paper is well-structured and generally easy to follow. However, some technical details require further clarification. For instance, the model inference process and communication costs could be elaborated upon. Please refer to the "Weaknesses" section for a more detailed discussion.

weaknesses:
1. The reviewer has some concerns about the experimental results presented in the paper. Specifically, only one sequence-based baseline (MADT) is used, and the improvements demonstrated by the proposed method over this baseline are relatively small, with some falling within confidence intervals. To strengthen the experimental section, the reviewer suggests comparing the proposed method with more competitive baselines o, such as [a], which have been reported to achieve better results on SMAC than MADT. This would provide a more convincing demonstration of the effectiveness of the proposed method.

2. The reviewer found some technical contents that need further clarification. Specifically, the mechanism for information sharing during the testing and inference phases remains unclear. The current configuration, which appears to involve a chain of information sharing, could potentially introduce dependencies between agents during inference. If the reviewer understands correctly, the authors use the S4 convolutional nature to address the issue.However, the precise methodology remains unclear. This is a crucial technical aspect of the paper and requires further elaboration.


3. The reviewer found some citations missing, which may lead to confusion. For example, the Methodology section discusses the concept of Sequentially Expanded MDPs without providing any accompanying citations. This omission could lead to confusion among readers regarding the original contributions of this work versus those of prior research.The reviewer recommends that add appropriate citations, such as [b], and explicitly differentiate between the elements that are drawn from existing literature and the novel contributions introduced in this work. This clarification will enhance the overall transparency and understanding of the research presented.

[a] Offline Multi-Agent Reinforcement Learning with Knowledge Distillation, Tseng et al, 2022 

[b] ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency, Li et al, 2022

questions:
1. In a sequentially expanded MDP, how to decide the order of agents? Does the order have a significant impact on the overall performance? 

2. L52 - L53. Please provide citations for the mentioned regularization methods. It is unclear to the reviewer what ‘these regularization techniques’ are referring to and why it is challenging.

3. Could you elaborate on the potential communication cost and delay by sharing the information among all agents? Would it be scalable to hundreds of agents? 

4. There are some inconsistencies between figure 2 and Algo 1. In Figure 2, the agent doesn’t take observations as input, but in Algo 1 it does. 

5. Algo 1, what is $d^t_i$? Please provide a formal definition.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731068652147

thread_type:
official_review

<thread_object>
creator: reviewer_zdmt

<content>
summary:
This paper introduces a Structured State Space Sequence (S4) model to enhance the learning performance of offline pre-training combined with online fine-tuning in multi-agent reinforcement learning (MARL). Experimental results demonstrate that the proposed approach outperforms both offline RL-based and transformer-based MARL algorithms across several SMAC and RWARE tasks.

strengths:
This paper is first  to apply the S4 model to sequence-based offline MARL tasks, contributing a novel framework for improving performance in these settings.

weaknesses:
First of all, the writing in the paper lacks clarity, and the logical flow is sometimes hard to follow. There are several instances of grammatical errors and awkward phrasing that undermine the overall readability of the paper.



The paper primarily combines the S4 model with the SE MDP framework, which does not represent a significant departure from existing methods. While the S4 model itself is innovative, its application to SE MDP does not present a new solution that clearly advances the field. The motivation for the proposed method is not fully convincing. The paper asserts that the introduction of the S4 model to the SE MDP framework provides an advantage over MADT; however, MADT is also compatible with SE MDP, and the paper does not sufficiently clarify the comparative benefits of S4. The paper should provide a more detailed analysis comparing S4 with other multi-agent methods (such as MADT, QMIX, etc.) in the same experimental settings, particularly focusing on key aspects like cooperation, information sharing, and stability. A thorough comparison would help determine whether S4 offers substantial improvements over current approaches.



It is easy to see that using s4 model improves the ability of representation, and thus the results are not surprising.  The experimental evaluation is insufficient, and the paper would benefit from comparisons across a broader range of SMAC maps.  Additionally, the paper mentions that online fine-tuning has limited effectiveness, with some cases showing performance degradation. However, the authors do not provide a clear explanation of why this occurs, nor do they offer potential solutions to mitigate this issue. It is worth noting that in the SMAC 2c vs 64zg task, the proposed MADS4 method shows lower average returns with on-policy training without offline pretraining compared to MAPPO (which achieves a 100% win rate and an average return of 20.0), raising concerns about the practical effectiveness of the method.

questions:
1. What does the sequential decision-making process bring to the cooperation in MARL, particularly in environments with limited communication or delayed feedback?
2. Could you provide a detailed computational comparison between the MADS4 and MADT methods, particularly in terms of training time, scalability, and computational complexity?
3. In Figure 4, could you clarify the number of interactions represented on the x-axis? Specifically, how many interactions correspond to the 60$\times$1e3 training iterations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732640396631

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their detailed comments and valuable suggestions. As the paper revision deadline is tomorrow, we kindly request the reviewers to review our rebuttal and share any additional feedback. This will enable us to address any remaining concerns thoroughly in the revised manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to use the state space model S4 for multi-agent tasks. However, the idea is only moderately motivated, without a clear intuition. The evaluation domains are on the relatively simple side of the spectrum, which might undermine the true evaluation of the proposed approach. Furthermore, the comparison is not comprehensive as it only compares with the transformer-based approach, neglecting part of the literature that approaches these tasks from other perspectives. Overall, this paper is below the bar of ICLR.