FlickerFusion: Intra-trajectory Domain Generalizing Multi-agent Reinforcement Learning

Authors: Woosung Koh, Wonbeen Oh, Siyeol Kim, Suhin Shin, Hyeongjin Kim, Jaein Jang, Junghyun Lee, Se-Young Yun
Primary area: reinforcement learning
Keywords: Domain Generalization, Multi-agent Reinforcement Learning, Reliability, Safety

Abstract:
Multi-agent reinforcement learning has demonstrated significant potential in addressing complex cooperative tasks across various real-world applications. However, existing MARL approaches often rely on the restrictive assumption that the number of entities (e.g., agents, obstacles) remains constant between training and inference. This overlooks scenarios where entities are dynamically removed or $\textit{added}$ $\textit{during}$ the inference trajectory—a common occurrence in real-world environments like search and rescue missions and dynamic combat situations. In this paper, we tackle the challenge of intra-trajectory dynamic entity composition under zero-shot out-of-domain (OOD) generalization, where such dynamic changes cannot be anticipated beforehand. Our empirical studies reveal that existing MARL methods suffer $\textit{significant}$ performance degradation and increased uncertainty in these scenarios. In response, we propose FlickerFusion, a novel OOD generalization method that acts as a $\textit{universally}$ applicable augmentation technique for MARL backbone methods. FlickerFusion stochastically drops out parts of the observation space, emulating being in-domain when inferenced OOD. The results show that FlickerFusion not only achieves superior inference rewards but also $\textit{uniquely}$ reduces uncertainty vis-à-vis the backbone, compared to existing methods. Benchmarks, implementations, and  model weights are organized and open-sourced at $\texttt{\href{flickerfusion305.github.io}{\textbf{flickerfusion305.github.io}}}$, accompanied by ample demo video renderings.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730515750577

thread_type:
official_review

<thread_object>
creator: reviewer_imdx

<content>
summary:
The paper introduces FLICKERFUSION,  a novel method aimed at enhancing out-of-domain (OOD) generalization in Multi-Agent Reinforcement Learning (MARL) environments where the composition of entities changes dynamically within a trajectory. The authors identify two significant challenges: intra-trajectory dynamic entity addition and zero-shot OOD generalization, which existing methods fail to handle effectively. FLICKERFUSION utilizes stochastic input-space dropout to maintain observation space consistency without additional parameters, achieving superior performance across a newly developed benchmark suite, MPEV2. Empirical results show that FLICKERFUSION outperforms existing approaches in both inference reward and uncertainty reduction.

strengths:
1.The paper addresses a critical yet under-explored problem in MARL, specifically the intra-trajectory dynamic entity composition and its impact on generalization. The proposed approach, which uses entity dropout and stochastic fusion, is original and impactful.

2.The introduction of the MPEV2 benchmark enhances the paper's contribution, allowing a rigorous assessment of the proposed method. The results across various environments are convincing and demonstrate clear performance improvements.

3.The code and benchmarks are made publicly available, supporting reproducibility and further research in this domain.

4.This paper includes many baselines to  validate its efficiency.

weaknesses:
1.This paper addresses some open problems in MARL. However, it lacks discussion on several related areas, such as multi-agent policy transfer [1] and sudden policy changes [2]. Including a more comprehensive analysis of these works could strengthen the paper.

2.The definition of OOD is unclear and too simple.

3.The use of the 'Flicker Fusion Phenomena' in this study is intriguing. However, the random masking approach might result in the loss of team semantic information. Alternative methods, such as learning to group[3] or applying the widely used mean-field theory in MARL[4], could potentially offer more advantages. Could the authors elaborate on the strengths and weaknesses of their approach compared to these methods?

4.Could this paper be further enhanced by incorporating Large Language Models (LLMs) like ChatGPT[5]?

Ref

[1] Qin R, Chen F, Wang T, et al. Multi-Agent Policy Transfer via Task Relationship Modeling[J]. arXiv e-prints, 2022: arXiv: 2203.04482.

[2] Zhang Z, Yuan L, Li L, et al. Fast teammate adaptation in the presence of sudden policy change[C]//Uncertainty in Artificial Intelligence. PMLR, 2023: 2465-2476.

[3] Sheng, Junjie, et al. "Learning structured communication for multi-agent reinforcement learning." Autonomous Agents and Multi-Agent Systems 36.2 (2022): 50.

[4] Yang Y, Luo R, Li M, et al. Mean field multi-agent reinforcement learning[C]//International conference on machine learning. PMLR, 2018: 5571-5580.

[5] Sun C, Huang S, Pompili D. LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions[J]. arXiv preprint arXiv:2405.11106, 2024.

questions:
1.How does FLICKERFUSION perform in environments with extremely high or low densities of entity changes? Is the dropout strategy adaptable enough to handle these scenarios effectively?

2.Could this method be extended to other benchmarks, such as SMAC, MAgent, or CityFlow?

Ref

[1] Yuan L, Zhang Z, Li L, et al. A survey of progress on cooperative multi-agent reinforcement learning in open environment[J]. arXiv preprint arXiv:2312.01058, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649381270

thread_type:
official_review

<thread_object>
creator: reviewer_wmnb

<content>
summary:
This paper addresses multi-agent reinforcement learning (MARL) with a dynamic number of entities. Unlike existing methods that introduce extra parameters, the proposed novel method, FLICKERFUSION, functions as an augmentation technique that selectively drops entities to ensure the observation space aligns with the policy network, regardless of changes in the number of entities. The experiments and ablation studies demonstrate the superior performance of the proposed method and clarify its operational mechanisms.

strengths:
1. The issue of zero-shot out-of-domain (OOD) generalization in MARL is significant in bridging the gap between laboratory MARL research and real-world deployment.
2. The concept of FLICKERFUSION, which involves dropping out entities and recovering lost information, is novel and inspiring.
3. Although the paper uses only one series of benchmarks, MPEV2, it includes several different scenarios and baselines to demonstrate the robust performance of the proposed method.
4. The presentation is clear, and the details are effectively conveyed in the text and the Appendix.

weaknesses:
1. The conceptualization of the paper appears to be excessive. For example, "intra-trajectory" is discussed in detail in the introduction section but is not mentioned in the methods section, which may confuse the audience. Additionally, some key points and implementations of the proposed method are not conveyed in the introduction. For instance, the technique of dropout is missing from both the abstract and the introduction.
2. The paper seems to overlook situations where there are different types of entities with varying lengths and/or compositions of observation space. In such cases, the stochastic dropout approach may not be effective.

questions:
1. How would the proposed method operate in situations where there are different types of entities with varying lengths and/or compositions of observation space?
2. In Figure 5, I suggest that the authors use a different line type for the proposed method to clearly distinguish FLICKERFUSION from the baselines.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694255484

thread_type:
official_review

<thread_object>
creator: reviewer_v8mz

<content>
summary:
This paper introduces FlickerFusion, a method designed to improve how multi-agent reinforcement learning (MARL) models generalize to environments with a dynamically changing number of agents. The approach involves randomly omitting agents’ observations during training, which helps the model perform better in new, dynamically changing environments in a zero-shot manner. The authors also present a modified version of Multi Particle Environments that supports variable numbers of agents to test their method.

strengths:
- This paper tackles an important generalization capability in MARL.
- FlickerFusion is straightforward and can be easily integrated into various MARL frameworks.

weaknesses:
- MARL is generally challenging to stabilize during training. By altering observations at each timestep, this method should increase instability in principle, yet the paper does not explain how FlickerFusion manages to achieve less variability in results compared to the baseline QMIX, as shown in Table 1 and Figure 5.
- The evaluation relies solely on the Multi Particle Environments (MPE), which has a relatively simple observational setup. It remains uncertain if FlickerFusion would be effective in more complex environments such as SMAC.
- Figures 3 and 5 in the paper would be more informative if they included quantiles instead of only the means.

questions:
- In MPE, instead of randomly omitting entities, wouldn’t it make more sense to omit those farther from the ego agent based on their distance?
- How does omitting observations differ from the practice of dropping out nodes in standard neural network training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711639984

thread_type:
official_review

<thread_object>
creator: reviewer_ix1w

<content>
summary:
This paper addresses the MARL problem that entities like teammates or adversaries are dynamically removed or added during the inference trajectory. The key idea is to dropout observation of some sampled entities at each step, denoted as flickers, and fuse these flickers to emulate a full-view across the temporal dimension. Numerical results and demos on MPEv2 demonstrate FlickerFusion’s promising performance.

strengths:
- The idea of dropping out observation of some sampled entities at each step and fuse these flickers to emulate a full-view across the temporal dimension is inspiring.
- Authors extended the widely-used MARL benchmark MPE, which can be beneficial for relevant future research.
- FlickerFusion is compared with 11 baselines, providing sufficient and solid results.
- Demo videos clearly show how FlickerFusion works.

weaknesses:
- “Dropout” is the core idea and concept of this paper, but it only appears at the end of Introduction and has not been clearly defined. It would be better if authors define and emphasize it in Abstract of the beginning of Introduction, and distinguish it with the dropout operation in the neural network literature.
- Preliminaries is a bit too complex, it would be better to simplify it and relate it to the previous text, including some practical examples for illustration.
- The importance of each entity is different, but the flicker operation treats all entities equally by sampling uniformly. A module that evaluates entities’ intention / importance / uncertainty, using techniques like opponent modeling[1,2], should be developed to better guide the sampling of the flicker operation.
- The environment dynamics would change dramatically when the number of entities change. It is not clear how FlickerFusion handles this problem and realizes generalization. A clear study on this problem is vital for the application on more complex scenarios. 
- The learning curves should demonstrate standard deviation.
- More related work about varying agent numbers[3] and intra-trajectory teammate shift[4] should be discussed.

[1] Georgios Papoudakis, et al. Agent Modelling under Partial Observability for Deep Reinforcement Learning.

[2] Kiarash Kazari, et al. Decentralized Anomaly Detection in Cooperative Multi-Agent Reinforcement Learning.

[3] Lei Yuan, et al. Multi-agent Continual Coordination via Progressive Task Contextualization.

[4] Ziqian Zhang, et al. Fast Teammate Adaptation in the Presence of Sudden Policy Change.

questions:
See Weaknesses.

And, for this dynamic entity number problem (in MPEv2), is it possible to build a mapping that maps the dimension-varying vector observation to images (the ones you used to render the demos), so that we can simply and directly utilize a CNN to handle the issue?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732014003512

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all the reviewers for providing thoughtful reviews. We are especially encouraged to see that the reviewers recognize:
- **Salience** of the zero-shot intra-trajectory domain generalizing **problem setting** (v8MZ, wmnB, imDX)
- **Novelty** and **simplicity** of our **method** (iX1W, v8MZ, wmnB, imDX), with mentions that it was **inspiring** (iX1W, wmnB)
- **Sufficient experimental evidence** (iX1W, wmnB, imDX)
- Extension of MPE to **MPEv2**, will be **beneficial for future research** (iX1W, imDX)
- **Demo videos** that further **aid** the **understanding** of the readers (iX1W)
- **Fully open-source** codes, benchmarks, and weights (imDX)
- **Clear presentation** (iX1W, wmnB)

To avoid redundancy, we address two concerns that were brought up by more than one reviewer here. The remaining concerns and inquiries are carefully answered in each reviewer’s reply. Note, additions or changes to the revised text have been highlighted in 
${\color{blue}blue}$, for reviewers, AC and PC’s convenience. Changes to the figures are not explicitly indicated with ${\color{blue}blue}$, but are disclosed in the responses.

### **[C1] Clarification on Dropout**
Reviewers iX1W and wmnB’s feedback improves the clarity of our paper. First, we updated the **Abstract** to include a brief description of our method—mentioning dropout. Second, we provide a definition, then concretize the explanation of our augmentation as dropout-based in **Sec. 1, Introduction’s Contribution** with appropriate citation [1]:

-   “FlickerFusion stochastically drops out parts of the observation space, emulating…” (**Abstract**)
-   “The augmentation is based on the widely-adopted dropout mechanism (Srivastava et al., 2014)—where neurons (including the input layer), are randomly masked out. FlickerFusion differs from Srivastava et al. (2014) as it leverages the dropout mechanism only in the input layer, such that, the model can remain in an emulated in-domain regime, even when inferenced OOD” (**Sec. 1**)
    
Moreover, we have guided the readers by alluding to dropout in the following parts of **Sec. 1**:
-   “...as the entity dropout can…”
-   “...for the entity dropouts that the agents will…”
-   “...Here, dropping entities from agents' observations…”
-   “...stochastic input-space drop out at…”.

### **[C2] Visualize Uncertainty in Reward Curves**

After incorporating Reviewers v8MZ and iX1W’s feedback, our results are *even more convincing*. As visualized in **Fig. 5**, no baseline’s quartile 1 to 3 meaningfully overlap with our method’s quartile 1 to 3. We have updated the reward curves (**Fig. 3 (b)** and **Fig. 5**) to include quartile 1 to 3 shading. The figures’ captions have also been updated to reflect this. We have chosen to use quartile 1 to 3 shading as the standard deviation values are already present in **Fig. 3 (c)** and **Fig. 5 (top-left)**. This provides a more *holistic evaluation*, aligned with Reviewer v8MZ’s suggestion on using quartile shading.

### **Reference**
[1] Srivastava et al. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” JMLR 2014.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper includes an approach for improving domain generalization in multi-agent RL (MARL), that stochastically drops parts of the observation space during training. This helps emulate or prepare for the entity dropouts (denoted as flickers) that is experienced at test time. The authors validate the method on several tasks of the multi-particle environment (and its enhancements), demonstrating improved generalization and reduced uncertainty in dynamic environments.

In the initial reviews, the submission received positive feedback for its originality and practical relevance. Reviewers highlighted the novelty of focusing on intra-trajectory generalization in MARL and praised the clarity of the method and its experimental results. Concerns were raised about the connection to standard dropout, writing, and reporting of results/analysis. The authors provided -- clarifications during the discussion phase, included inter quartile distances in the plots, and improved the writing of preliminaries and some visualizations -- addressing these concerns satisfactorily. With these edits and clarifications, two reviewers increased their overall ratings.

The AC recommends acceptance and concurs with the unanimous positive consensus of the reviewers. This work introduces a timely and impactful contribution to improve generalization capabilities of MARL policies.