FlickerFusion: Intra-trajectory Domain Generalizing Multi-agent Reinforcement Learning
Authors: Woosung Koh
Wonbeen Oh
Siyeol Kim
Suhin Shin
Hyeongjin Kim
Jaein Jang
Junghyun Lee
Se-Young Yun
Venue: ICLR 2025 Poster (2025)
Keywords: Domain Generalization
Multi-agent Reinforcement Learning
Reliability
Safety
Primary area: reinforcement learning
TL;DR: We present a novel, simple yet effective method for reliable and safe domain generalization in multi-agent reinforcement learning

Abstract: Multi-agent reinforcement learning has demonstrated significant potential in addressing complex cooperative tasks across various real-world applications. However, existing MARL approaches often rely on the restrictive assumption that the number of entities (e.g., agents, obstacles) remains constant between training and inference. This overlooks scenarios where entities are dynamically removed or $\textit{added}$ $\textit{during}$ the inference trajectory—a common occurrence in real-world environments like search and rescue missions and dynamic combat situations. In this paper, we tackle the challenge of intra-trajectory dynamic entity composition under zero-shot out-of-domain (OOD) generalization, where such dynamic changes cannot be anticipated beforehand. Our empirical studies reveal that existing MARL methods suffer $\textit{significant}$ performance degradation and increased uncertainty in these scenarios. In response, we propose FlickerFusion, a novel OOD generalization method that acts as a $\textit{universally}$ applicable augmentation technique for MARL backbone methods. FlickerFusion stochastically drops out parts of the observation space, emulating being in-domain when inferenced OOD. The results show that FlickerFusion not only achieves superior inference rewards but also $\textit{uniquely}$ reduces uncertainty vis-à-vis the backbone, compared to existing methods. Benchmarks, implementations, and  model weights are organized and open-sourced at $\texttt{\href{flickerfusion305.github.io}{\textbf{flickerfusion305.github.io}}}$, accompanied by ample demo video renderings.

Published as a conference paper at ICLR 2025
FLICKERFUSION: INTRA-TRAJECTORY DOMAIN GENERALIZING BULTI-AGENT RL
Woosung Koh'; Wonbeen Oh', Siyeol Kim', Suhin Shin', Hyeongjin Kim', , Jaein Jang', Junghyun Lee", Se- Young Yun Yonsei University, reiss. koh@yonsei. ac.kr KAIST AI, (jb_lee00, yunseyoung @kai ac
ABSTRACT
Multi-agent reinforcement learning has demonstrated significant potential in ad- dressing complex cooperative tasks across various real-world applications. How- ever, existing MARL approaches often rely on the restrictive assumption that the number of entities (e.g., agents, obstacles) remains constant between training and inference. This overlooks scenarios where entities are dynamically removed or added during the inference trajectory- common occurrence in real-world en- vironments like search and rescue missions and dynamic combat situations. this paper, we tackle the challenge of intra-trajectory dynamic entity composition under zero-shot out-of-domain (OOD) generalization, where such dynamic changes cannot be anticipated beforehand. Our empirical studies reveal that existing MARL methods suffer significant performance degradation and increased uncertainty in these scenarios. In response, we propose FLICKERFUSION, novel OOD gen- eralization method that acts as universally applicable augmentation technique for MARL backbone methods. FLICKERFUSION stochastically drops out parts of the observation space, emulating being in-domain when inferenced OOD. The results show that FUSIIN not only achieves superior inference rewards but also uniquely reduces uncertainty vis-à-vis the backbone, compared to existing methods. Benchmarks, implementations, and model weights are organized and open-sourced at fl Elickerfusion305 github. io, accompanied by ample demo video renderings.
INTRODUCTION
Multi-agen reinforcement learning (MARL) is gaining significant attention in the research community as it addresses real-world cooperative problems such as autonomous driving (Antonio & Maria- Dolores, 2022), power grid control (Biagioni et al., 2022), and AutoML (Wang et al., 2023b). Nevertheless, MARL research is commonly over-simplified, in that often, restrictive assumptions made. Among these assumptions, obstacles) available during training and inference the same (Wang et al., 2021b; Liu et al., 2023; Zang et al., 2023; Hu al., 2024). In response, Iqbal al. (2021) introduced the concept of dynamic team composition, where the number of agents varies between training and testing. This concept has been further explored in subsequent works (Hu et al., 2021; Wang al., 2023a; Shao al., 2023; Tian et al., 2023; Yuan et al., 2024). Other variations including situations where the policies of uncontrollable teammates arbitrarily change intra-trajectory have also been studied (Zhang et al., 2023b). Despite extensive research, two crucial characteristics ((1) and (2)) frequently encountered in real-world MARL deployments have been overlooked.
focus on where the number of entities (e.g., agents,
(1) Intra-trajectory Entrance. Consider the following examples where entities enter and leave ttraa-rrajectoy. During a post-tsunami search and rescue (SAR) mission, additional obstacles may enter and leave during the inference trajectory (Drew, 2021), potentially due to secondary hazards like collapsing infrastructure. Similarly, in national defense, additional adversaries or allies dynamically enter and leave during the inference trajectory (Asher et al., 2023). Whether they are allies that
Work done while an intern at KAIST AI
1 "orresponning author

---

Published as a conference paper at ICLR 2025
Trained Inaer-rraecctory
Agents
-264
-1.0
-50
Mean Reward
Adversaries
1000
-150
-200
Inference
Inje-trajecttrry
Intra-Trajectory
216_
QMIX MLP
CAMA Model
XMIIXAAttettoon
(a) uutra-traecctory entity addition (b) Intra-trajectory entity addition (c) Train (3M steps) and inference t 23 an ally agent
t = 44 of an adversary
with equal number of entities.
Figure I: Motivating visualizations for trtra-trajyctory domain generalizing dynamic entity compo- sition in the Tag environment. The models are trained on varying numbers of preys and predators sampled uniformly between and 4. During inference (test-time), each scenario starts with one prey and one predator, with an additional to 3 entities added during the trajectory.
require cooperation or adversaries and obstacles that the allies need adapt this requires on-the-fly fly adaptation. Exiting entities have been studied in MARL, such as SMAC (Samvelyan et al., 2019) where units die, but scenarios where entities enter intra-trajectory have not yet been studied.
Despite seemingly small differences, the dynamics of entering entities differ from dying entities. Consider simple predator- prey environment, Tag, in which the ally agents must move to avoid being tagged by adversaries. Unlike the simple case where an entity dies, intra-trajectory introduces novel scenarios where entities enter from the outskirts of the observed region. For instance, when ally enters, the existing allies and the new ally can enact a dispersion strategy to spread the attention of adversaries (Fig. la). Or, when a new adversary is introduced, the allies must immediately shift their initially planned trajectory to evade the new adversary (Fig. lb). The agents' decentralized policies must adapt to this sudden shift the scenario, without additional training intra-tran tura-trajectory. Indeed, shown in Fig. 1c, existing MARL approaches' performance deteriorates when entities enter intra-trajectory.
(2) Zero-shot Domain Generalization. Such performance degradation is more pronounced when the agents are presented with an entity composition they have never been explicitly trained on. For instance, in Tag, even though agents trained with scenarios with at most four adversaries, they may be presented with five or more adversaries during inference. One naive solution is to explicitly train every possible dynamic scenario priori, and this is the approach taken by prior literature dynamic team composition (Iqbal et al., 2021; Hu et al., 2021). The key underlying assumption is that knows the number of entities beforehand during training, which is unrealistic. In search and rescue (SAR) missions (Niroui et al., 2019), one cannot know all possible combinations of obstacles and targets (humans to rescue) in advance; in national defense, knowing all potential adversaries' combinations during combat priori is impossible. Even when some information of the inference combination is available a priori, the total number of combinations can be very large and impractical to train on. This motivates to study these dynamic MARL scenarios under zero-shot out-of-domaan (OOD) generalization (Min al., 2020), where here, domain refers to the entity composition.
Prior Attempts. These unexpected dynamic scenarios present a non-trivial problem-eae agent is forced to observe an input-space greater than was trained on. Prior works (Hu et al., 2021; Zhang et al., 2023a; Shao et al., 2023) all took the approach of appropriately expanding the Q or policy network, via additional parameters, using inductive bias. However, empirically, all fail to deliver reasonable performance in the considered OOD MARL environments, let alone being consistent, across environments, and within environments (seeds)-a even after extensive hyperparameter tuning (Table 1; Fig. (top-left)). The high uncertainty of existing methods especially renders them unreliable for safety-critical applications (Knight, 2002) such as autonomous vehicles (Pek et al., 2020; Feng al., 2023), where the deployed method must be robust to changes between train and test-time.
2

---

Published as a conference paper at ICLR 2025
Contribution, We propose propoosee novel, orthogonal approach for OOD MARL, FLICKERFUSION, that augments the agents' observation space, introducing no additional parameters test-time. The augmentation is based on the widely-adopted dropout mechanism (Srivastava et al., 2014)-whers neurons (including the input layer), are randomly masked out. FLICKERFUSION differs from Sri- vastava et al. (2014) it leverages the dropout mechanism only in the input layer, such that, the model can remain in an emulated in-domain regime, even when inferenced OOD. A high-level schematic overview is provided in Fig. 2. Prior works do not take this alternative approach as an immediate concern becomes, given no additional parameters, parts of the input space are lost. The stochasticity of the augmentation effec-
tively preserves the information of the entire
Q or Policy Network
observation- agggreggannnggggggnnnnnnnn the stochastic partial observations across time results in an emulated full observation. Note that the maximum number of entities need not be known beforehand, as the entity dropout can adapt to any OOD entity com- position at test-time. The training is done simi- larly to "prepare" for the entity dropouts that the agents will experience test time (Sec. 3). Re- markably, this simple lose and recover approach attains state-of-the-an performance vis-â-vis in- ference reward and uncertainty reduction, rela- tive to an extensive list of existing MARL and other model- as no prior benchmarks can evaluate such dy- namic MARL scenarios, we also standardize and present 12 benchmarks (Sec. 4).
FlickerFusion
Observation
Dynamic
Figure 2: Schematic diagram of FUSKKERUSSON, FUSIONN can be universally attached be- tween any dynamic observation space and any Q or policy network. It ensures the network does not have to initialize extra parameters even as observa- tion size changes at test-time.
methods (Sec. 5). Finally,
Analogy with the Flicker Fusion Phenomena. FLICKERFUSION is named after the infamous Flicker Fusion phenomena in physiology (Simonson & Brozek, 1952). The Flicker Fusion phenomena occurs when an agent perceives a flickering light source as a steady, non-flickering illusion as the frequency of the flicker passes certain threshold (Landis, 1954; Levinson, 1968). Here, dropping entities from agents' observations represents flickers, and through a stochastic input-space drop out at each time step, we fuse the flickers to emulate a full-view across the temporal dimension. We revisit this analogy in Sec. 3.3. Video visualizations of FLICKER are available on our project site.
PRELIMINARIES
Notations. For a positive integer n E N, denote [n] := 21. ,n]. For two vectors a and b, + b is the concatenated vector (a b). For set A, |A| is its cardinality, 2 is its power set, A(A)
is the probability distribution's possibility set over A, and A' is the set of all possible A-valued
sequences of arbitrary lengths. Also, for an integer n S |A], (m) = (B c A: m X n matrix A, denote nrow (A) = n to be the number of row.
[B| = n). For a
DeC with Dynamic Entities. We follow the de facto MARL framework, Centralized Training Decentralized Execution (CTDE; Rashid et al. (2018); Foerster et al. (2017)). Our setting can be described as a Decentralized Markov Decision Process (Dec-MDP; Bernstein et al. (2002); Olichock & Amato (2016) with dynamic entities (Iqbal et al., 2021; ; Schroeder de Witt al., 2019), described by the tpple (E = Eg Erú &, o, U, P, R, 7). E is the set of all possible entities, that may enter or leave intra-trajectory. and E is partitioned into the set of (trainable) agents E, and other non-agent entities &,, For L E N, we assume each entity takes one of L types (e.g., ally, adversary, obstacle), and Φ- : {el, e , er.} is the set of all possible entity types where e,'s 's are the elementary bases of RL For each entity e E E, we denote de E & as its type.
State-Action Spaces. Let S C Rd be the shared state space of all the entities, and assume that each entity E has a state representation s" S. Following Iqbal et al. (2021), we assume that s = fo @ 6f where fo e Rd is the feature vector (e.g., velocity, location). To deal with potentially entering and leaving of entities, we define the joint state space S as
3

---

Published as a conference paper at ICLR 2025
the space of joint states of all possible entity combinations: S UEgs S(E) with S(E) { _ {(e,8)) : e E E] : 8 E being the space of all possible joint states for given entity com- bination E _ E. We assume full observability, ie., each agent at their current state s" observe o" == {(e, : E} E S. Similarly, the joint action space U is defined as U == UE,Cc. (Ea)
with UEa)) = { u u(a,u)) a Eo} u E u) for some shared action space U.
Policy Learning. Under decentralized exccution, each agent a has its own policy n" T H ('17") A(S), where 7" (S x U² is the observation-action trajectory available agent Then, the joint policy π is defined as л : (Ea. (T")aE H (-Ir)) DacE. where Eg S & the current agent composition. The global reward function is R S U + which we assume to be
deterministic. The global state-action Q-value is defined as
(", B) E
Y R(s ue) 80 = 8, = u, Eo E
(1)
where 7 E (0, 1) denotes the discount factor, and is taken w.r.t. the randomness of P, and æ Eq. (1) is trained centrally via QMIX (Rashid et al., 2018):
Q 0) :-
0l: QN (T uN, : h(,,0,(s)))) 0%))
(2)
where u(.,:;(( 0n)) is monotonic mixing function, parameterized by hypernetwork (Ha et al., 2017) conditioned on the global state s. The parameters 0 = (0).... 0 0n) are centrally trained with the DQN loss (Mnih et al., 2013). After training, the inference for each agent is done decentrally as choosing arg max, (""r", uz 0%)
FLICKERFUSION: A NEW APPROACH TO OOD MARL
As mentioned in the Introduction, we take different direction from existing methods. Therefore, we first revisit the two most popular MARL backbones and unpack the innate cause of domain-shift performance degradation under them (Sec. 3.1), then introduce our approach (Sec. 3.2, 3.3).
3.1 TOWARDS DOMAIN GENERALIZATION ACROSS ENTITY COMPOSITION
In our context, domain is a set of tuples (I",, C No: such that
N2 min E E, d= =e)) 120
max (e E E, : 6² = ee]l S NÝ a.s., VE [4]. 120
(3)
In other words, the number of type e, entities is between a lower bound, N/. and an upper bound, NY. Given training occurs in-amnii. D,, then, it is out-of-domain (OOD), Do. if there exists at least one & E [L.] such that N/ of Do is strictly greater than No of D, (see "Qut-of-Domain in Fig. 3a). Thus, in Do. the learner faces entity-type combinations that she has never seen before. We illustrate why naïve application of usual MARL backhones (QMIX-IIP and QMIX-Artention) are sub-optimal for domain generalization. Note that while we focus on off-policy backbones as they are the most commonly used, all our discussion can be trivially extended to on-policy networks. Let us denote 0% the parameters of the in-domain trained QMIX.
MLP Backbone. First, consider QMIX-MLP (Rashid et al., 2018), where an MLP parametrizes the Q-function. When using QMIX-MLP in OOD scenarios, one must expand 0% to 000 = 0 + 0 to take OOD observation ob. as input. This is because 1o0 > l0 due to the additionally introduced entities, where o8, is the in-domain observation. But, because no additional training is allowed during execution, the choice of initialization of 0% critical.
Due to the structure of MLPs, 0/2 must be initialized without prior knowledge or inductive bias of the encountered OOD scenario, resulting in sub-optimal performance. This approach disregards one salient prior knowledge: the entity rype, 6" Suppose the newly added entity is the same type as the one the learner has during training. In that case, it is only natural to initialize the expanded part using corresponding parts of the already trained Q-network. This can be implemented using OMIX-Attention (Iqbal et al., 2021), with some essential modifications described below.
4

---

Published as a conference paper at ICLR 2025
Standard Deviation Across 5 Seeds
Trained In-Domain
7.36
70
62.65
1000
Standard Deviation
50
Reward
Out Domais
-1500
50
In Domain
40
In-Domain
-2300
60
3000
Domain
20
0031
Domainn
10
0032
4000
0.0
0.5
1.0
15
2.0
25
300
In-Domain
0001
0002
Number of Other Ensiy Type
Time Step
(a) Entity count domain generaliza- (b) Drop in reward under domain gen- (c) Risc in uncertainty under domain generalization
tion (Orange: OOD1; Green: OOD2) eralization
Example out-of-domain reward and uncertainty performance (color coded). Shaded region in (b) indicate quartile 3.
Figure 3:
Attention Backbone. We consider the tokenizer (Vaswani et al., 2017) defined as T(f", 6²) MLPP(S @ 6²), and let X E xdr be a matrix whose each row corresponds to the tokenized embedding of each entity, and dy the token embedding size. Then, even though need to introduce additional parameter 0/2 to account for observation of higher dimensionality, it can be initialized by utilizing the tokenizer and the currently observed entity type. However, there still another source of sub-optimality from the inherent structure of the attention mechanism and the fact that the size of the attention matrix changes, namely, nrow crow(XPo (XP)) > nrow(X (X Pr) in
QKK T
Attention Q K, V) == softmax
V
(4)
ddz
Q = 0:=X,WOK X XW* XWV. X = +
Token; X. = Token;
(5)
E.
where ① denotes concatenating the vectors row-wise and Wq, W K WV Rdv are trainable weights. Thus, even with the entity type øf prior, the change in X's size leads to sub-optimality in Eq. (4) and (5) as the model has never been trained on entity compositions larger than nrow (
Empirical Example. For a concrete example, we visualize OOD performance degregation on one of the benchmarks we describe later in Sec. 4. As seen in Fig. 3b and 3c, both inference reward and uncertainty sharply deteriorate under OOD generalization. Although this example study uses representative backbone MARL method, QMIX-Attention, this pattern of deterioration persists across different environments and methods.
3.2 FLICKERING VIA ENTITY DROPOUT
While past works remedy OOD generalization via injecting inductive biases within 0- our orthogonal approach ensures that nrow(X X Po) = nrow(X (XP)), no longer requiring 0z.- In turn, the key design question becomes how many and which entities to drop.
How Many to Drop?
We consider the approach of artificially limiting the visibility as follows:
X Do [I:]. I ([urow(X X [oo]]
(6)
nrow (X.)(X))'
where Poo [I, is the submatrix of Po consisting of the rows of indices index set I. This ensure that nrow Po) nrow (o Pt). This approach has a trade-off: one needs not to introduce new parameters 0/2 as the observation size is retained, but one loses information regarding the entities indexed by I' := [nrow(X" (oww(XP)))) Z. We refer to this solution (Eq. (6)) as domain-aware entity dropout (DAED). It is domain-aware it artificially augments the observation such that it is of
5

---

Published as a conference paper at ICLR 2025
the same size as in-domain. Under the attention backbone, this corresponds to masking the tokens indexed by I', which has been effectively used in prior MARL literature (Iqbal et al., 2021; Shao et al., 2023). Under an MLP backbone, the non-EDed s"'s are concatenated.
The choice of I dictates which and how much information lost, making it a critical design choice. Let Ntrain [N ,.. Nirain] where Nirain is the upper bound the number of type ea entities encountered during training (Eq. (3)). Let N := [Ni"* .... ,."" N be the vector of the number of entities per type that is encountered during inference. We desire [I'l to be as small as possible to minimize the information loss while augmenting the observation size to that of the in-domain. This is done by dropping max(0,N2) N N(rain) entities for each type ez.
Which to Drop? Now that we have determined how many entities of each type to drop, we discuss which ones drop. Following Iqbal al. (2021), we use randomized drop-outs: given A = Ninf - Ntrasn, each agent independently and uniformly samples A(e] number of entities of type to drop. This simple solution is decentralized across the agents and, "on-average", results in "dispersed" view of the entities for the agents. This is formalized in the following statement, whose proof is deferred to Appendix A:
Proposition 3.1. Let [Na] and [Ne] be the set of agents and entities of type f, respectively. Suppose each agent drops De entities of type f, at uniform and independently. Let d(i) be the (random) number of agents that have dropped the entity [Ne). Then, we have the following in-expectation guarantee:
d(t)
De Ne
Ac(Ne Ae) NA
E
Σ iE(NN)
S
NA
á = (d(1)/NA) is the empirical ratios of the number of agents that have dropped entity i, and f (Ar/Ne)iEIN,) is the uniform ratio. Thus, even with random sampling scheme, || pall: is small, i.e., the entities dropped in way that is minimally overlapping across agents. While this encourages dispersed view across the agents, if the entities to be dropped remain fixed throughout the trajectory, there is still some permanently lost information from the perspective of each agent.
3.3 FUSING THE FLICKERS
We now demonstrate how we can recover some of the lost information, further improving the trade- off of Eq. (6) to our favor. For each (active) agent let D. C F, be the set of dropped entities from the perspective of a. If D2 remains fixed throughout the trajectory, then a never sees Da- We alleviate this problem by stochastically changing Da at each t. This also has the added benefit that any intra-trajectory dynamics changes can be immediately responded to on-the-fly. In essence, CLICKERFUSION fuses (stochastic sampling at each t) the flickers (ED). The pseudocodes for train and inference modes are presented in Alg. and 2.
Train. Alg. 1 shows the pseudocode for FLICKERFUSION in the training phase, taking the in- domain entity composition Nirain and learning frequency hyperparameter b inputs. For each episode, the environment (entity composition, line 3) and the number of entities to be dropped (Arrii line 6) are both randomized. Here, ED(., takes the agent a, vector of entities to be dropped A, and entity list E,, and outputs a randomly dropped out observation Og, satisfying Eq. (6). This ensures that the network well-trained over various in-domain combinations (line 3) as well as various partial observations (line 6), where Uniform( a, b) outputs random vector whose i-th coordinate uniformly sampled between a, and b, We apply this training procedure to all the baseline algorithms for our empirical study later.
Inference. Alg. 2 is the pseudocode for FLICKERFUSION in the inference phase, again tak- ing the in-domain entity composition ttaain as input. At each time f, the number of entities to be dropped A is computed from the perspective of agent a, where Observe(t, a) outputs the entity combination vector observed by agent a (line 6). We implement stochastically chang- ing D. by calling ED at each such that the random partial observation (O,,, as well as the set of agents not dropped (N,a). are obtained from DAED (line 7), using which the next action is

---

Published as a conference paper at ICLR 2025
chosen greedily (line 9). Importantly, all this is done decentrally across agents at each time t.
Algorithm 1: FLICKERFUSION (Train)
Algorithm 2: FLICKERFUSION (Inference)
Input: Ntraan b; for each episode do Nitrann Uniform(1, Ntrain);
Input: Ntrain;
for t = 2.2. do
for a E Et,a (decentrally) do if is newly introduced then Initialize 7" + +t::
CreateEnvironment/ (arionntttrrttiin Nrtrain);
for t 1,2, do
A Atrain &- Uniform(O, N
A.T/ Observe(l, a) N
Oa fe_+-ED(o, A Arain, Ee) for each
Oo. No + ED(a, Ai" EE);;
agent E Et,a:
7 [(n):
Add to central replay buffer;
Choose
if L % b = then
UA arg max, (" u; 03 0):
10
Batch sample from replay buffer; Train 0 via QMIX;
T t T U (fueteeNaa):
To concretely demonstrate the effect of call- ing DAED every t (line 7), consider toy sce- nario where L 1, nrow(X*)
4, and nrow(XXo) 5. As illustrated in Fig. 4, each agent's view stochastically changes each I. seen by the green, executing the algorithm at every t ensures smooth transition to any intra- trajectory OOD dynamics. The important in- tuition that aggregating these views along the temporal axis gives virtually full view; this is illustrated more clearly on our project site.
Figure 4: From the perspective of the red agent (A), the dash-lined and lighter background color entity is dropped out. Due to stochasticity, the en- tity dropped commonly differs across t.
4 MPEv2: AN ENHANCED MARL BENCHMARK
To ensure that this problem setting does not fall prey to the standardization issue raised by Papoudakis et al. (2021), we enhance Multi Particle Environments (MPE; Lowe et al. (2017)) to MPEv2, consisting of 12 oourcsourced benchmarks. Although of different nature, the enhancement is conceptually similar to the enhancements made to the StarCraft Multi aggett Challenge (SMAC; Samvelyan et al. (2019)) that led to SMACv2 (Ellis et al., 2023) with stochasticity and SMAC-Exp (Kim et al., 2023) with multi-stage tasks. While SMAC and MPE are the two most popular MARL. environments (Rutherford et al., 2024), we choose to improve MPE due its lower computational requirements (Papoudakis et al., 2021), ensuring accessible research. This is pertinent as OOD dy- namic entity composition is inherently computationally burdensome, due to dimensionality explosion vis-à-vis the simulator (benchmark) and model (parameter) size.
MPEv2 consists of six environments, three extensions to the original MPE, and three novel. These environments were developed priori to the experiments. All environments support an arbitrary dynamic entity composition, including intra-trajectory addition and deletion. As visualized in Fig. 3a, each environment has two benchmarks, OOD1 and OOD2, which assess generalization performance by increasing the number of parameterized agents and non-agent entities, respectively. Fine-grained details for each benchmark are presented in Appendix B.
Brief Descriptions. First, we take the three most appropriate (in terms of our dynamic scenarios) MPE environments, (1) Spread, (2) Adversary, (3) Tag, and add-on arbitrary dynamic entity scenarios. These are then standardized to six benchmarks, two for each environment. The remaining three environments the agents act as bodyguards to the targets. The targets stochastically move point-to-point given a time interval. The agents' objective is to emulate guarding multiple moving targets. Furthermore, they
newly created.
(4) Guard is two-entity (agents, targets) environment where
7

---

Published as a conference paper at ICLR 2025
Table 1: Empirical results, 3M steps (mean final inference reward -o)
MPEv2 Enviroument
Spread
Repel
Tag
Method
00DI
00D2
OODI
00D2
OODI
O0D2
FLICKERFUSOON-ATRN
-198.4 1170
-230.1 119A
845.6 1w1
1070.2 12184
-2.1 1ax
-9,4 13x
FLICKERFUSIN PUP
-183.7 =214
200.00 #254
906.1 =423
TT89.0 :486
-18.8 #243
-33.7 =159
ACORM (Hu
-24.44 1241
330.33 1216
153.6 17753
98.1 1738
-247.9 1183
-578.2 17817
ODIS (Zhing et 20236]
-257. =:82
-376.5+545
-3332. 222944
-2982. 20:444
-1992.042m4
-1723 4213.4
CAMA (Shao et 2023)
-262.2 +1065
-64.1
698.2
928.1 +:95.
-2..6 #378
-31.2
REFIL Oofbal et 2021)
-205.0 :41.0
-289.2 #:12
404.6 1:312
822.4 2:44.4
-15.2 ch1
UPDET (Hu et al. 2021)
-222.0
-273.9
2665.
-1929.0
-22.35
T2288 2422
DotProd (Kadia
2021)
-36.88 =96.1
-55..9 11304
12538.6 =s2na
.365.7 21642
761114 472788
-3302.5 234168
DG-MAMI. (Wang et ot,, 2021a)
-217.3 #17.9
-254.0 #153
343.2 +438
623.0 +-204
.39.2 #23.5
-50.8 =208
SMLDG (
.258.2 438.4
.200,4 +31.8
11711.0 42993.7
-1211.7 21568
-210.6 1135
-185.9 21488
MLDG c1 al. 20108)
.413.1 #47.1
-610. +344
-12763.4 #2135
.9711.1 #2084
-2625.9 #4899
-2333.2
=4121
QMIX- (laball et al., 2021)
190.1 +8.
-251.5 +420
719.0
927.4
#3151
.14.4
22.66
QMIX-MLP (Rashod et 2018)
-20.1
#27.4
-242.8 +313
380.0
+105.0
883.7 #225
-41.6 #165
66.5 =364
MPEv2 Eaviroument
Guard
Adversary
Hunt
Method
O0D1
00D2
OOD1
00D2
OODI
O0D2
BLICKEEFFSION-AAiitte
-128..3 11133
-1160.0
60.9 136.5
9.9 1oss
-297.1
-337.5 1129
FLICKER USIUN-MLP
-1127.2 1462
992.99 1.81
56.3 2301
9.3 2133
-278.5 117
-30.3 :115
ACORM (ilu et al., 2024)
-2074.3
-2049.6 _zmo
56.1 L1k1
13.1 1192
-367.2 1
-397.3
ODIS (Zhagg 20233)
-1449.3 1:90.
-1442.8 =2ms
-5688 231,7
-188.7 230822
-667.55 4wa
-657.8 =20077
CAMA (Shao 2023)
-5002.5 1
-4656.9
_
50.7 gm
26.5 1145
-1063.1 17475
-1131.3 L1M5
(E 2021)
-1445.8 =:10.
-1294.7
=
14.5 =166
-1.77 #214
-305.0 a1ss
-347.9 2148
UPDeT (Hu et 2021)
-2845.5 1.77440
-2637.0
1
.64.6 1mn1
406 1995
-340.8 1871
-383.5 11005
Meta DotProd (Kedia
2021)
-7507.4 1.30011
-7620.4
82.3 1153
34,4 1218
-95600 1516
-1087.9 27380
DG-MAML. (Wing et at. 2031aa
-1885.3 #137.3
-1989.
=138.5
+34.7
=405
-384. #1152
.416 =1234
SMLDG (Li et all, 2020)
-2765.9 1,443
-2591.8
-12.7
-134.8
-601.8
-623.0
MLDG (Li 2018)
-10509.2 =162.2
-9508.4
=
-76.2 #1113
-162.4 =200A
-1164.1 +5:30
-1386.9 =3248
QMIX- OMIX-Atttettoon (lebaal et al., 2021)
-1252.3
1.50
-1202.8
11774
34.7 1781
2.1 1imo
-305.3 197
-34.99 11sT
QMIX-MLP (Raslid 2018)
-T464.6 21362
-1325.1
1177
#343
-8.1 2132
-357.8 4521
-357.5 2144
Green: ours Yellow: MARL methods ; Red: other -odel-agnostic methods: : Blue: MARL backbones
Bold represents the best result across column, and underlined represents the second-best
penalized for colliding with the targets. (5) Repel a two-entity (agents, adversaries) environment where agents try to maximize their distance from adversaries without leaving the play region. The play region defined for each environment in Appendix B. (6) Hunt a two-entity (agents, adversaries) environment where agents hunt down the adversaries. The adversaries (deterministically) move away from the agents at each time step, therefore requiring agents to cooperate intelligently.
EMPIRICAL STUDY
Baselines and Setting. We examine the empirical performance of FLICKERFUSION against 11 relevant baselines. The baselines are divided into 3 categories: (i) MARL backbone methods, (ii) MARL backbone methods with model-agnostic domain generalization methods, and (iii) recent relevant MARL methods. (i) The two backbone architecture we use are QMIX-MLP (Rashid et al., 2018) and QMIX-Attention (Iqbal et al., 2021). (ii) To ensure that our method is most competitive vis-à-vis domain generalization, we implement four model-agnostic domain generalization methods on top of the best performing backbone, QMIX- AMIX-Attention. These are MLDG (Li et al., 2018), SMLDD (Li al., 2020), DG-MAML (Wang et al., 2021a), and Meta DotProd (Kedia et al., 202.). More recent methods are highly domain- and model-specific, leading to logically flawed implementations if applied to our problem setting. We are the first enact such rigor in empirically examining model-aggostie domain generalization methods in the MARL problem setting. (iii) The remaining five baselines are most relevant to our problem setting. UPDeT (Hu al.. 2021). REFIL (Iqbal et al., 2021), CAMA (Shao et al., 2023), and ODIS (Zhang et al., 2023a) are representative methods that focus on dynamic team composition. CAMA and ODIS specifically study zero-shot OOD generalization and show ttateof-the-as results. Finally, we include the most recently proposed state-of-the-art general MARL method, ACORM (Hu et al., 2024).
While related, we do not include Wang et al. (2023a) and Tian et al. (2023). Wang et al. (2023a)'s implementation is complex, closed-source and only empirically examined with a PPO (Schulman et al., 2017) backbone. On-policy RL backbones resulting in poor scalability (Papoudakis et al., 2021). Similarly, Tian et al. (2023)'s implementation is complex, closed-source, with only two baselines and no study against their backbone architecture.
notoriously expensive to train for MARL,
8

---

Published as a conference paper at ICLR 2025
FickkrFusion
ACORM
CAMA
UPDeT
Meta DotProd
SMLDG
OMix-Attention
DG-MAMIL
QMIX-MLP
Uncertainty Distribution Across 12 Benchmarks
FlickerFusion Backbone
Inference Mean Reward
803
Method
403
203
223
2000
2230
5300
1300
Seed Standard Deviation
Time Step
1nd
FlickerFusion Model-Agnostic Methods
FlickerFusion
Inference Mean Reward
Info Inforence rence Mean Reward
000
-2090
-0290
200
-4290
-10080
200
Time Step
Time Step
Ind.
Figure 5: Additional empirical study visualizations. Top-left is the box-and -hisker plot uncertainty distributions across methods. The remainder reward curves of Repel (OODI), demonstrating that SUNnKKFFUSIOO FUSION (dark blue, red) not only performs well at 3M steps, but also converges faster. The shaded regions indicate quartile 1 to complementing the top-left standard deviation statistics.
FLICKER
For readers that are interested in improved generalization through multi-task learning, they can refer to Qin et al. (2024).
We average results over seeds per (benchmark, method) pair. These seeds are randomly picked and never changed. For fair comparison, the same degree-of of freedom and sample size are used for hyperparameter tuning on each (benchmark, hyperparameters that may significantly influence performance are equalized across methods. Each seed trained for 3 million steps, and the 8-sample mean inference reward curve recorded. Training steps were determined a priori and never changed. Following Agarwal et al. (2021) 1); Bettini et al. (2024), we also report the inter-quartile mean (IQM) and the 95% inter-quartile range (IQR) uncertainty distribution in Appendix D, which do not diverge from main text findings.
offrredo--
in Appendix
Overlapping
Results. Table 1 presents the overall results of final mean rewards. FLICKERFUSION ranks first in 10 out of 12 benchmarks, and it always ranks at least second except for Adversary (OOD2). Using FLICKERFUSION method with QMIX-MLP and QMIX-Attention leads to improvements 22 out of 24 times compared to not using it, highlighting the effectiveness of FLICKER FUSIOON as promising plug-in method for OOD MARL. Fig. 5 (top-left) illustrates the nox-and-whisker plot of the uncertainty of each method across all benchmarks. Observe how FLICKERFUSION achieves the lowest median uncertainty relative to baselines, suggesting that it is the most robust OOD MARL method across different environments. For instance, Meta DotProd (grey), despite achieving good performance in Adversary, displays high uncertainty across the considered environments. This renders its one-off superior performance in Adversary pragmatically meaningless. Another observation is that FLICKERFUSION improves agent-wise policy heterogeneity, further detailed Appendix
We also present the full reward curves over the entire horizon for Repel (OOD1) in Fig. 5. Note that ODIS's reward curve is unavailable for the first million steps as it includes pre-training stage (Berg al., 2023), and even after that stage, it is still not visible due to its poor performance. The remaining 11 benchmarks' results are presented in Appendix E.
Ablation DAED vs. ED. We investigate the effect of being domain aware when doing ED. FLICKERFUSION without DA (domain-aware) is implemented by replacing line 6 of Alg. 2 with

---

Published as a conference paper at ICLR 2025
Table 2: Ablation empirical results, 3M steps (final inference reward)
MPEx2 Environment
Spread
Repel
Guard
Adversary
Hunt
Method
00DI
00D2
00D1
00D2
OODI
00D2
00D1
00D2
FLACKERPUSION- PUSON-ASNNiiiiiii
-198.4
-2M0.1
845.6
1,070.2
-2.1
-9.4
-1,288.3
-1,160.0
60.9
9.9
-297.1
-337.5
DA
-212.5
-240.2
854.1
1,114,9
-6.0
-21.2
-1,410.3
-1,158.4
46.1
-17.1
-316.0
-3330
PLICKEPFUSION-MLP
-18.77
906.1
1,189,0
-33.7
56.3
-278.5
-30.3
wo. DA
-196.0
-222.4
577.8
1,046.0
-204.9
-427.
-144776
-1.184.5
43.7
3.3
-285.6
-325.1
Bold represents the best result across backbone type and column
A in. Uniform(0, N i.e., the number of dropped entities is determined independently from
the in-domain entity composition Nirain .A seen in Table 2, DA dramatically improves performance under the MLP backbone, but as much under the attention backbone. This is consistent with the discussion in Sec. 3.1. Since QMIX-Attention has greater level of inductive bias encoded into 0g via o than QMIX- MLP, the effect from DA is relatively weaker.
Compute. Finally, we examine the training compute overhead caused by FLICKERFUSION. When training 3 million steps, on an RIX 3090 24GB and 15 core machine, QMIX- PLP-PPLLCXERFUSSIONN
only incurs an additional 4646.4s - 4838.3s (+4.1%) and 5560.5s + 6066.1 1s (+9.1%) run-time cost, respec- tively. Moreover, memory cost is reduced as additional 02 is not needed.
and
DISCUSSION AND TAKEAWAYS
Novel universal inductive bias injection. We show new inductive bias encoding method for OOD MARL. We rigorously investigate the cause of poor performance under OOD. Then, we systematically identify orthogonal approach. Our findings highlight that OOD MARL problems are better solved via input augmentation, rather than encoding priors into new parameters. Our approach is also universal in that it can easily apply to both MI.P and attention backbones.
Potential in non-attention backbones. Unlike other MARL generalization methods that only work with an attention backbone (Shao al., 2023; Zhang et al., 2023a), FLICKERFUSION works exceptionally well under MLP and attention architectures. Surprisingly, FLICKERFUSION-MLL often beats all other existing attention- methods, even with smaller model (parameter) size (Table 1). Further, during hyperparameter tuning, all else equal, we find no meaningful relationship between model size (parameters) and performance. This echoes other domain literature, which shows that attention may not always be the most optimal design choice (Liu et al., 2021; Zeng et al., 2023).
No reward-uncertainn trade-off. A salient advantage of FLICKERFUSION is its significant reduction in uncertainty in OOD inference relative to baselines (Fig. (top-left)). Remarkably, FLICKERFUSSIO is the only method where the uncertainty decreases relative to backbone. Also, interestingly, we observe negative relationship between reward performance and uncertainty. Mean- ing, better methods vis-à-vis reward (f) is also on average better in terms of uncertainty (1). We provide scatter plots and linear curve fitting analysis Appendix G.
Online over offline data. While ODIS uses additional one million offline data set generating step which results in significant computational overhead, it performs poorly. While it is important to note that ODIS may be useful when abundant offline data is available a priori, it does not work well under our problem setting. In short, self-supervised pre-training from offline data for MARL domain generalization should be used cautiously, especially if offline data is not already available.
Naïve model-agnostic approaches are ill-advised for MARL. Unfortunately, we do not observe any meaningful improvement in performance when implementing model-agnostic domain generaliz- ing methods (see the red part in Table 1). On the contrary, the worst performers in terms of reward and uncertainty come from this pool of methods. However, this is not surprising as these methods not created with MARL tasks in mind.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korean government (MSIT) (No.2019-0-00075. Artificial Intelligence Graduate School Program (KAIST); No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. In Advances in Neural Information Processing Systems, volume 34, pp. 29304-29333 Curran Associates, Inc., 2021. forum? fitruuumiiuuuvv - 041KBe.
URL https: http///pppweeeeeeeeeeeeeeeeeeeet net. wtwprrrrreeeeeeeeeooomttaaaaiii
Guillen-Peed Perez Antonio and Cano Maria-Dolores. Multi-Agent Deep Reinforcement Learning to Manage Connected Autonomous Vehicles Tomorrow's Intersections. IEEE Transactions on Vehicular Technology, 71(7):7033-7044 2022. URL http:///eeeeepeeeeeeeeeee iee...e..ee..e...............eee org/ document/ 9762548.
Derrik Asher, Anjon Basak, Rolando Fernandez, Piyush Sharma, Erin GZaroukian, Christopher Hsu, Michael R Dorothy, Thomas Mahre, Gerardo Galindo, Luke Frerichs, John Rogers, and John Fossaceca. Strategic maneuver and disruption with reinforcement learning approaches for multi- agent coordination. The Journal of Defense Modeling and Simulation, 2064): 2023, doi: 10, 1177/15485129221 104096. URL https : /ww./iiii//////////..oo org/ 10. 1177 11111111771111111922222144111111
Cameron Berg, Vittorio Caggiano, and Vikash Kumar. SAR: Generalization of Physiological
Dexterity via Synergistic Action Representation. In Proceedingg of Robotics: Science and Systems, Daegu, Republic Korea, July 2023, doi: 10.15607/RSS 2023. 202...2X.X000 URL https: www.
beboticsproceedie
org/ sss19/ 900707/0000/0000//00 ..0.7..tm....
Daniel S. Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The Complexity of Decentralized Control of Markov Decision Processes. Mathematics of Operations Research, 27 (4):819-840, 2002. doi: 10.1287/moor.27...81922 URL https: /ttp//i..oo//.11/.///.// moor 27 8...899.99
Matteo Bettini, Amanda Prorok, and Vincent Moens. BenchMARL:
Benchmarking Multi-Agent Reinforcement Learning- Journal of Machine Learning Research, 25(217):1-10 2024. URL
www./ppper..aaapeoeee.o.o/ http::/ org. papers/pppppaaaaaaaa / /233/22222222 1 1112.1111..0 12112.. html.
David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rohit Chintala, Jennifer King, and Ahmed S. Zamzam. PowerGridworld: framework for multi-agent reinforcement learning in power systems. Proceedings of the Thirteenth ACM International Conference on Future Energy Systems, e-Energy '22, pp. 565 560-570, New York, NY, USA, 2022. Association for Computing Machinery. doi: 10.111553538637.355588 3533666 URL https: doi org 114.....333111111111 3539616.
Daniel Drew. Multi-Agen Systems for Search and Rescue Applications. Current Roboties Reports, 2:189-200. 2021. URL htttt:://dddl 1401///000000000/333544440-40400
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Ma- hajan, Jakob Nicolaus Foerster, and Shimon Whiteson. SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning. In Thirty-seventh Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: // /ttp
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X. Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615 (7953):620-62) 627, Mar 2023, ISSN 1476-4687. doi: 00.10388/44188--02--573333 URL https:
//doi //dooioog//0.... 1038 0355:/44455-00000333333333

---

Published as a conference paper at ICLR 2025
Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip H. S. Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1146-1155. PMLR, 06-11 Aug 2017. URL https:: /ttp//ooceedeeooooooooooooooooce ngs ml press v70 woet//oferrerrrrree wtw.rsstrrrrbho... 7b.html.
Le. David Ha, Andrew M. Dai, and Quoe Le. HyperNetworks. International Conference on Learning net wttp://orrmiiddddee
Representations, 2017. URL httph::p///.mm https::/
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. UPDeT: Universal Multi-agent RL via
Policy Decoupling with Transformers. In International Conference on Learning Representations, 2021. URL https: htpp//ppeeeeiieeeeeeeeee wtt.://.eeeeeeeeeeeeeeeeeeee iit.rrmmiidiieeeeeeee 9ADKx.
Zican Hu, Zongzhang Zhang, Huaxiong Li, Chunlin Chen, Hongyu Ding, and Zhi Wang. Attention- Guided Contrastive Role Representations for Multi-agent Reinforcement Learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttpettsss/////eeeeeeeee/eee /pppnrevieeee..../// net/forum dd
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Boehmer, Shimon Whiteson, and Fei Sha. Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learn- ing. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4596-4606. PMLR, 18-24 Jul 2021. URL ht.pps: http://pooeeeen...eee.. www.mmrrr..rs....mmmmmmmmmmmmm.. / v1 39/ aballlalaaaaammcccaa html.
Akhil Kedia, Sai Chetan Chinthakindi, and Wonho Ryu. Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single- Task Regularization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 407-420, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/11/201. findings-emmlp.7.. URL https: : www.aaaolnnnccccccccaoaaaooallla ftt.://oooo...o./.......iiiiiiii
37.
Mingyu Kim, Jihwan Oh, Yongsik Lee, Joonkee Kim, Seonghwan Kim, Song Chong, and Seyoung Yun. The StarCraft Multi-Agen Exploration Challenges: Learning Multi-Stage Tasks and Envi- ronmental Factors Without Precise Reward Functions. JEEE Access, 11:37854 37868, 2023. doi:
(tt.://ACCCSS.022....... 3266652. URL https: http:://eeeeeppooeeee ieee. .ee..rrg///..eeeee.......e..o..e www.ee/rraabbaatttoeeeeeee absttact///tttt//a/ document/1009944
John C. Knight. Safety critical systems: challenges and directions. In Proceedings of the 24th International Conference on Software Engineering. ICSE '02, pp. 547-550, New York, NY. USA, 2002. Association for Computing Machinery. ISBN 158113472X. doi:
URL htt.s: doi org/10 1145/581339 581406.
Camey Landis. Determinants of the Critical 259-286, 1954. doi: 10. 15/phh5//ys htttppyysee..94..4..... 1954.34.2.259. URL https: /tw.///oiiii//// org/10 1152/
Threshold. Physiological Reviews, 34(2):
physrev. 1954 . 34. 259. PMID: 13155188.
John Z. Levinson. Flicker Fusion Phenomena. Science, 160(3823):21-88.. 1968. doi: 10.1126/science. 160.3823.21. URL https: //ww:///wwwwww http:://wwwwiiiinee. scwence. www.einnneoeeoeeeeeeeeooooooooee oogg wwwp//ii///o//eeeeeeeeeeeeeeeeee /at//////////// httbb//...11///ceenncro. 160. 3823.21.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to Generalize: Meta-Learning for Domain Generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 1...6609aaiivv3211 1196. URL pestss://ojs aaai. wawaai.org/iinoe hwtp//ndxxxx../// www.nnde.ppi.. phpp/ AAT/artiicee/view/////
1159..
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Sequential Learning for Domain Generalization. In Computer Vision ECCV 2020 Workshops, pp. 603-619, Cham, 2020. Springer International Publishing. ISBN 778-3-030-66415-2 URL https: /tt////nnni /iink. springer com/ chapter/10. 100 1007 978 8 3 0300
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay Attention to MLPs. In Advances in Neural Information Processing Systems, volume 34, pp. 9204-9215. Curran Associates, Inc., 2021. URL btttss:/cpperreet eww net forum?id--enx-oooeee
12

---

Published as a conference paper at ICLR 2025
Shunyu Liu, Yihe Zhou, Jie Song, Tongya Zheng, Kaixuan Chen, Tongtian Zhu, Zunlei Feng, and Mingli Song. Contrastive Identity-Awar Learning for Multi-Agent Value Decomposition. Proceedings of the AAAI Conference Artificial Intelligence, 37(10):11595-1160,, 2023. URL httpps: aaxxxxxixx xix. org/ alss/ 2211 12712.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- Agent Actor-Critic for Mixed Gooperative-Competiine Environments. In Advances in Neural Information Processing Systems, volume 30. pp. 6382-6393, Curran Associates, Inc., 2017. URL org/ abs /106 02275.
httpps:/aaae httpps////xxi...//e/
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Zheng-Jun Zha, and Yongdong Zhang. Nomain-Aware Visual Bias Eliminating for Generalized Zero-Shot Learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12661-12670, 2020. doi: 10.1 N0.1199CVPR422602000001111 9156892.
URL https: httes///eeeeeepeeeeeeeeeee ieee wtg.eeeeeeeeeeooeooeooeeeeeeeeee wtt//oooo//ooooo///oo/ooo
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5602, 2013. URL
Farzad Niroui, Kaicheng Zhang, Zendai Kashino, and Goldie Nejat. Deep Reinforcement Learning Robot for Search and Rescue Applications: Exploration in Unknown Cluttered Environments.
IEEE Robotics and Automation Letters, 4(2):610-617, 2019. URL htttp:://eeeee:: orre.
leee. orgg http://docuc/////eee/mmmo 8606991.
Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs.
SpringerBriefs in Intelligent Systems. Springer Cham, 2016.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: wwt.//veneeeeeeeeeeeeee net/ forum? wew.:mmid=IIix----00
Christian Pek, Stefanie Manzinger, Markus Koschi, and Matthias Althoff. Using online verification to prevent autonomous vehicles from causing accidents. Nature Machine Intelligence, 2(9):518-528 Sep 2020, ISSN 2522-5839, doi: 10. 10.10386-2255-020--22333 1038 / a 2256 020 022502255222 y.
URL
Rongjun Qin, Feng Chen, Tonghan Wang, Lei Yuan, Xiaoran Wu, Yipeng Kang, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent policy transfer via task relationship modeling. Science China Information Sciences, 67(8):182101 2024.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Rein- forcement Learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4295 4304. PMLR, 10-15 Jul 2018. URL https: http:///pocceeieeeeeeeoo mlr presss//////e / vv0 www.saaiiillaaa aat.sshidlaahmml....
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garōar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly. Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Lange, Shimon Whiteson, Bruno Lac- erda, Nick Hawes, Tim Rocktäschel, Chris Lu, and Jakob Foerster. JaxMARL: Multi-Agent RL. Environments and Algorithms in JAX. In Proceedings of the 23rd International Confer- ence on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 2444-2446, 2024. URL https: //dl:/ddll/ acm org doi /aaaab//1000 /10. 5555 00555533635633 3663188.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, pp. 2186-2188, 2019. URL 3332052..
https://di acm. dog /10.
13

---

Published as a conference paper at ICLR 2025
Christian Schroeder de Witt, Jakob Foerster, Gregory Farquhar, Philip Torr, Wendelin Boehmer, and Shimon Whiteson. Ageti-Agent Common Knowledge Reinforcement Learning. In Ad- vances in Neural Information Processing Systems. volume 32, pp. 9927-9939. Curran As- sociates, Inc., 2019. URL https::/t///t http:////pppppsppepp cttpippspcppppeeeeeeeeee fileerrffllss///ee paper/2019 httml html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv: 2017. URL
org/abs/ 06347.
Jianzhun Shao, Hongchang Zhang, Yun Qu, Chang Liu, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Complementary Attention for Multi-Agent Reinforcement Learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 50776-30793 PMLR, 23-29 Jul 2023. URL https: http::///pccceeiingss./. mlr. press/v200 sha html.
Ernst Simonson and Josef Brozek. Flicker Fusion Frequency: Background and Applications. Phys- iological Reviews, (2(3):349-3788 1952. doi: 10.1 10.1152/hyyreev.19222 195...2..3333499 URL https:
//doi. orggiirrgg/i/ 10. (w112//pyyreee 1952. 32.3. 3. 349. PMID: 12983227.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting Journal of Machine Learning Research, 15(56):1929-1958, 2014. URL http://jmlr:
srivastaval4e 4a.mma.
Zikang Tian, Ruizhi Chen, Xing Hu, Ling Li, Rui Zhang, Fan Wu, Shaohui Peng, Jiaming Guo, Zidong Du, Qi Guo, and Yunji Chen. Decompose a Task into Generalizable Subtasks Multi- Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36, pp. 78514-78532. Curran Associates, Inc., 2023. URL https: id=akyOdKv9ip.
www.eewwweeeooooeeeeeeeeeeeeeooe form??
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30, pp. 6000-6010. Curran Associates, Inc., 2017. URL https: // //aaxiv..rg//aaaaa....aaa /070. 03762.
Bailin Wang, Mirella Lapata, and Ivan Titov. Meta-Learning for Domain Generalization Semantic Parsing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 366-379, Online, June 2021 la Association for Computational Linguistics. doi: 10.18653/v1/2021 htt.///.0...a1lll111111aaaaaaaaa
maacl-main.33 main.33. URL https:
//aclanthology / 2021
Jiangxing Wang, Deheng Ye, and Zongqing Lu. Nlutual-Informatiin Regularized Multi-Agent Policy Iteration. Advances in Neural Information Processing Systems, volume 36, pp. 2617-2635. Cur- ran Associates, Inc., 2023a. URL https::////c openrevieww net/ wtrtt//ormmiiii Fow.ummdddiiin331--
Tonghan Wang. Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: Learning Roles to Decompose Multi-Agent Tasks. International Conference on Learning Representations, b021b. URL /tttps://operrevviiee net.i//wweeeeeeeee.eee......ee.e forum? C=TTUVG6vKN jK.
Zhaozhi Wang, Kefan Su, Jian Zhang. Huizhu Jia, Qixiang Ye, Xiaodong Xie, and Zongqing Lu. Multi- Agent Automated Machine Learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 11960-11969, 2023b. URL https: htteeeeeeeeeee.eeeeeeeeeeeeeeeee ieee. org/abstt abstracte /ttpd//t///t//////////ooo mom.mm001.1120055...
Lei Yuan, Lihe Li, Ziqian Zhang, Fuxiang Zhang, Cong Guan, and Yang Yu. Multiagent continual coordination via progressive task contextualization. IEEE Transactions Neural Networks and Learning Systems, 2024.
Yifan Zang, Jinmin He, Kai Li, Haobo Fu, Qiang Fu, Junliang Xing, and Jian Cheng. Automatic Grouping for Efficient Cooperative Multi-Agent Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 36. pp. 06105-461211 Curran Associates, Inc., 2023. URL https::////............tt...ss / //tpppeeeeeeeeeeee net fert/formmiiii toee//foormmidddo777yyyyyy
14

---

Published as a conference paper at ICLR 2025
Ailing Zeng, Muxi Chen, Lei Zhang. and Qiang Xu. Are Transformers Effective for Time Series Forecasting? Proceedings of the AAAI Conference on Artificial Intelligence, 3799111121111128,, Jun. 2023. doi: 10.1669/aaa..v3779925.66. SEAT/article/viewww//aeo
URL https::/ojss.... aaai. .or.oorg///ooooo htt//index... php/
Fuxiang Zhang, Chengxing Jia, Yi-Chen Li, Lei Yuan, Yang Yu, and Zongzhang Zhang. Discovering Generalizable Multi-agent Coordination Skills from Multi-task Offline Data. The Eleventh Internationaal Conference on Learning Representations, 2023a. URL htttttts///ppeeeeee/e net/ fort/? d=533 yFA.d3yyUPdd
Ziqian Zhang, Lei Yuan, Lihe Li, Xue, Chengxing Jia, Cong Guan, Chao Qian, and Yang Yu. Fast teammate adaptation in the presence of sudden policy change. In Uncertainty in Artificial Intelligence, pp. 2476-2476. PMLR, 2023b.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
Preliminaries
3 FlickerFusion: A New Approach to OOD MARL
3.1 Towards Domain Generalization Across Entity Composition
3.2 Flickering via Entity Dropout
3.3 Fusing the Flickers
4
MPEv2: An Enhanced MARL Benchmark
5
Empirical Study
Discussion and Takeaways
10
Proof of Proposition 3.1
17
Deferred Details for MPEv2
18
B.1
Common Features MPEv2
18
B.2
Tag.
18
B.3 Spread
19
B.4 Guard
20
B.5 Repel
B.6 Adversary
22
B.7 Hunt
23
Hyperparameters
25
C.I Common Hyperparameters
C.2 Baseline Model-specific Hyperparameters
26
Additional Metrics
27
Empirical Study (Extended)
28
F Agent-wise Policy Heterogeneity
34
Relationship Between Reward and Uncertainty
37
16

---

Published as a conference paper at ICLR 2025
A PROOF OF PROPOSITION 3.1
(We follow the proof for sample complexity of learning discrete distribution as a note by C. Canonne)
Pa), where Da E de is the random entity subset of size A to be dropped by agent a. For each [Ne]. 1 E Do] ~ Ber (Ae/Nz), and thus, d(i) ~ Bin(NA, Ae/Ne). We then conclude as follows:
Note that d(i)) EaE[Na) [
d(t)
D
d(i)
De 2 N₂
Σ
Σ
(Jensen's 's inequality)
NA
NE
NA
[E(NN)
Ne(
A
27
Σ
d(1)
NA (c(NN)
N₂
A
A
Σ
NA Ne
1
(d(i) ~ A./N.)))
NA RE[NA
N₂
AN Ac)
NA
17

---

Published as a conference paper at ICLR 2025
B DEFERRED DETAILS FOR MPEv2
B.1 COMMON FEATURES OF MPEV2
In MPEv2, entities are placed in 2-dimensional continuous environment. Distances within this environment are computed using Euclidean metrics, The distance d(e,, Bg) between two entities e; and e, with radii r; R and rj IR is computed as:
d(e,, e4) Ip: P)íl2 74 rj
(AI)
where p, E R2 and Pj E R are position vectors. The play areas are square and centered at the origin [0, 0). The coordinates are constrained within the range x [-B, B) and y [-B, B), where B is the half-width of the play area. Each agent can select from five discrete actions at every timestep. Agents receive a global reward and observe the global state.
Action Shape
u E u N
[no_action, move_left,
Action Space
move_right, move_down, move_up]
Table 3: Action space for agents in MPEv2
The notation vector (n) denotes a vector of size n. 2 denotes vector concatenation. U(:,) denotes
uniform sampling. j.j is the Kronecker delta function, defined as dú,j = { if =
if #
B.2 TAG
This environment features nh agents and Rade heuristic adversaries that move towards the closest agent. The primary objective of this scenario for the agents to learn to avoid collisions with adversaries while remaining within the defined boundaries.
Entity Types
[agent, adversary]
(entity_type (2) @ entity pertttyyposile (2) @ entity_vel (2) (n, + Madv) @
Observation Space
last_actionn (5)
Episode Timesteps
200
(tmax)
Play Area
{(x.)) y) x (-2,2,, 2,,2), R)
Table 4: Tag environment properties
Let Eg be the set agents and Endr the set of adversarics. The reward for each for this environment given as:
R(-) = Σ
Loutside (Ka.Yo) max (-25, min (-5leal- - -5lvel-2) -2
Σ Lcoltision (a, adv)
QEE,
advEEadu
(A2)
Agents receive penalty of -2 for each collision with adversary and additional penalty for traveling beyond the boundaries, with the total boundary penalty capped at -25.
18

---

Published as a conference paper at ICLR 2025
n nénie
énitt ni ade
ni
intra
adv
In-Domain
31,3)
U(1,3)
OOD1
U(1
OOD2
0
U(1, 2)
Table 5: Number of entities for each domain
inst and nin't represents the initial quantities for agents and adversaries, respectively. n, nentre and na Mady denotes the number of entities added intra-trajectory. These variables defined either as fixed values or as ranges from which the number of entities can be randomly sampled.
Agent
Adversary
Vinit
U(- U.20. 0.25B)
U(-B, -0.6B)
Vinit
U.20., 0.25B)
U00.6B, B)
Tintra
B (-1,1))
B. (-1,,,) (6..0 + 62,2) 8e,1 8g,3)
(62.0 + 6,,2) + 8,,1 62,3)
Vintra
B (-(1(11,1)
B. (-1,1)
(6c.1 + 02,3) + 60.0 S2.2
(6e,1 + 6e,3) 60,0 8,.2
Table 6: Entity spawning location
The position entity at the beginning is given as P. := [Tinsts Yimit] and the position of entity introduced during the trajectory is given as P. = [Tintra Vintra Agents initially placed within a smaller square centered around the origin and adversaries are distributed along the corner of the second quadrant using uniform random sampling. The intra-trajectory spawning location P. for agents and adversaries is determined by randomly selecting one of the four edges E {00 2, of square boundary.
B.3 SPREAD
This environment consists of n,, agents and ntar target landmarks. The agents are trained effectively distribute themselves among targets.
Entity Types
[agent, target]
[entity__yye (2) @ entity (2) @ entity_vel (2) ) (na + Maar) $
Observation Space
last_action (5)
Episode Timesteps (fmax)
100
Play Area
((r,)) 11,1 [-1, 1111,,,, R)
Table 7: Spread Environment Properties
Let E, be the set of agents and Eaar be the set of target landmarks. Agents are rewarded for each t based on the proximity of each target to its nearest agent such that:
R(-) Σ
min (d(a, tar)).
(A3)
acE,
19

---

Published as a conference paper at ICLR 2025
n nénie
Thtar
ni
intra
tar
In-Domain
(1,3)
U(1,3)
OOD1
U(1
OOD2
0
U11
Table 8: Number of Entities for Each Domain
tuit init and tar represent the initial quantities for agents and targets, respectively. n and na na
denote the number of entities added intra-trajectory. These variables are defined either as fixed values or as ranges from which the number of entities can be randomly sampled.
Agent
Target
Winit
U(-B,B) B, B)
U(-BBB) B)
Vinit
U( U(-B,B)
U U(-B,B)
Vintra
B B-IU(-1,1) (6.,0 + 6,22 + t 6,,3]
B (-(1,1)
(6.,0 + 62,2) 6,,1 - 8,,3]
Dintra
B. (.(U(-1,1)
B (U(
(6.,1 + 6 + 6..0 S.,2
(6.,1 + 6c,3) 6e,0 8,.2
Table 9; Entity spawning location
The position of entity at the beginning given as Pe [m Dinal and the position of entity introduced during the trajectory is given as Pe [Tantres Vintral- The intra-trajectory spawning location Рe for agents and landmarks is determined by randomly selecting one of the four edges E {0, 1,2, 3) of square boundary.
B.4 GUARD
This environment has na agents and narr heuristic targets. The objective is to minimize the distance between each target and its closest agents. Targets move in random directions every 50 timesteps. while remaining within the boundaries. At a high level, the agents must learn to organize into groups cover and track the moving targets.
Entity Types
lagent, target]
(entity_type (2) @ entity_por (2) @ entity (2) (na + Mea)) $
Observation Space
last_ auttacction( (5)
Episode Timesteps
200
(tmax)
Play Area
((x,v) : 2- 2,, R)
Table 10: Guard Environment Properties
Let Eg be the set of agents and Exar be the set heuristic targets. Then, the reward for each t for this environment is given as:
R(-) = - Σ Σ
min d(a, tar)) - 1coltision (a, tar) where i i=
nh
(A4)
aEE,
Nlar
tare Rm
Agents are rewarded based on the distances of each target to its i-closest agents, and are penalized for collisions with targets.
20

---

Published as a conference paper at ICLR 2025
n nénit
Thtar
ni
intra
tar
In-Domain
(1,3)
U(1,
OOD1
U(1,3)
OOD2
0
U(1
Table 11: Number of Entities for Each Domain
na and minit represent the initial quantities for agents and targets, respectively. For in-domain training and testing, these quantities can vary within specified range, which is used to randomly determine the number of agents and targets, n intra-trajectory.
intra and aaar denote the number of entities added
Agent U(-B, B)
Target
Einit
U(- U(-0.5B,
Vinst
U(-B, B)
((-0.,B,0.5B)
Dintra
B (1) (6..0 + 5,22) + 6,,1 5,,3)
B (-,,,1) (6..0 + 6.,2) 5,,1 8c
Vintra
B (U(--1 (6.,1 + 5,,3) + 6,,0 5,,2
B. [U( (11,) (5,,1 6,,3) 5,,0 8,.2
Table 12: Entity spawning location
The position of entity at the beginning is given Pe [Kenst, Yivit] and the position of entity introduced during the trajectory given as P. [Tintras Nintral Agents and targets are initially distributed across the play using uniform random sampling. The intra-trajectory spawning location Pe for agents and targets is determined by randomly selecting one of the four edges (0, 1, 2, of square boundary.
B.5 REPEL
This environment consists of na agents and Rade heuristic adversaries which move towards the closest agent. Agents are trained to maximize their distance from adversaries while staying within defined boundaries.
Entity Types
[agent, adversary]
[ (entity_type (2) ① entity_pos (2) @ entity_vel (2) (n, Mado) $
Observation Space
last_action [
Episode Timesteps
150
(tmax)
Play Area
{(x, y) x E [- 2.5,, y 2.5, 2.5] 2, E RR.
Table 13: Repel Environment Properties
Let E, be the set of agents and Endv be the of adversaries. The reward for each for this environment is given as:
R(-) Σg REE.
Dontside (Zas Va) max (-25, min (-5leal-2, - -5lval-2))]+
Σ
min d(a, adv). aeE,
advEEado
(A5)
21

---

Published as a conference paper at ICLR 2025
Thus, agents are rewarded based how far the nearest agent is to each adversary and receive penalty for traveling beyond the boundaries, with the total boundary penalty capped at -25.
nanit init
inåt. n
intra na
intra addv
T2
In-Domain
(1,33
U(1,3)
00D1
5
U11 2)
0OD2
5
0
(1, 2)
Table 14: Number of Entities for Each Domain
na and nade represent the initial quantities for agents and adversaries, respectively. For in-domain training and testing, these quantities can vary within specified range, which is used to randomly determine the number of agents and adversaries. na added intra-trajectory.
intro denote the number of entities intra and n adv
Agent
Adversary
Pinit
U(- B)
U U(-0.2,0.2)
Vimit
U(-B,B)
U(- U(-0.2,0.2)
Pintra
B B- (-,,11
B (U,,,1)
(ō0.0 + 8,,2) + 8G,1 6c,3)
(8..0 + d.,2) 8.,1 8,,3)
Yintra
B (-1,1)
B. (-1,)) (6.,1 d,,3) 5 6,,2)
(6c.1 + 6c,3) + 6,,0 6e
Table 15: Entity spawning location
The position of entity at the beginning is given as = [Tinits Fiwit] and the position of entity introduced during the trajectory given as Pe [aintra Yintral. Adversaries are initially placed within a smaller square centered around the origin and agents are distributed across the play area using uniform random sampling. The intra-trajectory spawning location P. for agents and adver- saries is determined by randomly selecting one of the four edges E {0, 1, 2, of square boundary.
B.6 ADVERSARY
This environment features na agents, Nadv heuristic adversaries, Nhar target landmarks, and rate decoy landmarks. Adversaries cannot distinguish between target and decoy landmarks, and move towards the landmark closest to any agent. high level, the agents must learn stay close the targets while guiding adversaries away from them.
Entity Types
[agent, adversary, target, decoy]
[ (entity_typp (4) @ entity (2) @ entity_vel (2)) (na Nadv) +
Observation Space
last_action [5)
Episode Timesteps
150
(fmaax)
Play Area
(1, v) x [-13, 1.3), 1.3),, RR)
Table 16: Adversary Environment Properties
22

---

Published as a conference paper at ICLR 2025
Let &a be the set of agents, Eadv the set of adversaries, and &tar the set of target landmarks. The reward for each t for this environment is given as:
R(- =
Σ tor
min a(a, tar)
Σ
min d(adv, tar).
(A6)
aEEa
advELedv
tarECrar
Hence, agents are rewarded based the proximity of the nearest agent and adversary each target, or more specifically, the difference between the sum of minimum distances from adversaries to each target and the sum of minimum distances from agents each target.
na
init
init
init dec
innra aa
ni
intra adv
intra tar
intra n dec
In-Domain
U11,3)
U11,3)
U(1
U(1
OOD1
5
5
U(1,2)
U(1,2)
U(1,2) 2)
0 U(1
1
0OD2
5
5
U(1
U(1
1
Table 17: Number of Entities for Each Domain
na inst 6,68 nimit and dee represent the initial quantities for each entity type. This can be given set value or range of values used to randomly determine the number of agents and adversaries. intra denote the number of entities added intra-trajectory. intra intra intre and dec
n ady tar adv
Agent U(-BBB)
Adversary U(-B,B)
Target ((-B.B)
Decoy
Pinit
U(-B,B)
Vinit
U(-B,
U(-B,B)
U B, B) U(-B,B)
U(-B,B)
Fintra
B. (U(-1,
B (-1,1),
U(-B,B)
(6e,0 )e,2) + 8,,1
(ōe,0 + 8,,2) + +
Se,3l
de,3l
Yintra
B (U1-),))
B (-(11,1)
U(-B,B)
U(-B,B)
(6e,1 + 52,3) + 82,0
(ō.,1 + 80,3) + 60,0 6e,2
62
Table 18: Entity spawning location
The position of entity at the beginning is given P.. [Finits Yimit] and the position of entity introduced during the trajectory is given as P. [Iantra Fintral- All entities are initially distributed across the play area using uniform random sampling. The intra-trajectory spawning location p. for agents and adversaries determined by randomly selecting one of the four edges E {0, 1, 2, 3) of square boundary.
B.7 HUNT
This environment consists of n, agents and Rade heuristic adversaries which move away from the closest agent while staying within defined boundaries. Agents are trained to minimize their distance from adversaries.
23

---

Published as a conference paper at ICLR 2025
Entity Types
[agent, adversary]
[ [entity__yye (2) @ entity posssy_posooo__ (2)
Observation Space
+ entity_vel (2)) (Na Mode) $
last_action (5)]
Episode Timesteps
200
(maas)
Play Area
(1r,)) x [-2,
2/:
Table 19: Hunt Environment Properties
Let & be the set agents and Endv the set of adversarics. The reward for each for this environment is given as:
R(-)
Σ
min d(a, adv) + 1 E min d(a, adv)
(A7)
Tuudo advEEadv
eEEn
nh aEE.
advEEado
Thus, agents are rewarded based on are rewarded based on the average minimum distances the nearest adversaries, as well as the average minimum distances from adversaries to the nearest agents.
ninit
To
intra
intra add
ado
In-Domain
311,3)
(11
1
O0D1
U(1, 2)
O0D2
5
0
U(1, 2)
Table 20: Number of Entities for Each Domain
init and nade represent the initial quantities for agents and adversaries, respectively. For in-domain training and testing, these quantities can vary within specified range, which is used to randomly determine the number of agents and adversaries. n intra and n adv denote the number of entities added intra-trajectory.
init
Agent
Adversary
Pinit
U(-B, 0.25B) (0.255 B)
U(-0.25B,0.255
Vinit
U(-B, -0.25B) U (0.25BB, B)
U(-0.25B,0.254)
Vintra
B (.(U(11,1)
B B-IU(-1,1) (8..0 de,2) se,1 8,.3]
(8.. + 8c,2) + t8011 Seal
Vintra
B B-1,(-1.1)
B. (-(1,)1 (5c,1 + de,3) + 8.,0 5c.2
(6 + 5,,)) + 6000 5.,2]
Table 21: Entity spawning location
The position of entity at the beginning is given P. [Minits Yemit] and the position of entity introduced during the trajectory is given as Pe [Fantras Dentral Adversaries are initially placed within smaller square centered around the origin and agents are positioned discrete regions at the corners of the play area. The intra-trajector spawning location Pe for agents and adversaries is determined by randomly selecting one of the four edges e {0, 1, 2,3) of square boundary.
24

---

Published as a conference paper at ICLR 2025
C HYPERPARAMETERS
C.I
COMMON HYPERPARAMETERS
In the tables below are the hyperparameters used for MLP backbone models (Table 22) and attention backbone models (Table 23).
Parameters
Tag
Spread
Guard
Repel
Adversary
Hunt
Test interval
20000
20000
20000
20000
20000
20000
Test episodes
15
15
15
15
15
Epsilon start
1
Epsilon finish
0.3
0.05
0.05
0.05
0.05
0.05
Epsilon anneal steps
4000000
500000
500000
500000
1000000
500000
Number of parallel envs
32
8
8
8
8
Batch size
32
32
32
Buffer size
5000
5000
5000
5000
5000
5000
Max timesteps
3000000
3000000
3000000
3000000
3000000
3000000
Mixing embed dimension
32
32
32
32
32
32
Hypernet embed dimension
128
128
128
128
128
128
Hypernet activation
abs
abs
abs
abs
abs
abs
Learning rate
0.0005
0.0003
0.0003
0.0003
0.0005
0.0003
RNN hidden dim
128
64
64
128
128
128
RNN input dim (ours)
128
32
64
128
32
256
RNN hidden dim (ours)
128
64
256
128
64
256
Table 22: Hyperparameters (MLP backbone)
Parameters
Tag
Spread
Guard
Repel
Adversary
Hunt
Test interval
20000
20000
20000
20000
20000
20000
Test episodes
15
15
15
15
15
15
Epsilon start
Epsilon finish
0.3
0.05
0.05
0.05
0.05
0.05
Epsilon anneal steps
4000000
500000
500000
500000
1000000
500000
Number of parallel
32
8
8
Batch size
32
32
32
32
32
32
Buffer size
5000
5000
5000
5000
5000
5000
Max timesteps
3000000
3000000
3000000
3000000
3000000
3000000
Mixing embed dimension
32
32
32
32
32
32
Hypernet embed dimension
128
128
128
128
128
128
Attention embed dimension
128
128
128
128
128
128
Hypernet activation
abs
softmax
softmax
softmax
sofumax
softmax
Learning rate
0.0003
0.0003
0.0003
0.0003
0.0005
0.0003
RNN hidden dim
128
64
64
64
128
128
RNN input dim (ours)
128
64
128
128
256
64
RNN hidden dim (ours)
128
64
128
512
256
256
Table 23: Hyperparameters (Attention backbone)
25

---

Published as a conference paper at ICLR 2025
C.2 BASELINE MODEL-SPECIRI HYPERPARAMETERS
In this section, the hyperparameters used for CAMA (Table 24) and UPDeT (Table 25) models are detailed.
Parameters
Tag
Spread
Guard
Repel
Adversary
Hunt
Cross Entropy weight
0.0005
0.005
0.005
0.005
0.005
0.0005
Club weight
0.1
0.1
0.5
0.1
0.1
0.1
Default attention concentration rate
1
1
Default communication message compression rate
0.5
0.5
0.5
0.5
0.5
0.5
Gradient clipping
10
10
10
10
10
Table 24: CAMA Hyperparameters
Parameters
Tag
Spread
Guard
Repel
Adversary
Hunt
Transformer embed dimension
64
64
64
64
128
64
Number of heads
Number of blocks
2
2
2
2
2
2
Table 25: UPDeT Hyperparameters
Parameters
Tag
Spread
Guard
Repel
Adversary
Hunt
Number of heads
1
1
Number of blocks
1
1
1
5.0
5.0
5.0
5.0
5.0
5.0
0.001
0.001
0.001
0.001
0.001
0.001
A
5.0
5.0
5.0
5.0
5.0
5.0
Steps skill discovery
1000000
1000000
1000050
1000050
1000050
1000000
Training steps of QMIX for ex- tracting offline data
1000000
1000000
1000000
1000000
1000000
1000000
Skill dimension
3
3
3
4
3
Table 26: ODIS Hyperparameters
26

---

Published as a conference paper at ICLR 2025
D ADDITIONAL METRICS
See Table 27 and for IQM values. Furthermore, see Fig. 6 for the 95% 1QM uncertainty figure.
Table 27: Empirical results, 3M steps (IQM final inference reward)
MPEy2 Environment
Spread
Repel
Tag
Method
OODI
00D2
OODI
OOD2
OODI
00D2
FLICKERFUSION- ATTENNIOO
-193.0
-224.1
784.2
1141.3
-2.92
-1..70
FLICKERFUSION-MLA
-190.8
-223.7
912.0
1204.6
-20.23
-58.55
ACORM
-229.5
-323.4
559.3
418.0
-241.22
-260.37
ODIS
-246.7
-343.1
-2686.9
-3291.3
-1273.53
-1300.17
CAMA
-198.0
-283.5
787.1
1005.5
-14.08
-18.55
REFIL
207.5
-300.3
549.0
877.2
-3.11
-12.09
UPDET
-00855
-266.5
-240.0
-350.3
-135.24
-13.667
Meta DotProd
.40.99
-642.8
-12447.5
-9302.7
-6186.50
5793.31
DG-MAML
-21.00
-25..9
593.6
770.8
-39.12
-53811
SMLDG
248.0
-303.2
-16..0
88.5
-182.94
-130.53
MLDG
-47.11
-623.0
-12665.1
.9750.1
-193.37
-162.54
EMIX-ATTENTE
-192.6
-267.9
752.6
1008.9
-1,99
-25.32
QMIX-MLP
-18..4
-216.9
466.8
1016.5
-3.888
-56.07
MPEx2 Environment
Guard
Adversary
Hunt
Method
OODI
OOD2
OODI
OOD2
OODI
OOD2
FLICKERFUSION- ATTENTION
-1280.2
-144.22
63.9
6.9
-297.1
-340.3
FLICKEREFUSION-ML
-1127.0
-967.2
53.1
8.8
-277.1
-306.1
ACORM
-2109.8
-2076.8
61.1
12.6
-339.6
-388.8
ODIS
-158.33
-1332.0
-56.3
-172.7
-654.6
-625.7
CAMA
-499.22
-4328.6
52.1
22.7
10022.7
-1168.4
REFIL
-1455.5
-1261.0
17.9
-20.9
-306.8
346.1
UPDET
-182077
-1559.6
-60.7
-321.3
334.0
Meta DotProd
-7422.2
-7705.7
79.9
33,9
-870.4
-97.2
DG-MAML
-1550.0
-1561.5
26.4
18.0
-334.0
-357.3
SMLDG
26117.
-2269.9
-10.5
-130.1
-470.6
-470.9
MLDG
-10570.8
-9552.6
-68.1
-131.2
1108.3
-1470.7
EMIX-ATTENTION
1142.66
-1356.2
39.9
-4.0
-312.5
-355.5
QMIX-MEE MLP
-1251.3
-1161.0
33.8
2.3
-304.6
-343.9
Green: ours Yellow: MARL methods
Red: other model-agnostic methods
Blue: MARL backbones
Bold represents the best result across column, and underlined represents the second-best
Distribution of Uncertainty across 12 benchmarks
PEDO
SPLDG
DG PVML
Model
M0O De
UnDeT
2506
9000
3008C
12500
19559
17500
202
400
680
12:00
1482
seed 95% 1OR
Figure 6: Box and whisker plots aggregated across all benchmarks
27

---

Published as a conference paper at ICLR 2025
E EMPIRICAL STUDY (EXTENDED)
The figures below visualize the empirical study results (Figure 7 to 12) for each environment.
Inforsacc Meain Reward
Inference Meann Reward
Monn
-1030
Step
Time Step
(b) FLICKERFUSION vs. Model-Agnostic Domain Gen- eralization
FLICKERFUSION Backbone
Inference Moan Reward
SPEEY
M
Time Stop
Time
(d) FLICKERFUSION vs. MARL Methods (zoomed out for ODIS)
(c) FLICKERFUSION vs. MARL Methods
Inference Mean Reward
1000
-2000
Time Stop
(c) FLICKERFUSION vs. MARL Methods (zoomed ODIS)
Figure 7: Empirical results on Repel (OOD2)
28

---

Published as a conference paper at ICLR 2025
Inference Mean Reward
Inference Mean Reward
-X0
Method FlickerFusson HLP
Method FickerFusicn MLP
FlickerP Pn Atttetion OMIX Atteeiittnnooon
FlsckerFusion Attenbion OMIX Attention
13 Step
Step
(a) FLICKERFUSION Backbone
(b) FLICKERFUSION vs. Backbone
Inference Mean Reward
Inforence Mean Reward
-00
M
Photovd
Masa Locnoo
Hare Lode
Time Step
Time Step
(c) PLICKERFUSION vs. Model-Agnostic Domain Gen-(d) FLICKERPUSION vs. Model-Agnostic Domain Gen-
cralization
eralization
Inference Mean Reward
Inferenca Moxa Reward
Step
Step
(e) FLICKERFUSION MARL Methods
(1) FLICKERFUSION MARL Methods
Figure 8: Empirical results on Spread (left: OODI, right: OOD2)
29

---

Published as a conference paper at ICLR 2025
Inference Mean Reward
Inference Mean Reward
1.00
-443
-0000
Miehed Crick Male
Nothed
22
03
20
Step
Step
(a) FLICKERFUSION Backbone
(b) FLICKERFUSION vs. Backbone
-008
Inference Mean Reward
-2000
Inforance Mean Roward
-200
-0030
-100
-4020
-000
-100
Time Step
Time Step
(c) PLICKERFUSION vs. Model-Agnostic Domain Gen-(d) FLICKERFUSION vs. Model-Agnostic Domain Gen- eralization
cralization
Inference Mean Reward
Inference Moan Reward
2012
Step
Time Step
(e) FLICKERFUSION MARL Methods
(1) FLICKERFUSION MARL Methods
Figure 9: Empirical results on Guard (left: OOD1, right: 0OD2)
30

---

Published as a conference paper at ICLR 2025
Inference Mean Reward
Mathed
Inference Mean Reward
-100
-120
--33
203
13
24
Step
Step
(a) FLICKERFUSION Backbone
(b) FLICKERFUSION vs. Backbone
Inference Mean Reward
Inforence Mean Roward
-50
-2:0
-212
-11
2155
Time Step
Time Step
(c) PLICKERFUSION vs. Model-Agnostic Domain Gen-(d) FLICKERPUSION vs. Model-Agnostic Domain Gen- cralization
eralization
Inference Meain Reward
Inference Moan Roward
Makked
Step
Step
(e) FLICKERFUSION MARL Methods
(I) FLICKERFUSION MARL Methods
Figure 10: Empirical results on Tag (left: OOD1, right: 00D2)
31

---

Published as a conference paper at ICLR 2025
Inference Mean Reward
Inference Moan Reward
13
Stop
Step
(a) FLICKERFUSION Backbone
(b) FLICKERFUSION vs. Backbone
Inference Mean Reward
Inforence Mean Roward
Time Step
Time Step
(c) PLICKERFUSION vs. Model-Agnostic Domain Gen-(d) FLICKERFUSION vs. Model-Agnostic Domain Gen-
cralization
eralization
Inference Mean Reward
Inferance Moan Roward
Step
Step
(e) FLICKERFUSION MARL Methods
( FLICKERFUSION MARL Methods
Figure 11: Empirical results on Adversary (left: OODI, right: OOD2)
32

---

Published as a conference paper at ICLR 2025
Inference Mean Reward
Inference Mean Reward
203
Meched Ciec Mal
Nathed
20
Step
Step
(a) FLICKERFUSION Backbone
(b) FLICKERFUSION vs. Backbone
Inference Hean Reward
Inforence Mean Reward
.00
-C00
10000
Time Step
Time Step
(c) PLICKERFUSION vs. Model-Agnostic Domain Gen-(d) FLICKERFUSION vs. Model-Agnostic Domain Gen-
cralization
eralization
Inference Meen Reward
Inferenca Moan Reward
Step
Step
(e) FLICKERFUSION MARL Methods
(I) FLICKERFUSION MARL Methods
Figure 12: Empirical results on Hunt (left: OOD1, right: 0OD2)
33

---

Published as a conference paper at ICLR 2025
F AGENT-WISE POLICY HETEROGENEITY
Diversity across in Figure 13 highlight agent-wise policy heterogeneity. We visualize the attention matrix of QMIX-Attention and FLICKER visualization, snapshot the matrices the middle of the episode (6maz /2). Resolving agent-wise homogeneity enhances the aggregate strategic expressivity of the cooperative system. This can be qualitatively validated on our site's demo rendering videos. We visualize all attention matrices for the remaining benchmarks in (Figure 14 to 22).
during inference. For consistent
Spread:
10
1
10
LG
L2
27
L2
27
:5
14
:3
11
Agants
Agents
10
13
9
11
10
10
5
0
35
L4
22
O
24
14
20
4D
15
15
12
11
10
0
20
6
10
Entities [inci. Agents)
Agents)
(a) QMIX attention matrix
(b) FLICKERFUSION attention matrix
Figure 13: Agent-wise policy heterogeneity significantly improved in Spread (OOD1)
Spread: 0002 QMIX
Spread: 00D2 PlickerFusion
::
25
32
27
11
12
11
Entities (Inci. Agental
Entittie (incl. Agenta]
(a) QMIX attention matrix
(b) FLICKER attention matrix
Figure 14: Agent-wise policy heterogeneity improved in Spread (OOD2)
Adversary: 0O01 QMIX Attention Matrix (t cn 75)
Adversary: 00D1 FllckerFusiio Attention Matrix
Cuttien Ageetni
(a) QMIX attention matrix
(b) FLICKERFUSION attention matrix
Figure 15: Aise policy heterogeneity improved in Adversary (OOD1)
34

---

Published as a conference paper at ICLR 2025
Adversary: 0002 QMIX Attention Matrix de 75)
Agunta
Agpents
16
(incl.
(a) QMI attention matrix
(b) FLICKERFUSION attention matrix
Figure 16: Aient-wise policy heterogeneity improved Adversary (OOD2)
Attention (t 100)
100)
25
25
17
15
33
23
25
Q
Agents
18
15
15
Agentt
28
30
2
5
25
17
15
0
0
10
59
20
3
23
18
100
G
O
27
(incl. Agents)
Entities Agents)
(a) QMI attention matrix
(b) FLICKERFUSION attention matrix
Figure 17: Agent- policy heterogeneity improved in Guard (OODI)
Guard: O0D2 QMIX Attention Matrix (t == 100)
Guard: OOD2 FllckerFusion Attention Matrix (t z= 100)
13
23
Agents
27
23
45
20
18
:8
20
27
17
12
27
34
22
Entities (incl. Agents)
Entities (incl. Agents)
(a) QMIX attention matrix
(b) FLICKERFUSION attention matrix
Figure 18: Agent-wise policy heterogeneity improved in Guard (OOD2)
Repal: QMIX
75)
Repel: FlickerFusion Attention Matrix (t 75)
12
14
IL
12
7
15
15
12
0
33
14
15
15
Agenta
Agents
15
15
12
L5
O
17
16
22
7
10
16
13
12
13
12
11
14
11
11
10
Entities (incl. Agents)
(incl. Agents)
(a) QMIX attention matrix
(b) FLICKERFUSION attention matrix
Figure 19: Agent-wise policy heterogeneity improved in Repel (OOD1)
35

---

Published as a conference paper at ICLR 2025
Repel: OOD2 QMIX Attention Matrix 75)
Repel: OOD2 FlickerFusion Attention Matrix 75)
Agenta
Agents
28
Entities (incl. Agents)
Entittie (inct. Agenta]
QMIX attention matrix
FLICKERFUSION attention matrix
Figure 20: Agent-wise policy heterogeneity improved in Repel (OOD2)
Tag: QMIX Attention
100)
Tag:
Attention
100)
13
13
10
11
14
G
10
Aganta
5
G
33
10
10
13
14
14
12
6
13
10
11
0
16
17
[incl. Agents)
Agents)
(a) QMIX atlention matrix
(b) FLICKERFUSION attention matrix
Figure 21: Agent- wise policy heterogeneity improved in Tag (OOD1)
Tag: QMIX Matrix
Tag:
100)
::
S7
:5
20
Agenta
Agents
17
14
17
10
10
Entities (incl, Agents)
Entittis (incl, Agents)
(a) QMIX attention matrix
(b) FLICKERFUSION attention matrix
Figure 22: Agent-wise policy heterogeneity improved in Tag (OOD2)
36

---

Published as a conference paper at ICLR 2025
G RELATIONSHIP BETWEEN REWARD AND UNCERTAINTY
See Fig. 23 and 24.
Adversary (0OD1)
Adversary (0002)
faavard
Free Roword
100
Standard Deviation acrosa Seeds
Standard Deviation across Seeds
a
Guard (O0D1)
Guard (OOD2)
Final Rewnre
4000
Standard
Standard Deviation Seads
(d)
(OCD1)
(OODZ)
Rewward
Fina Reward
Standard Deviation acroas Seads
20 Standard Devintion Seads
(c)
Figure 23: Negative relationship between final reward and seed-wise uncertainty (left: OODI, right: O0D2). Blue dots are results for each method. Red curve is fitted.
37

---

Published as a conference paper at ICLR 2025
Repel (0DD1)
Repel (0002)
-00
Final Raawerd
Final Rewardd
Standard Deviation
Standord Deviation
(a)
(b)
Spread (00D1)
Spread (0002)
Final Rowerd
Finnl
200
Standard Deviation across Seads
Standard Deviation across Beeds
(c)
Tag (00D1)
Tag (OOD2)
-10
Fleal Roward
Floel
6:00
400
500
100
700
(e)
(I)
Figure 24: Negative relationship between final reward and seed-wis wise uncertainty (left: OODI, right: OOD2). Blue dots are results for each method. Red curve is fitted.
38


Reviews:
[{"review_id": "Sv4UqLQ7Zg", "reviewer": "Reviewer_iX1W", "summary": "This paper addresses the MARL problem that entities like teammates or adversaries are dynamically removed or added during the inference trajectory. The key idea is to dropout observation of some sampled entities at each step, denoted as flickers, and fuse these flickers to emulate a full-view across the temporal dimension. Numerical results and demos on MPEv2 demonstrate FlickerFusion’s promising performance.", "questions": "See Weaknesses.\n\nAnd, for this dynamic entity number problem (in MPEv2), is it possible to build a mapping that maps the dimension-varying vector observation to images (the ones you used to render the demos), so that we can simply and directly utilize a CNN to handle the issue?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea of dropping out observation of some sampled entities at each step and fuse these flickers to emulate a full-view across the temporal dimension is inspiring.\n- Authors extended the widely-used MARL benchmark MPE, which can be beneficial for relevant future research.\n- FlickerFusion is compared with 11 baselines, providing sufficient and solid results.\n- Demo videos clearly show how FlickerFusion works.", "weaknesses": "- “Dropout” is the core idea and concept of this paper, but it only appears at the end of Introduction and has not been clearly defined. It would be better if authors define and emphasize it in Abstract of the beginning of Introduction, and distinguish it with the dropout operation in the neural network literature.\n- Preliminaries is a bit too complex, it would be better to simplify it and relate it to the previous text, including some practical examples for illustration.\n- The importance of each entity is different, but the flicker operation treats all entities equally by sampling uniformly. A module that evaluates entities’ intention / importance / uncertainty, using techniques like opponent modeling[1,2], should be developed to better guide the sampling of the flicker operation.\n- The environment dynamics would change dramatically when the number of entities change. It is not clear how FlickerFusion handles this problem and realizes generalization. A clear study on this problem is vital for the application on more complex scenarios. \n- The learning curves should demonstrate standard deviation.\n- More related work about varying agent numbers[3] and intra-trajectory teammate shift[4] should be discussed.\n\n[1] Georgios Papoudakis, et al. Agent Modelling under Partial Observability for Deep Reinforcement Learning.\n\n[2] Kiarash Kazari, et al. Decentralized Anomaly Detection in Cooperative Multi-Agent Reinforcement Learning.\n\n[3] Lei Yuan, et al. Multi-agent Continual Coordination via Progressive Task Contextualization.\n\n[4] Ziqian Zhang, et al. Fast Teammate Adaptation in the Presence of Sudden Policy Change.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZMU7RYWmU", "reviewer": "Reviewer_v8MZ", "summary": "This paper introduces FlickerFusion, a method designed to improve how multi-agent reinforcement learning (MARL) models generalize to environments with a dynamically changing number of agents. The approach involves randomly omitting agents’ observations during training, which helps the model perform better in new, dynamically changing environments in a zero-shot manner. The authors also present a modified version of Multi Particle Environments that supports variable numbers of agents to test their method.", "questions": "- In MPE, instead of randomly omitting entities, wouldn’t it make more sense to omit those farther from the ego agent based on their distance?\n- How does omitting observations differ from the practice of dropping out nodes in standard neural network training?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper tackles an important generalization capability in MARL.\n- FlickerFusion is straightforward and can be easily integrated into various MARL frameworks.", "weaknesses": "- MARL is generally challenging to stabilize during training. By altering observations at each timestep, this method should increase instability in principle, yet the paper does not explain how FlickerFusion manages to achieve less variability in results compared to the baseline QMIX, as shown in Table 1 and Figure 5.\n- The evaluation relies solely on the Multi Particle Environments (MPE), which has a relatively simple observational setup. It remains uncertain if FlickerFusion would be effective in more complex environments such as SMAC.\n- Figures 3 and 5 in the paper would be more informative if they included quantiles instead of only the means.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wBNAFfmgqy", "reviewer": "Reviewer_wmnB", "summary": "This paper addresses multi-agent reinforcement learning (MARL) with a dynamic number of entities. Unlike existing methods that introduce extra parameters, the proposed novel method, FLICKERFUSION, functions as an augmentation technique that selectively drops entities to ensure the observation space aligns with the policy network, regardless of changes in the number of entities. The experiments and ablation studies demonstrate the superior performance of the proposed method and clarify its operational mechanisms.", "questions": "1. How would the proposed method operate in situations where there are different types of entities with varying lengths and/or compositions of observation space?\n2. In Figure 5, I suggest that the authors use a different line type for the proposed method to clearly distinguish FLICKERFUSION from the baselines.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The issue of zero-shot out-of-domain (OOD) generalization in MARL is significant in bridging the gap between laboratory MARL research and real-world deployment.\n2. The concept of FLICKERFUSION, which involves dropping out entities and recovering lost information, is novel and inspiring.\n3. Although the paper uses only one series of benchmarks, MPEV2, it includes several different scenarios and baselines to demonstrate the robust performance of the proposed method.\n4. The presentation is clear, and the details are effectively conveyed in the text and the Appendix.", "weaknesses": "1. The conceptualization of the paper appears to be excessive. For example, \"intra-trajectory\" is discussed in detail in the introduction section but is not mentioned in the methods section, which may confuse the audience. Additionally, some key points and implementations of the proposed method are not conveyed in the introduction. For instance, the technique of dropout is missing from both the abstract and the introduction.\n2. The paper seems to overlook situations where there are different types of entities with varying lengths and/or compositions of observation space. In such cases, the stochastic dropout approach may not be effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UbHtKrq85q", "reviewer": "Reviewer_imDX", "summary": "The paper introduces FLICKERFUSION,  a novel method aimed at enhancing out-of-domain (OOD) generalization in Multi-Agent Reinforcement Learning (MARL) environments where the composition of entities changes dynamically within a trajectory. The authors identify two significant challenges: intra-trajectory dynamic entity addition and zero-shot OOD generalization, which existing methods fail to handle effectively. FLICKERFUSION utilizes stochastic input-space dropout to maintain observation space consistency without additional parameters, achieving superior performance across a newly developed benchmark suite, MPEV2. Empirical results show that FLICKERFUSION outperforms existing approaches in both inference reward and uncertainty reduction.", "questions": "1.How does FLICKERFUSION perform in environments with extremely high or low densities of entity changes? Is the dropout strategy adaptable enough to handle these scenarios effectively?\n\n2.Could this method be extended to other benchmarks, such as SMAC, MAgent, or CityFlow?\n\nRef\n\n[1] Yuan L, Zhang Z, Li L, et al. A survey of progress on cooperative multi-agent reinforcement learning in open environment[J]. arXiv preprint arXiv:2312.01058, 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper addresses a critical yet under-explored problem in MARL, specifically the intra-trajectory dynamic entity composition and its impact on generalization. The proposed approach, which uses entity dropout and stochastic fusion, is original and impactful.\n\n2.The introduction of the MPEV2 benchmark enhances the paper's contribution, allowing a rigorous assessment of the proposed method. The results across various environments are convincing and demonstrate clear performance improvements.\n\n3.The code and benchmarks are made publicly available, supporting reproducibility and further research in this domain.\n\n4.This paper includes many baselines to  validate its efficiency.", "weaknesses": "1.This paper addresses some open problems in MARL. However, it lacks discussion on several related areas, such as multi-agent policy transfer [1] and sudden policy changes [2]. Including a more comprehensive analysis of these works could strengthen the paper.\n\n2.The definition of OOD is unclear and too simple.\n\n3.The use of the 'Flicker Fusion Phenomena' in this study is intriguing. However, the random masking approach might result in the loss of team semantic information. Alternative methods, such as learning to group[3] or applying the widely used mean-field theory in MARL[4], could potentially offer more advantages. Could the authors elaborate on the strengths and weaknesses of their approach compared to these methods?\n\n4.Could this paper be further enhanced by incorporating Large Language Models (LLMs) like ChatGPT[5]?\n\nRef\n\n[1] Qin R, Chen F, Wang T, et al. Multi-Agent Policy Transfer via Task Relationship Modeling[J]. arXiv e-prints, 2022: arXiv: 2203.04482.\n\n[2] Zhang Z, Yuan L, Li L, et al. Fast teammate adaptation in the presence of sudden policy change[C]//Uncertainty in Artificial Intelligence. PMLR, 2023: 2465-2476.\n\n[3] Sheng, Junjie, et al. \"Learning structured communication for multi-agent reinforcement learning.\" Autonomous Agents and Multi-Agent Systems 36.2 (2022): 50.\n\n[4] Yang Y, Luo R, Li M, et al. Mean field multi-agent reinforcement learning[C]//International conference on machine learning. PMLR, 2018: 5571-5580.\n\n[5] Sun C, Huang S, Pompili D. LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions[J]. arXiv preprint arXiv:2405.11106, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)