HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
Authors: Kale-ab Tessera
Arrasy Rahman
Amos Storkey
Stefano V. Albrecht
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Reinforcement Learning
Parameter Sharing
Specialisation
Behavioural Diversity
Primary area: reinforcement_learning
TL;DR: We propose an agent-conditioned hypernetwork (HyperMARL) that enables adaptive cooperation in MARL by decoupling observation- and agent-conditioned gradients, resolving a key interference problem caused by coupling agent IDs with observations.

Abstract: Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of *coupling agent IDs with observations*. Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates -- raising a fundamental question: *can shared policies adapt without these intricacies?* We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and *decouple* observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, **HyperMARL**, avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL.

HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
Kale-ab Abebe Tessera'
Arrasy Rahman² Amos Storkey'
Stefano Albrecht
School of Informatics, University of Edinburgh, Edinburgh, UK Schooo of Computer Science, University of Texas at Austin, Austin, TX, USA DeepFlow, London, UK (k. tessera, storkeysttoeeeeeeooo Red. ac.uk, arrasydutexas.cm edu
Abstract
Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achiev- ing this adaptivity remains critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure largely due to cross-agent gradient in- terference, problem we find is surprisingly exacerbated by the common practice coupling agent IDs with observations. Existing remedies typically add com- plexity through altered objectives, manual preset diversity levels, sequential updates - raising a fundamental question: can shared policies adapt without these intricacies? We propose solution built key insight: agent-conditioned hypernetwork can generate agent-specific parameters and decouple observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL, avoids the com- plexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL. achieves performance competitive with six key baselines while preserving behavioural di- versity comparable to non- parameter sharing methods, establishing it as versatile and principled approach for adaptive MARL. The code is publicly available https: thub. com www./aaaeabesssreaaaaaaaaaaaaaaa HaaprrrraaayyArMRR....
Introduction
Specialist and generalist behaviours are critical to collective intelligence, enhancing performance and In Multi-Agent Reinforcement
adaptability in both natural and artificial systems [52] 43 44
23
50)
Learning (MARL) (2) this translates to critical need for policies that can flexibly exhibit specialised,
homogeneous, mixed behaviours meet diverse task demands 226 6).
Optimal MARL performance thus hinges on being able represent the required behaviours. While No Parameter Sharing (NoPS) [30] enables specialisation by using distinct per-agent networks, it suffers from significant computational overhead and sample inefficiency [1I]. Conversely, Full Parameter Sharing (FuPS) 45 18 15), which trains a single shared network, improves efficiency but typically struggles to foster the behavioural diversity necessary for many complex tasks 24 26
This failure of FuPS, particularly for diverse behaviours, was hypothesised to be gradient interference among agents, whereby their updates negatively impact each other's 's learning [1T [54). We not only empirically validate this hypothesis but also demonstrate a critical insight: this conflict is significantly exacerbated by the common practice coupling observations with agent IDs within shared network (Fig. for coupling, Sec. 3.2 for results).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Hypernetwork
Loss
Agent Cond Grads
Grads
Obs-Cond
7
Agont spocifc
weights
ID Encodar
vi'
[o], id' Coupled
To
Docoupled
Figure 1: HyperMARL Policy Architecture. Common agent-ID conditioned shared MARL policy (FuPS+ID, left) HyperMARL (right), which uses igent-conditioned hypernetwork generate agent-specific weights and decouples observation- and agent-conditioned gradients.
Balancing FuPS efficiency with the capacity for diverse behaviours therefore remains central open problem in MARL. Prior works have explored intrinsic-rewards [26] [22], role-based alloca- tions 48 49 specialised architectures [24] 28 6. sequential updates [54), and sharing parameters
within clusters of agents [TT]. However, these remedies introduce their own intricacies: they often alter the learning objective, require prior knowledge of optimal diversity levels, necessitate main- taining agent- specific parameters or require sequential updates. This raises fundamental question: Can design a shared MARL architecture that flexibly supports both specialised and homoge- neous aehaviours-willee altered learning objectives, manual preset diversity levels, or sequential updates?
Guided by our observation-ID coupling insight, we propose HyperMARL. a novel agent-conditioned hypernetwork [19] architecture. HyperMARL generates per-agent weights on the fly (Fig. D and explicitly decouples observation- and agent-conditioned gradients (Sec. /43).. This choice is motivated by hypernetworks' proven effectiveness at resolving gradient conflicts in multi-task RL 33 and continual learning [47]. Our work establishes their effectiveness for the problem of cross-agent interference in MARL. Indeed, HyperMARL empirically attains lower policy gradient variance than FuPS, and we show this decoupling is critical for specialisation (Sec. 5.2 6. 6.1 confirming its role in mitigating interference.
We validate HyperMARL. on diverse MARL benchmarks including Dispersion and Navigation (VMAS) S Multi-Agent MuloCo (MAMuJoCo) [37), SMAX 411. and Blind-Particle Spread (BPS) across environments with two to thirty agents that require homogeneous, heterogeneous, or mixed behaviours. HyperMARL consistently matches or outperforms NoPS, FuPS, and diversity- promoting methods such as Diversity Control (DiCo) (6) Heterogeneous-Ar Agent Proximal Policy Optimisation (HAPPO) (54) Kaleidoscope (28) and Selective Parameter Sharing (SePS) [1I, while achieving NoPS-level behavioural diversity while using a shared architecture.
Our contributions are as follows:
We identify that cross-agent gradient interference in shared policies is critically exacerbated by the common practice of coupling agent IDs with observations (Sec. 3.2).
We propose HyperMARL (Sec. 4p. an agent-conditioned hypernetwork architecture, test the hypothesis that explicitly decoupling these gradients enables adaptive (diverse, homogeneous, mixed) behaviours without the complexities of prior remedies (e.g., altered objectives, preset diversity levels, or sequential updates).
Our extensive evaluation (Sec. 5) across 22 diverse scenarios (up to 30 agents) shows HyperMARL achieves competitive returns against six strong baselines, while achieving NoPS-level behavioural diversity. We further show this decoupling is empirically linked to reduced policy gradient variance
and is critical for specialisation (Sec. 5.2 Sec. 6.1
2 Background
We formulate the fully cooperative multi-agent systems addressed in our work as Dec-POMDP 34
A Pec-POMDP is tuple, (1,S,
R, {O*)ee O, Po. 7). where Il is the set of agents of size

---

P2
P2
A
B
A (0.5,0.5)
(1.1)
A
(1.1)
(0.5.0.5)
B
(L.
(0.5.0.5)
B
(0.5.0.5)
(1.1)
Two-Play Two-Player Payoff
Two-Player Payoff matrix
matrix
N-player Interaction
N-player Interaction
(a) Specialisation Game
(b) Synchronisation Game
Figure 2: Specialisation and Synchronisation Games. The Specialisation game (left), which courages distinct actions, and the Synchronisation game (right), where rewards encourage identical actions. Depicted are their two-player payoff matrices (pure Nash equilibria in blue) and N-player interaction schematics. While simple in form, these games are challenging MARL benchmarks due non-stationarity and exponentially scaling observation spaces (temporal version).
[1, S is the set of global states with initial state distribution Po. A is the action space for agent i where A A, is the joint action space. R : S x A - IR is the shared reward function, Of is the observation space for agent with the joint observation space O x x O O X x
[0. 1] is
the probability of joint observation o E O, i.c. O(o, a, s) = Pr(o, St, a a;-1), : A x & + [0 the state transition function i.e. (s, a, 8²) Pr (s++1 1 St; a.) and yy the discount factor.
In this setting, each agent receives a partial observation of E 0' These observations are accumulated into an action-observation history 4; - (oh ag....... ( Each agent acts based on their decentralised policies (a"))) The joint history and joint action are defined as follows h, = n*)
(h4,..., h2) and a (at,-.. (0,,-..,,;, a.") The goal is to learn optimal joint policy π
that maximizes the expected discounted return as follows π arg max Expepo. (h)). where G(h) Σ 7 a.),
Specialised Policies and Environments. We say an environment specialised if its optimal joint policy contains at least two distinct, non-interchangeabll agent policies Def. in App. C). Under this mild condition, tasks such as Dispersion (5.2) or our Specialisation Game require agents to learn complementary roles rather than identical behaviours.
3 Are Independent or Fully Shared Policies Enough?
Standard independent (NoPS) and fully parameter-shared (FuPS) policies face inherent trade-offs in MARL. NoPS allows for uninhibited agent specialisation but can be sample inefficient and computationally expensive. FuPS, often conditioned with an agent ID (FuPS+ID), is more efficient but can struggle when agents must learn diverse behaviours 11 This section investigates the limitations of these common policy architectures.
To probe these limitations, we introduce two illustrative environments: the Specialisation Game, rewarding distinct actions, and the Synchronisation Game, rewarding identical actions. Both are inspired by prior work [10 5 35 and extended here to N-agent and temporal settings where
agents observe prior joint actions (see Appendix E for full definitions).
3.1 Limitations of Fully Shared and Independent Policies
FuPS without agent IDs provably cannot recover optimal pure Nash equilibria in the non-temporal 2-player Specialisation Game (Proof App. Eb. In practice, however, FuPS often conditioned with agent IDs, and MARL policies must handle complexities beyond static, two-player interactions. We therefore evaluate standard architectures in the temporal n-player versions of these game
We compare three standard architectures trained with REINFORCE [I]:: 1) NoPS; independent
'We use simplified notation here omitting explicit dependence on state transitions and distributions for brevity.
Results for non non-temporal (oormll-form) variants in App. E.5

---

Table Average evaluation reward (mean + 95% CI) for temporal Specialisation vs. Synchronisation using REINFORCE (10 seeds). Bold: highest mean, no CI overlap. Neither fully shared nor independent policies consistently achieve the highest mean reward.
Specialisation
Synchronisation
#Ag
NoPS
FuPS
FuPS+ID
NoPS
FuPS
FuPS+ID
2
0.8810.00
0.50.00.0
0.6410.10
0.3310.12
1.0.10.00
0.91:10.09
0.74
0.25
0.40:0.07
0.32:00.3
1.0000000
0.67:10.15
0.002
0.12110.00
0.2510..3
0.14100.0
0.54:10.10
16
0.64
0.06::0.00
0.13::0.22
0.07:10.00
1.00:0.00
0.55::0.14 14
0.75
Avo. Ryve. Rewwand
0.59
0.23
0.39
Confect 24
In Agents,
Agents
(a) Avg. evaluation reward
(b) Avg. gradient conflict
NoPS
FTD
FuPS+ID (No State)
HyperMARI.
Figure 3: Multi-agent policy gradient methods in the Specialisation environment. The FuPS+ID (No State) ablation outperforms FuPS+ID, showing near-orthogonal gradients (b), indicating that observa- tion-ID decoupling is important. HyperMARL (MLP) enables this decoupling while leveraging state information, and achieves better performance and reduced gradient conflict than FuPS+ID.
policies (Tg. (a"|o")) FuPS: single shared policy (To(a")o')): and 3) FuPS+ID: shared policy incorporating a one-hot agent ID (To(a*)o |o², id' )). All use single-layer networks, 10-step episodes and 10, 000 training steps (further details in Table 1 in App. [].
Empirical Performance. Table that neither NoPS nor FuPS consistently achieves the highest mean evaluation rewards. NoPS excels in the Specialisation Game but is outperformed by FuPS (optimal) and FuPS+ID in the Synchronisation Game Furthermore, the performance gaps widen as the number of agents increases (notably of both fully independent and fully shared policies.
and 16), highlighting the scalability challenges
3.2 Why FuPS+ID Fails Specialise: The Problem of Gradient Conflict
Despite being universal approximator (20) FuPS+ID often struggles to learn diverse policies in practice (Table I I (44). ). A key reason is gradient conflict: when single network processes both observation and agent ID id', updates intended to specialise agent (based on id') conflict with updates for agent j (based on id"), particularly they share similar observations but require different actions. This obstructs the emergence of specialised behaviours (conflict measured via inter-agent gradient cosine similarity, App.
Importance of Observation and ID Decoupling. To investigate the effect of entangled observation and ID inputs, we introduce an ablation: FuPS+ID (No State), where the policy re id'') conditions only on the agent ID, ignoring observations. Surprisingly. FuPS+ID (No State) outperforms standard FuPS+ID in the Specialisation Game for all tested N (Figure Ba), even when N < (where observation spaces small, suggesting the issue is not merely observation size). Figure reveals why: FuPS+ID (No State) shows near-zero gradient conflict (nearly orthogonal gradients), whereas standard FuPS+ID exhibits negative cosine similarities (confticting gradients).
These results show that naively coupling observation and inputs in shared networks can lead to destructive interference, hindering specialisation. While discarding observations is not a general solution (most tasks require state information). this finding motivates designing architectures that can leverage both state and agent IDs, while minimising interference. Section 4 introduces HyperMARL

---

(Figure I which explicitly decouples observation- and agent-conditione gradients through agent- conditioned hypernetworks, leading to improved performance over FuPS variants and reduced gradient conflict compared to standard FuPS+ID (Figure
4 HyperMARL
We introduce HyperMARL, an approach that uses agent-conditioned hypernetworks to learn diverse or homogeneous policies end-to-end, without modifying the standard RI. objective or requiring manual preset diversity levels. By operating within a fully shared paradigm, HyperMARL leverages shared gradient information while enabling specialisation through the decoupling of observation- and agent-conditioned gradients. We present the pseudocode in Sec. F.T with additional scaling F.3 and runtime (F4) details.
4.1 Hypernetworks for MARL
As illustrated Figure 1 for any agent with context e' (i.e., either a one-hot encoded learned embedding), the hypernetworks generate the agent-specific parameters:
8³ (e"),,
Ø² = (e'),
(1)
where h5 and by the hypernetworks for the policy and critic, respectively. The parameters 0² and of define each agent's policy Tg. and critic Vgr, dynamically enabling either specialised or homogeneous behaviours required by the task.
Linear Hypernetworks Given a one-hot agent ID, 1 m1x= linear hypernetwork h generates agent-specific parameters (' as follows
0 0'=77(1")
W +
(2)
where W 1R"x the weight matrix (with m the per-agent parameter dimensionality and n the number of agents), and b E TR. Ixm is the bias vector. Since I is one-hot encoded, each @ corresponds specific row of W plus the shared bias b. If there is no shared bias term, this effectively replicates training of separate policies for each task (in our case, for each agent) 3 since there are no shared parameters and gradient updates are independent.
MLP Hypernetworks for Expressiveness To enhance expressiveness, MLP Hypernetorrks incorpo- rate hidden layers and non-linearities:
0 = h5 f (9%, (e'))
where 9J, an MLP processing the agent context e", and f is a final linear output layer.
Unlike linear hypernetworks with one-hot agent IDs, MLP hypernetworks do not guarantee dis- tinct weights for each agent. Additionally, they increase the total number of trainable parameters, necessitating careful balance between expressiveness and computational overhead.
4.2 Agent Embeddings and Initialisation
The agent embedding e' a one-hot encoded ID for Linear Hypernetworks. For MLP Hypernet- works, we use learned agent embeddings, orthogonally initialised and optimised end-to-end with the hypernetwork. HyperMARL's hypernetworks are themselves initialised such that the generated agent-specific parameters (0', σ') initially match the distribution of standard direct initialisation schemes (e.g., orthogonal for PPO, preserving fan in/out), promoting stable learning.
4.3 Gradient Decoupling in HyperMARI
HyperMARI. mitigates A core difficulty in FuPS is cross-agent gradient interference II 54
this by generating each agent's parameters through a shared hypemetwork, thereby decoupling components of the gradient.
agent-conditioned and observation-condiiiie
For conciseness only show the policy parameters in this section.
5

---

Hypernetwork gradients. Consider fully cooperative MARL setting with a centralised critic, we can formulate the policy gradient for agent follows 2 25
J(0') = a A(h, at) V log no (af 1:)
where h, and a, are the joint histories and joint actions for all agents, / denotes the parameters of agent i, and A(h,, a,) Q(,,, a,) V(b,) the advantage function.
Decoupling, In HyperMARL each agent's policy weights are produced by the hypernetwork lg: 0¹ (e"), optimise single parameter vector vh. Applying the chain rule and re-ordering the expectations:
V Σ
(e)
B B [A(he, a.) V log Te. (a) Rt]]
J. (agont-conditionne)
Z, (observation conditione))
Agent-conditioned factor J, This Jacobian depends only the fixed embedding e' and the hypernetwork weights 15, therefore, is deterministic with respect to mini-bates baiibbtchh samples (as e and & are fixed per gradient step), separating agent identity from trajectory noise.
Observation-condition factor Z4. The expectation averages trajectory noise per agent for its policy component TTs, prior to transformation by J, and aggregation.
The crucial structural decoupling in Equation ensures HyperMARL first averages noise per agent (via factor Z,) before applying the deterministic agent-conditioned transformation J, This mitigates gradient interference common in FuPS+ID D. 54], where observation noise and agent identity become entangled (see Equation (2 in App.
This is the MARL analogue of the task/state decomposition studied by [42] Eq. 18] in Meta-Re RL. Section [5. empirically verifies the predicted variance drop, and ablations in Section 6.1 demonstrate that disabling decoupling degrades performance, underscoring its critical role.
Experiments
Our empirical evaluation of HyperMARI. assesses whether agent-conditioned hypernetworks can enable adaptive (specialised or homogeneous) policies without altered RL objectives, preset diversity levels, or sequential updates. We structure our experiments to directly answer two key research questions: Q1: Specialised Policy Learning: Can HyperMARL effectively learn specialised policies via a shared hypernetwork? Q2: Effectiveness in Homogeneous Tasks: HyperMARL competitive in environments that necessitate homogeneous behaviours?
To address these questions, HyperMARL evaluated against six representative modern baselines across carefully selected suite of MARI.benchaan (22 scenarios, to 30 agents). All experiments use least seeds (details in App. G.2.
5.1 Experimental Setup
HyperMARL is evaluated across 22 scenarios from five diverse MARL environments (Dispersion [5). Navigation
Environments.
Table 2: MARL environments for evaluating HyperMARL.
5
MAMuloCo (37),
SMAX 41), BPS (TI (Table 2 These were specif- ically chosen to rigorously test performance across varying complexities, agent counts (2 to 30), and dis- tinct behaviours (heterogeneous, homogeneous, mixed). Full details in Appendix G.1
Ens.
Agents
Action
Behaviour
Dispersion Navigation
Discrete
Hetero.
2,4,8
Continuous
Homo. Hetero., Mixed
MAMuJoCo
Hetero.
SMAX
2-20
Discrete Discrete
Homo.
BPS
15-30
Heteroo
Baselines.
We evaluate HyperMARL against modern parameter sharing (PS) and diversity-promoting baselines. Core PS comparisons use FuPS+ID and
NoPS. For specialisation, we include privileged baselines: DiCo 6 (shared and non-shared weights, preset diversity levels); HAPPO (54] (shared critic, sequential actor updates); Kaleidoscope [28] (learnable masks, critic ensembles, diversity loss); and SePS (pre-training phase, agent clustering).

---

ID
NoPS
MLP Hyper.
Lin. Hyper.
10h Wwwentteeeee
1E IXT
Milic
(a) IPPO
(b) MAPPO
(c) Actor Gradient Var.
(d) Policy Diversity
Figure 4: Performance and gradient analysis. (a.b) IPPO and MAPPO Dispersion (20M timesteps)
IQM of Mean Episode Return with 95% bootstrap Cls: Hypernetworks match NoPS performance
while FuPS struggle with specialisation. Interval estimates in App. I..31 (c) Actor gradient variance:
Hypernetworks achieve lower gradient variance than FuPS+ID. (d) Policy diversity (SND with
Jensen-Shannon distance): Hypernetworks achieve NoPS-level diversity while sharing parameters.
We use IPPO/MAPPO 13 53 as the underlying algorithm for all methods except Kaleidoscope and SePS (see App. H.I and App. H.2 for these results).
Adhering to best evaluation practices [36], we use original codebaseshhpprpaaanneerr and envi- ronments for which baselines were tuned. HyperMARL uses identical observations and generates architectures of equivalent capacity to baselines. Training and evaluation (App. G.2.I) and hyper- parameters (App. follow each baseline's original setup. We detail our baseline and environment selection criteria in Table with architecture details in G.2
Measuring Policy Diversity. To measure the diversity of the policies we System Neural Diversity (SND) (Equation 5 with Jensen-Shannon distance (details in App. G.2.2)
5.2
Q1: Specialised Policy Learning
Learning Diverse Behaviour (Dispersion) Figures 4a and 4b show that FuPS variants (IPPO-FuPS. MAPPO-FuPS - (o)) can struggle to learn the diverse policies required by Dispersion (even when running for longer Fig. 19 while their NoPS counterparts (IPPO-NoPS, MAPPO-NoPS- converge to the optimal policy, corroborating standard FuPS limitations to learn diverse behaviour, In contrast, HyperMARL. (both linear and MLP variants) match NoPS performance, suggesting that a shared hypernetwork can effectively enable agent specialisation. SND policy diversity measurements (Fig. 4d) confirm FuPS variants achieve lower behavioural diversity than NoPS, while HyperMARL. notably matches NoPS-Ievel diversity.
Gradient Variance. HyperMARL (IPPO and MAPPO vari- ants) also exhibits lower mean policy gradient variance than
IONI Epicode Return
FuPS+ID across actor parameters (Fig. 4c). This aligns with
their ability to learn diverse behaviours and supports the hypoth- esis that its gradient decoupling mechanism (Sec. 4.3) enhances training stability.
6002
5302
0007
1002
Diversity at Complexity and Scale (MAMuJoCo). In the chal- lenging MAMuJoCo heterogeneous control tasks (Table 3). Hy- perMARL (MLP variant) is broadly competitive. Notably, un- like HAPPO and MAPPO (independent actors), HyperMARL uses shared actor and parallel updates, and yet manages strong performance, even in the 17-agent Humanoid-v2 notoriously difficult heterogeneous task[54] (Fig. 5). matching methods that employ independent actors and sequential updates.
Training Steps (M)
Hepertmans (MAPPO))
Figure 5: 17-agent Humanoid learn- ing dynamics (IQM, 95% CI). Hyper- MARL,, utilising shared actor architec- ture, outperforms MAPPO-FuPS (non- overlapping Cls) and matches the per- formance of methods employing non- shared or sequential actors. This chal- lenging environment is recognised for its high variance in outcomes across dif- ferent methods 54
A Adaptability (Navigation). Navigation tasks S assess adapt- ability to homogeneous, heterogeneous, and mixed goals (some agents have the same goals, others different). We compare HyperMARL. with baselines including DiCo (6) While using DiCo's is optimal preset diversity for n=2 agents, note that identifying appropriate diversity levels
7

---

Table 3: Mean episode return in MAMuJoCo for MAPPO VMM, 95% CI). HyperMARL achieves the highest IQM in 3/4 scenarios (bold), and is the only method with shared actors to demonstrate stable learning in the notoriously difficult 17-agent Humanoid environment (see Figure for learning dynamics), indicates CI overlap with the top score.
Scenario
HAPPO
FuPS+ID
Actors
HyperMARI (Ours)
Humznoid v2 17x1
6501.15 (3015 88, 7229.79)
566.12 (536,36, 36, 603.01)
6188,46 (5006.13 085 L.44
6544.10 (3865.00, 6664.89)
Wallierdd-2 2xd
4749.06" (4366-94, 623051)
4574.3 39" (4254.2) 5068.32)
4747.05"
(3974,76, 6249.58)
5064.66 (4635 10, 5423.42)
HallChhentshw222 2x3
6752
33042, 7172.98)
2771.11 (6424 (228.65)
6650.3
(5714.68,
7229 61)
7063.72 (6696 30, 7325, 36)
Ant-12 4s2
6031.92 (5924 32, 6149.22)
6148.58 (9988.63, 663, 6223.88)
6046.23 (9024.62, 6216.57)
5940. 16 (5485.77, 6280 59)
DiC2
DiCs
Hypeen Hapen
Hef2
"DOT
(a) Shared Goals
(b) Unique Goals
Ag
Lin Hypen
MLP Hyper.
Actrage
(c) Mixed Goals (Half Agents Share Goals)
Figure 6: Averuge Reward (IQM. 95% CI) in Navigation for IPPO Variants. HyperMARL adapts robustly across goal configurations- (a) shared, (b) unique, and (c) mixed. Both linear and MLP versions consistently match or outperform IPPO baselines and DiCo, with the margin widening as the number of agents grows.
Sample-efficiency curves appear in App. H.5
for DiCo with larger teams (n > 2) via hyperparameter sweeps proved challenging (see Tables 15 and [6].
Across all tested goal configurations (shared, unique, and mixed), HyperMARL consistently achieves strong performance (6igure(6). It generally matches or outperforms NoPS and FuPS, and outperform- ing DiCo. Interestingly, unlike in sparse-reward tasks like Dispersion, FuPS remains competitive with NoPS and MyPerMARL in Navigation scenarios requiring diverse behaviours for smaller teams (n E {2, 4}), likely due to Navigation's 's dense rewards, However, HyperMARL distinguishes itself as the strongest method for n=8 agents, highlighting its effectiveness in handling more complex coordination challenges.
5.3
Q2: Effectiveness in Homogeneous Tasks
SMAX. Finally, evaluate HyperMARI. (MLP) on SMAX, where recurrent FuPS is the established baseline 41 53 [7]. Figure 7 shows while some FuPS variants might exhibit marginally faster initial convergence on simpler maps, HyperMARL achieves comparable final performance all maps, using the same GRU backbone for partial observability. These results highlight two points: (i)
MAPPO RNN (FuPS)
IPPO ANN (RUPS)
HyparMARL (IPPO RNN)
HyperMARL (MAPPO RNN)
MA MAD AARA
(a) 2z3z
(b) 385z
(c) SMACr2 10 Units
(d) SMACx2 20 Units
Figure 7: 1QM and 95% CI of mean win rate in SMAX. Performance of FuPS Recurrent IPPO and MAPPO and HyperMARL (MLP) on SMAX. HyperMARL, performs comparably to FuPS baselines across all environments, demonstrating its effectiveness in tasks requiring homogeneous behaviours and using recurrency. Interval estimates in Fig. 22 in App. H.6
8

---

1000 Mean Eonadee Rantern
NoSS Hoa Apt
10M0 Meann EONaEdE Baater
8002
Meen
2000
2000
JA 11
8
10
10
15
8
10
15
20
Training
Trining Steps
Traiting
Traning Steps
(a) Humanoid w/o GD
(b) Dispersion w/o GD
(c) Humanoid w/o RF
(d) Dispersion w/o RF
Figure 8: Ablation results comparing HyperMARL to variants without gradient decoupling (w/o GD) and without reset fan in/out initialisation (w/o RF) across environments. Gradient decoupling (a.b) is consistently critical across both environments, while initialisation scaling (c,d) shows greater importance in the complex Humanoid task but less impact in the simpler Dispersion environment.
HyperMARL is fully compatible with recurrent architectures essential under partial observability, and (ii) it has intrinsic bias toward specialisation and can converge to homogeneous behaviour when it is optimal (also shown with strong same-goal Navigation performance (Fig. 6ai) even with large observation spaces and many agents.
Summary. Our empirical results confirm HyperMARL effectively addresses both research questions. For Q1 (Specialisation), across Dispersion, MAMuJoCo, and Navigation, HyperMARL learned specialised policies, matched NoPS-level diversity and performance where FuPS+IID struggled, and scaled to complex, high-agent-count heterogeneous tasks. For Q2 (Homogeneity), HyperMARI. demonstrated competitive performance against strong FuPS baselines in SMAX and shared-goa Navigation, confirming its versatility,
Ablations and Embedding Analysis
6.1 Ablations: Gradient Decoupling and Initialisation Scaling
We ablate two critical components of HyperMARL: gradient decoupling (Sec. 4.3 and initialisation scaling (Sec. 4.2p. In HyperMARL w/o GD. the hypernetwork is conditioned on [o,, et], coupling observation and agent-II) gradients. In HyperMARI. w/o RF, we remove the reset fan-in/out scaling that aligns the scale of generated parameters (0',05 with standard initialisers.
Gradient decoupling is essential; initialisation scaling grows with complexity. Figure sshows that removing GD consistently degrades performance across both Humanoid-v2 (17 agents) and Dispersion, showing that GD is an essential component of HyperMARL. Removing RF reveals a task-dependent effect: it is critical on Humanoid-v2, consistent with hypernetwork initialisation results (8), but has a minor impact on Dispersion, indicating that principled initialisation becomes more vital with increased complexity. We provide additional ablations in App. H.7
6.2 Analysis of Learned Agent Embeddings
1.005
Recall from 4.2 that each agent i possesses embedding et, which serves as input to the hypernetworks. For the MLP hypernetworks, these embeddings are learned end-to-end and are orthogonally initialised. Consequently, at initialisation (step 0), the pairwise cosine distance between any two distinct agent embeddings is 1.0, mirroring the separability one-hot IDs.
1.00
6.95
We probe how these embeddings e' adapt in the 4-agent Nav- igation task (same task as Figure(H.5) under two objectives with identical dynamics: (i) same goal (all agents navigate to a single shared target) and (ii) different goals (each agent navigates to its own unique target). At the end of training, we compute the mean pairwise cosine distance between agents' embeddings (lower values imply greater similarity) and com- pare to the orthogonal baseline of 1.0. (Cosine distance cosine similarity; 0 = identical direction, 1 orthogonal,
Figure 9: Embedding similarity reflects task demands. Mean pairwise cosine distance (dashed line 1.0). Same goal: contrac- tion (0.882 + 0.042). Different goals: near-orthogonal (1.010 + 0.017). Embeddings contract when shared policy is optimal and main- tain separation for specialisation.

---

2 opposite). shows the per-seed distributions, with the 1.0 initialisation value as a dashed line.
Agent embeddings adapt to task demands. When behaviour should be homogeneous (same goal), embedding directions become significantly more aligned, the mean pairwise cosine distance contracts to 0.882 + 0.042 (one-sample t-test vs 1.0: p = 0.0079). Conversely, when behaviour must differ (different goals), the embeddings maintain their separation, remaining near their orthogonal initialisa- tion at 1.010 = 0.017. These results demonstrate that the hypemetwork actively modulates the agent embeddings based on the task, promoting alignment for homogeneity while preserving separabilityy for specialisation. We also conduct a sensitivity analysis on HyperMARL's hyperparameters in App F:3] and see that agent embedding size can be an important hyperparameter, one that could correspond to the task's 's diversity requirements.
7 Related Work
Hypernetworks in RL and MARL. Hypernetworks are effective single-agent RL for meta- learning, multi-task learning, and continual learning [3 4 42 [1] In MARL, QMIX 40 used hypernetwork (conditioned on global state) to mix per-agent Q-values; however, each agent's own network remained a standard GRU, Parallel work, CASH [66,, conditions hypernetwork on local observations and predefined capability descriptors to target zero-shot generalisation across heterogeneous action spaces. By contrast, action spaces, conditioning only on agent IDs/larred embeddings, and we explicitly decouple agent-conditioned from observation-conddiionn gradients mechanism absent in CASH which we find critical for specialisation.
agent-conditioned hypernetworks in homogeneous
Variants of Parameter Sharing. While Full Parameter Sharing (FuPS) is the most common approach, several other variants exist. Selective Parameter Sharing (SePS) [ shares weights between similar groups of agents, identified by clustering agent trajectories during a tr-iiiiiiig phase. Pruning methods (SNP-PS, Kalcidoscope) [24] 28) split single network into agent -gent-speeiiic subnetworks using learned agent masks. AdaPS [27] combines clustering and pruning masks to create shared policies for different groups of agents. Concurrent to our work, GradPS [38] identifies neurons with conflicting gradient updates, clones these neurons, and assigns each clone to group of agents with low gradient conflict. Unlike these works, HyperMARL does not rely on pre-training (SePS), clustering algorithms requiring a preset number of clusters (SePS, GradPS, AdaPS), intricate pruning hyperparameters (SNP-PS, Kaleidoscope), auxiliary diversity losses (Kaleidoscope), or gradient conflict thresholds (GradPS).
Learning Diverse Policies. Shared parameters often limit policy diversity [T]] 24 17 26 Proposed
solutions include: maximising mutual information between agent IDs and trajectories (26). rose-based methods that assign distinct roles to agents 488 49). best-response (39] methods and approaches that
structural modifications constraints induce diversity 24 6 28 27). Outside FuPS/NoPS,
HAPPO (54) uses a non-shared centralised critic with individual actors updated sequentially to learn heterogeneous behaviours. In contrast to these works, HyperMARL. does not alter the learning objective, use sequential updates or require preset diversity levels.
8 Conclusion
We investigated why standard parameter sharing fails at behavioural diversity, identifying that cross-agent gradient interference is critically exacerbated by coupling agent IDs with observations. We hypothesised that explicitly decoupling these gradients would enable adaptivity without prior complexities, and confirmed this using our HyperMARL approach. Our results show this decoupling enables adaptive behaviours (up to 30 agents) and linked to reduced policy gradient variance. These findings establish gradient decoupling via HyperMARL as versatile, principled approach for adaptive MARI.. We discuss limitations App. A most notably parameter count, which can be remedied by parameter-cfficient hypemnetorks (c.g., chunked variants [47 9).
10

---

Acknowledgements
We would like to thank Samuel Garcin, Max Tamborski, Dave Abel, Timothy Hospedales, Trevor Mcinroe, Elliot Fosong, and Aris Filos-Ratsikas for fruitful discussions early versions of this work. We also wish to acknowledge the anonymous reviewers for their constructive comments and feedback that helped strengthen this work.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 2021.
[2] Stefano V Albrecht, Filippos Christianos, and Lukas Schäfer. Mulli-agent reinforcement learning: Founda- tions and modern approaches. MIT Press, 2024.
[3] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. In Conference on Robot Learning, pages 1478- 1487. PMLR, 2023.
[4] Jacob Beck, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Recurrent hyperettwork are surprisingly strong in meta-rl. Advances in Neural Information Processing Systems, 36, 2024.
[5] Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems, 2022.
[6] Mattco Bettini, Ryan Kortvelesy, and Amanda Prorok. Controlling behavioral diversity in multi-agent reinforcement learning. In Forty-first International Conference on Machine Learning, 2024.
[7] Matteo Bettini, Ajay Shankar, and Amanda Prorok. System neural diversity: Mcasuring behavioral beterogeneity in multi-agent learning. arXiv preprint arXiv:2305.02133 2023.
[8] Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference Learning Representattinss 2020.
[9] Vinod Kumar Chauhan, Jiandong Zhou, Ghadeer Ghosheh, Soheila Molaei, and David Clifton. Dynamic inter-treatment information sharing for individualized treatment effects estimation. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Staristies, volume 238 of Proceedings of Machine Learning Research, pages 3529-3537. PMIR, 02-04 May 2024.
[10] Filippos Christianos, Georgios Papoudakis, and Stefano Albrecht. Pareto actor-critic for equilibrium selection in multi-agent reinforcement learning. Transactions on Machine Learning Research (TMLR), 2023.
[11] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pages 1989-1998. PMIR, 2021.
[12] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In 34th Conference Neural Information Processing Systems, 2020.
[13] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Tom, Mingfei Sun, and Shimon Whiteson. independent learming you need the starcraft multi ageti- challenge? arXiv preprins 1rXiv:2011.09553 2020.
[14] Dominik Maria Endres and Johannes E. Schindelin. A new metric for probability distributions. IEEE 2003.
Transactions on Information theory,
[15] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learnin Advances in neural information processing systems, 29, 2016.
[16] Kevin Fu, Pierce Howell, Shalin Jain, and Harish Ravichandar. Learning flexible heterogeneous coordina- tion with capability aware shared hypernetworks. arXiv preprint arXiv:2501.0m 06058, 2025.
11

---

[17] Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 6863-6877 PMLR, 2022.
[18] Jayesh Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Pupers, São Paulo, Brazil, May 8-12. 2017, Revised Selected Pupers 16, pages 66 83. Springer, 2017.
[19] David Ha, Andrew Dai, and Quoe V Lc. Hypernetworks. arXiv preprint 1r0iv: 2016.
[20] Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):399 366, 1989.
[21] Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforce- ment learning with hypernetworks. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages 799-805. IEEE, 2021.
[22] Jiechuan Jiang and and Zongging Lu. The emergence of individuality. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 4992-5001. PMLR, 18-24 Jul 2021.
[23] R Kassen. The experimental evolution specialists, generalists, and the maintenance diversity. Journal of evolutionary biology, 173-9173--19, 2002.
[24] Woojun Kim and Youngchul Sung. Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.cm 00912, 2023.
[25] Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Haifeng Zhang. David Mguni, Jun Wang. Yaodong Yang, et al. Settling the variance of ageti- policy gradients. Advances in Neural Information Processing Systems, 34: 13458- 13470, 2021.
[26] Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991-4000 1-002, 2021.
[27] Dapeng Li, Na Lou, Bin Zhang, Zhiwei Xu, and Guoliang Fan. Adaptive parameter sharing for multi-agent reinforcement learning. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Procesing (ICASSP), pages 6035-6039. IEFF, 2024.
[28] Xinran Li, Ling Pan, and Jun Zhang. Kalcidoscope: Learnable masks for heterogencous multi-agent reinforcement learning. arXiv preprint arXiv:2410.085400 2024.
[29] Jianhua Lin. Divergence measures based the shannon entropy. IEEE Transactions Information theory, 37(1):145- 151, 1991
[30] Ryan Lowe, Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiiee environments. Advances in neural information processing systems, 30, 2017.
[31] Kevin R McKee, Joel Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):21, 2022.
[32] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pages 1928-1937. PmLR, 2016.
[33] Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypernetworks. arXiv preprint arXiv:2010.04104, 2020.
[34] Frans A Olichoek and Christopher Amato. A concise introduction to decentralized POMDPs. Springer. 2016.
[35] Martin J Osborne and Ariel Rubinstein. course in game theory. MIT press, 1994.
[36] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research. 25(318):1-63 2024.
12

---

[37] Bei Peng, Tabish Rashid. Christian Schroeder de Witt. Pierre-Alexandre Kamienny, Philip Torr Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neurul Information Processing Systems, 2202-1222,, 2021.
[38] Haoyuan Qin, Zhengzhu Liu, Chenxing Lin, Chennan Ma. Songzhu Mei, Siqi Shen, and Cheng Wang. Gradps: Resolving futile neurons in parameter sharing network for multi-agent reinforcement learning, In Forty-second International Conference on Machine Learning, 2025.
[39] Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V. Albrecht. Generating teammates for training robust ad hoc teamwork agents via best-response diversity. Transactions on Machine Learning Research (TMLR), 2023.
[40] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-55 51, 2020.
[41] Alexander Rutherford. Benjamin Ellis. Matteo Gallici, Jonathan Cook, Andrei Lupu, Garôar Ingvarsson. Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments and algorithms in jax. Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2444-2446, 2024.
[42] Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning, pages 9301-9312. PMLR, 2021.
[43] Chris R Smith, Amy Toth, Andrew V Suarez, and Gene Robinson. Genetic and genomic analyses of the division labour insect societies. Nature Reviews Genetics, 9(10):335-745 2008.
[44] James Surowiecki. The Wisdom of Crowds. Doubleday, New York, 2004.
[45] Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330-337, 1993.
[46] Leonid Nisonovich Vaserstein. Markov processes denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 72, 1969.
[47] Johannes Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypereeworks. In International Conference on Learning Representations, 2020.
[48] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning. pages 1876-9886, 2020.
[49] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng. Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.
[S0] Katherine Y Williams and Charles A O' O'Reilly III. Demography and. Research in organizational behavior, 20:77-140, 1998.
[51] Ronald J Williams. Simple statistical gradient-follovin algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
[52] Anita Williams Woolley, Ishani Aggarwal, and Thomas Malone, Collective intelligence and group performance. Current Directions in Psychological Science, 24(6):420-44 424, 2015.
[53] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 142224, 2022.
[54] Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneous- agent reinforcement learning. Journal of Machine Learning Research, 25(32): 2024.
13

---

A Limitations
Hypernetworks generate weights for target networks, which can lead to high-dimensional outputs and many parameters for deep target networks, particularly when using MLP- hypernetworks. While HyperMARL uses more parameters than NoPS and FuPS for few agents, it scales almost constantly with agent count, an attractive property for large-scale MARL. Parameter efficiency could be improved through chunking techniques [47] 91, or low-rank weight approximations. This parameter overhead is often acceptable in RL/MARL given typically smaller actor-critic networks, and HyperMARL's favorable agent scaling (see App.
B Broader Impact
This paper presents work whose goal is to advance the field of Multi-Agent Reinforcement Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
C Specialised Policies and Environments
Specialisation plays a key role in MARL, yet remains under-defined, so we define specialised enviromments and specialised policies.
Definition 1. An environment is specialised if the following both hold:
1. Distinct Agent Policies. The optimal joint policy π² consists of at least two distinct agent policies, i.e., j E such that n² + n².
Non-Interchangeabliiy. Any permutation the policies n², denoted as πτ, results in weakly lower expected return:
En-r> [G(h)] S En [G())]
with strict inequality if the joint policies are non-symmetric (i.e., swapping any individual policy degrades performance).
For example, consider specialised environment such football game, optimal team performance typically requires players in distinct roles (e.g., "attackers," "defenders"). Permuting these roles (i.e.. exchanging their policies) would typically lead to suboptimal results. Here, agents develop specialised policies by learning distinct, complementary behaviours essential for an optimal joint policy. While agents with heterogeneous capabilities (e.g., different action spaces) inherently specialised, homogeneous agents can also learn distinct policies. Such environments are analysed in Sections E. 5.2
D
Measuring Behavioural Diversity
D.1
Quantifying Team Diversity
We quantify policy diversity using System Neural Diversity (SND) diversity based on differences in policy outputs:
7 which measures behavioural
SND (1x1)ee)) n(n 1)
2
(p((
(5)
E1
where n is the number of agents, O is a set of observations typically collected via policy rollouts, (o;) and T² (or) are the outputs of policies and j for observation o,, and D is our distance function between two probability distributions.
In contrast to metric 46 shown Appendix D.2 JSD is robust metric for both continuous and discrete cases, and provides more reliable measure of policy distance.
7
we use
Distance (JSD) 14
29
D, rather than the Wasserstein
14

---

D.2 Finding a Suitable Distance Function for Policy Diversity
The choice of distance function D in Equation 5 crucial for accurately measuring policy diversity. In MARL, policies are often represented as probability distributions over actions, making the choice of distance function non-trivial.
6 use the Wasserstein metric for continuous policies [46] as distance function D, while 31 use the total variation distance for discrete policies. For discrete policies, Wasserstein distance would require a cost function representing the cost of changing from one action to another, which might not be trivial to come up with. On the other hand, although well-suited for discrete policies, TVD might miss changes in action probabilities because it measures the largest difference assigned to an event (i.e. action) between two probability distributions.
We consider a simple example to illustrate this point. Suppose we have two policies x² and x² with action probabilities as shown in Figure I0 stays constant, while x² changes gradually over timesteps. We see that even as π² changes over time, the TVD( x²)) between 'T' and x remains constant. This is because TVD only measures the largest difference between the distributions, and does not consider the overall difference between them. On the other hand, the Gensen-Shannon distance (ISD) [1]. which is the square root of the Jensen-Shannon divergence, does not have this problem as it is smooth distance function. Furthermore, it satisfies the conditions for being a metric it is non-negative, symmetric, and it satisfies the triangle inequality.
For continuous policies, as shown in Figure [T, JSD exhibits similar trends to the Wasserstein distance. Since JSD is reasonable metric for both continuous and discrete probability distributions, we will use it as the distance metric for all experiments and propose it as a suitable distance function for measuring policy diversity in MARL.
We also summarise the properties of the various distance metrics in Table
1.8
(Constant)
[Changing over Time!
Dsta nota between Parices Dtel.rt
0.700
0.775
0.50
3.55
6225
-8-
150
3.
00
23
Figure 10: Gradual changes in 2 result in gradual changes in the Sensen-Shannor distance (JSD), while the Total Variation Distance (TVD) can miss changes in action probabilities.
(Constarrt
(Converging to
Distances between Policies Die..a
2.4
0.44
Inobold
0.2
33
Mean
Mean
Time Stece
Figure 11: Sensen-Shannen distance (JSD) trends similarly to Wasserstein distance when we have continuous policies.
15

---

Method
Kinds of Actions
Metric
Smooth
Formula
Wasserstein Distance
46
Continuous
Metric
Yes
W(p.9)
(inf Jaxa l d)(a,
Total Variation Distance
Discrete
Metrie
(V(p.)) E ((()) [(m)
Jensen Shannon Divergence
Both
Divergence
JSDIp 9) Dxclp
ma) + R2 my
Jensea-Shannee Distance
Both
Metric
<JSD(p 11 g)
Table Measures for Policy Diversity
16

---

E
Specialisation and Synchronisation Games
To study the challenges of specialisation and coordination in isolated setting, introduce the Specialisation and Synchronisation Games, drawing inspiration from version of the XOR game [17], VMAS's Dispersion S and coordination and anti-coordination games in game theory [35]. These environments encourage agents to take distinct actions (Specialisation) or take identical actions (Synchronisation). Despite their deceptively simple payoff' structure, these games present substantial learning challenges non-stationary reward distributions driven by others' adapting behaviours and in their temporal extension, the joint observation spaces grows exponentially with the number of
agents.
E.1 Specialisation and Synchronisation Games Description
Specialisation Game. Agents are encouraged to choose distinct actions. In the simplest setting, it a two-player matrix game where each agent selects between two actions (A or B). As shown in Figure Zal agents receive payoff of 1.0 when their actions differ (creating two pure Nash equilibria on the anti-diagonal) and 0.5 when they match. This structure satisfies Definition[] [ since optimal joint policies require complementary, non-identical strategies. There is also a symmetric mixed-strategy equilibrium which each agent plays and with probability 0.5.
Synchronisation Game. Conversely, agents are encouraged to coordinate and choose identical actions. The payoff matrix inverts the Specialisation game's 's structure, agents receive for matching actions and 0.5 for differing ones. This creates two pure Nash equilibria along the diagonal of the payoff matrix (Figure 26). and incentivises uniform behaviour across agents.
N-Agent Extension. Both games naturally scale to n agents and possible actions. In Specialisation, unique actions receive a payoff of 1.0, while selecting the same action receives payoffs of f. where is the number of agents choosing that action. In contrast, in Synchronisation, agents receive maximum payoffs (1.0) only when all actions match. For partial coordination, rewards follow a hyperbolic scale, x-k+I. encouraging agents to align their choices. Visualisations in Figure 2 and detailed reward
profiles appear in Figure 12
E.2 General-n Payoff Definitions
Both games generalise naturally agents and possible actions. We show the reward profiles for n = agents in Figure [2]
Let a = (a'...., a²) E {I.....n] n and k2 = | a² = a} be the joint action profile and the count of agents choosing action a, respectively.
Temporal Extension. To model sequential decision makinng we extend each normal-form game into repeated Markov game, where the state at time & the joint action time i At each step / all agents observe a' -'- choose at, and receive the original Specialisation or Synchronisation payoff. This repeated setup isolates how agents adapt based past joint actions, exposing temporal patterns of specialisation and coordination.
E.2.1 Specialisation Game
The reward is formulated as follows:
1.0,
if ka = 1 (unique action);
T sppo (a)
if R (shared action).
E.2.2 Synchronisation Game
The reward is formulated as follows:
sync (a)
kig
so that r rynn = 1.0 when kg = n (all agents select the same action), and otherwise follows hyperbolic scale encouraging consensus.
17

---

Reward Profiles (n=5)
1.00
1.0
1.00
0.8
Reward
0.50
0.50
0.33
0.33
0.25
0.20
0.2
0.25
Specialisation
Synchronisation
0.0
Number of Agents Choosing Same Action
Figure 12: Reward profiles for the Specialisation (blue) and Synchronisation (orange) games with agents. In the Specialisation game, an agent's payoff peaks when it selects unique action, and then decays as when actions are shared. the Synchronisation game, payoffs follow a hyperbolic scale 1/(n k 1), reaching maximum only under full consensus, thereby incentivising aligmment.
E.3 Proof that FuPS cannot represent the optimal policy in the two-player Specialisation Game
Theorem 1. A stochastic, shared policy without agent IDs cannot learn the optimal behaviour for the two-player Specialisution Game.
Proof. Let π be shared policy for both agents, and let P(a, 0) represent the probability of any agent choosing action O. The expected return of π for each agent is:
[[R(T)]
P(a, = 0, = - = 0, a1 = =1)
(6)
+ P(au 1,0, 0) 1+ F(a, 1, =1)
(7)
0.5a² 2a:11 a) + 0.5(1 a)
(8)
=aa2+a+0s
(9)
-a 0.5) + 0.75
(10)
Thus, [[](T)] S 0.75 for all [0, 1], with the maximum at a 0.5. Therefore, shared policy cannot achieve the optimal return of 1, confirming the need for specialised behaviour to optimise rewards.
E.4 Measuring Agent Gradient Conflict
measure gradient conflicts, via cosine similarity between agents' gradients cos (0)
())
(g
g
where gi (8) (O..
lo
E.5
Empirical Results in N-player Specialisation and Synchronisation Normal-Form Game
To assess this limitations of FuPS and NoPS in practice, we compare three REINFORCE [51 variants in both games with 2, 4. 8, 16, 32 agents: NoPS (No Parameter Sharing), FuPS (Fully Parameter Sharing), and FuPS+ID (FuPS with one-hot agent IDs). All policies use single-layer neural networks with controlled parameter counts (see Appendix I for details).
18

---

Specialisation Game
Synchronisation Game
1.0
Avg. Evaluation Reward
0.8
(Mean = Std. Error)
0.4
0.2
PG- NoPS
PG FuPS
IIPS+ID
0.0
32
Number of Agents Actions
Number Agents Actions
Figure 13: Mean evaluation reward (mean + standard error) as a function of the number of agents/actions in the Specialisation (left) and Synchronisation (right) games. In the Specialisation game, vanilla policy gradients (PG, i.. REINFORCE) with FuPS collapse as the team grows, whereas identity-awart variant (PG-FuPS+ID) retains near-optimal performance. In the Synchronisation game, PG-NoPS performs well at small scales but degrades with more agents, while both PG-FuPS and PG-FuPS+ID remain at optimal reward across all scales.
19

---

Algorithm HyperMARL
1: Input: Number of agents n, number of training iterations K, MARL algorithm parameters (e.g.. MAPPO-specífic hyperparameters)
2: Initialise:
3: Hypemnetwrk parameters v [Ensure 0 and ø' follow standard initialization schemes, c.g..
orthogonal|
4: Agent embeddings (e [e (One-hot or orthogonally initialised learnable parameters)
5: Output: Optimized joint policy π
6: for each training iteration & 0, 1, .......K K
do
7:
for each agent i i= 1...... do ho (e)) (Policy parameters)
8:
9:
hh (e') {Critic parameters]
end for
11: Interact with environment using {TT. } to collect trajectories D
Compute shared loss C from using (V 1 (Standard RL loss function)
13: Update vi, and by minimizing L (Optimise parameters.)
end for
15:
Return = , ,..., - T")
HyperMARL Details
F1 HyperMARL Pseudocode
In Algorithm [ we present the pseudocode for HyperMARL, with HyperMARL-specifiee steps highlighted in blue. HyperMARL leverages hypernetworks to dynamically generate the parameters of both actor and critic networks. The weights of the hypernetworks and the agent embeddings automatically updated through automatic differentiation (autograd) based the computed loss. Additionally, Figure provides a visual representation of the HyperMARI. architecture. archhteecturr
F.2 Variance of the HyperMARL Gradient Estimator
Unbiased estimator. Following from Equation we can write the unbiased estimator for Hyper- MARLS gradient as follows:
9HM Vonn(ee)
A(h
(b)
a
(b)
Vo: a log Te.
(a (.(0) h (1((0)))
Σ J, Z,
2,
(HM')
where B trajectories { } are sampled i.i.d. and 2, is the empirical analogue of the observation-
conditioned factor.
Assumptions. (A1) trajectories are i.i.d.; (A2) second moments are finite; (A3) vh 0, e are fixed during the backward pass.
Variance expansion. covariance:
Since each J; deterministic under (A3), we may factor them outside the
Var (GHns) Cov Σ J,Zi, 2,
(by def. Var(X) = Cov(X, X))
) Σ Cov(JJ 2,)
(bilinearity of Cov)
= -E, Cov 2g 2,, J.
(pull deterministic matrices out of Cov)
(11)
20

---

Equation (1) makes explicit that all snpeector-iinducer covariance is captured Cov( 2, 2,), while the agent-conditioned Jacobians J, remain trajectory noise-free.
Mini-batch update and covariance. Let 2, be the unbiased mini-batch estimate Z; and TAm Σ; J,Z, the stochastic update. Because every J, deterministic (wrt. to mini-batch),
Var(Tim)) EJ. Cov( Z, 2z) J/
(12)
shows that HyperMARL first averages noise within each agent (7,) and only then applies J, FuPS+ID, by contrast, updates the shared weights 0 with every sample A Vo log me id], leaving observation noise and agent ID) entangled and making susceptible to gradient interference
(derivation in Appendix F.2
Equation
12)
E,3 Scalability and Parameter Efficiency
Hypernetworks generate weights for the target network, which lead high-dimensional out- puts and many parameters for deep target networks. This challenge is amplified in MIPP-aasd hypernetworks, which include additional hidden layers. Figure 14 shows scaling trends:
NoPS and linear hypernetworks: Parameter count grows linearly with the number of agents.
FuPS: More efficient, growth depends one-hot vector size.
MLP hypernetworks: Scale better with larger populations, since they only require embed- dings of fixed size for each new agent.
To reduce parameter count, techniques like shared hypernetworks, chunked hypernetworks 47 or producing low-rank weight approximations, can be used. While naive implementations are parameter- intensive, this might be less critical in RL and MARL which commonly have smaller actor-critic networks. Moreover, HyperMARL's near-constant scaling with agents suggests strong potential for large-scale MARL applications.
Scaling of IPPO Dispersion
-o-Pur's
One Hox
10M
Linerr Ulpprnets One Lot
-- MLP Uppernets Emheed
O NoPS
Nutabe of Parameters
10K
16
64
256
1024
Number of Agents
Parameter scaling for IPPO variants with increasing agents (4 to 1024). MLP Hypernet- works scale nearly constantly, while NoPS, Linear Hypernetworks, and FuPS+One- Hot grow linearly. Log scale on both axes.
Figure 14:
To isolate the effects of parameter count, we scaled the FuPS networks (Figure 15 to match the number of trainable parameters in HyperMARI. Despite generating 10x smaller networks, HyperMARI. consistently outperforms FuPS variants, showing its advantages extend beyond parameter count.
21

---

IQM
MAPPO-MLP Hypernetwork IPPO-MLP Hypernetwork
MAPPO-FUPS IPPO-FUPS
0.0
0.3
0.6
0.0
0.3
0.6
Mean Episode Return
Figure 15: Dispersion performance with four agents. FuPS variants match HyperMARL. in parameter count but still underperform.
F.4 Speed and Memory Usage
We examine the computational efficiency of HyperMARL compared to NoPS and FuPS by measuring inference speed (Figure (6a) and GPU memory usage (Figure 16b) we scale the number of agents. The benchmarks were conducted using JAX based) policy architecture. All experiments used fixed network sizes (64-dimensional embeddings and hidden layers) with batch size of 1 128 and 64 parallel environments, allowing to isolate the effects of varying agent count. Each measurement represents the average of 100 forward passes per configuration with operations repeated across 10 independent trials.
single NVIDIA GPU (T4) with recurrent (GRU-
Recurrent IPPO Forward Pass Speed Number of Agents (batch 125)
Rocarcent IPPO CPU Memory Usage Number of
12000
FiPS One Ii4
MLP Hojemeta
SRP Reyarto
Expereet but's
Uhogee (MT))
10000
Leae Eopertate
Porward Pas Thas (ax))
60
30
40
8000
30
4000
GP1
10
200
Numzher Ageans
Number of Agents
(a) Forward pass inference time
(b) GPU memory usage
Figure 16: Computational efficiency scaling with number of agents. HyperMARL offers a balance between NoPS and FuPS. Notably, in real-world deployments, NoPS incurs additional and synchronisation costs not reflected here, further widening the efficiency gap.
data transfer
The results demonstrate that HyperMARL offers a balance between the extremes represented by NoPS and FuPS. In practice, NoPS incurs additional data transfer and update costs, widening the efficiency gap.
F5 Sensitivity Analysis
We run a sensitivity analysis on the HyperMARL's hyperparameters in the 20-agent SMAX scenario with recurrent MAPPO, challenging setting for optimisation. The results are shown Table
Findings.
Similarly FuPS, and in general, deep RL methods like PPO, the learning rate is an important parameter for performance.
Agent embedding size an important hyperparameter. In these results, a smaller embedding size outperformed larger one. This could be due the homogeneous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence similar behaviours. This suggests the embedding size could correspond to the task's diversity requirements.
Other parameters, such as the width, have limited impact beyond a modest size.
22

---

Table 5: IQM and 95% CI of mean win rate across 5 seeds for the 20-agent SMAX scenario with recurrent MAPPO. Bold indicates the highest IQM score; * iindicates scores whose 95% confidence intervals overlap with the highest score.
Method
Embedding Dim
Hidden Dims
LR 0.0001
LR 0.0003
LR 0,0005
PaPS
NAA
N/A
0,1067 (0.0513, 0.2128)
0.3053 (0.2041, 0.4769)
*0.4213 (0,3623, 0.4800)
16
0.0799 (0.0250, 1.1
0.2946 (0. 1141, 0.3778)
0.4455 (0.3443, 0.5692)
HyperMARL
0.1233 (0.0750, 0.2162)
0.3338 (0.2692, 0.4286)
*0.3765 (0.3281. 0.4107)
128
0,1147 (0.0270, 0.1842)
*0,3787 (0.3509, 0.4603)
0,3327 (0.2931, 0.3548)
64,
0.1781 (0.1628, 0.2075)
*0.3498 (0.2807, 0.4500)
0.1944 (0. (0.1268, 0.3077)
0.1109 (0.0256, 0.2093)
0.3198 (0.2353, 0.3934)
*0.3483
HyperMARL
64
64
0,1360 (0.0303, 0.2821)
0.1191 00.0517, 0.1964)
0,1155 00.0794, 1500)
128
0.1193 (0.0750. 0.1579)
0.1036 (0.0517. 0.1594)
0.1 (0. 00.0455,00 0.1587)
64, 64
0.1353 (0.0541. 0.1818)
0.1283 00.0526. 0.2258)
0.0860 (0.0345, 0.1343)
23

---

G Experiment Details
G.1 Environments
(a) Dispersion.
(b) Navigation.
(c) MAMuJoCo.
(d) SMAX.
(c) BPS.
Figure 17: Multi Agett- enviromments used in our experiments.
Dispersion (VMAS) 5:: 2D environment where four agents collect unique food particles. This task requires specialised heterogeneous behaviours and resembles the Specialisation Game from Section E.
Navigation (VMAS) [5] Agents navigate in 2D space to assigned goals, receiving dense rewards based on proximity. Configurations include shared goals (homogeneous), unique goals (heteroge- neous), and mixed goals, where some agents share goals while others have unique ones.
Multi-Agent MuJoCo (MAMuJoCo) 37 A multi-agent extension of MuJoCo, where robot body parts (e.g., cheetah's 's legs) are modelled as different agents. Agents coordinate to perform efficient motion, requiring heterogeneous policies [54]
SMAX (JaxMARL) [1]]: Discrete action tasks with 2 to 20 agents on SMACv1- and SMACv2-style
maps. FuPS haselines have been shown optimal for these settings 53 17 indicating homogeneous behaviour is preferred here.
Blind-Particle Spread (BPS) 11 Blind-Particle Spread (BPS) is modified variant of the Multi-
Agent Particle Environment (MPE) Simple Spread task (30] with 15 to 30 agents. The environment contains landmarks of different colours, and each agent is assigned colour. Agents are 'blind' as they cannot observe their own assigned colour or the colours other agents. They must infer correct landmark navigate towards based only their observations and rewards. The number of colours thus dictates the number of distinct roles or behaviours the team must learn to successfully complete the task.
G.2 HyperMARL Architecture Details
For the Dispersion and Navigation results (Sec. 5.2 we use feedforward architectures, where we use HyperMARL to generate both the actor and critic networks. For the MAPPO experiments in Section 5.2 for fairness comparisons with HAPPO and MAPPO, we maintain the centralised critic conditioned on the global state and only use HyperMARL generate the weights of the actors. For the recurrent IPPO experiments Section 5.3 HyperMARI. only generates the actor and critic feedforward weights, not the GRU weights.
24

---

Table 6: Baseline Methods Selection and Justification. Selected methods (V) were chosen based in MARL, while excluded
on their relevance to parameter sharing and ppecialisation/eeneraiistt methods (X) did not align with our research objectives or had implementation constraints. Our experimental design systematically addresses key questions on agent specialisation and ho- mogeneity, therefore we selected baselines with demonstrated strong performance in their respective settings, ensuring fair and rigorous comparison.
Method
Category
Selected
Justification & Experimental Settings
IPPO
(NoPS, FuPS+1D)
NoPS/FuPS
Established MARL baseline implementing both independent (NoPS) and fully shared (FuPS+ID) policy configurations. Texied in: Dispersion, Navigation, SMAX (two SMACv] maps and two SMACv2 maps, with 10 and 20 agents).
MAPPO S
(NoPS, FuPS+ID)
NoPS/FuPS
Strong baseline with centralized critics for both NoPS and FuPS+ID architectures. Texted in: Dispersion. MAMuJoCo, SMAX (two SMACvI maps and two SMACv2 maps, with 10 and 20 agents).
DiCo 6
Architectural Diversity
Provides comparison with method employing preset diversity levels that balances shared and non-shared parameters. Tested Dispersion Navigation (as per original paper). Original hyperparameters used for n 2 agents; parameter sweeps conducted for > 2 to identify optimal diversity levels.
НАРPO 54
Sequential Updates
Enables comparison with a method designed for heterogeneous behaviours using sequential policy updates with agent-specific parameters. Tested in: MAMuJoCo, selecting 4/6 scenarios from the original paper, including the challenging 17-agent humanoid task. Walker and Hopper variants were excluded as MAPPO and HAPPO performed similarly in these environments.
Kaleidoscope 28
Architectural Pruning
Implemented for off- off-policy evaluation using its MATD3 imple- mentation with tuned MaMuJoCo hyperparameters. Tested in: MAMuJoCo environments Ant-v2, HalfChectah-v2, Walker2d- (overlapping with our IPPO experiments), and Swimmer- v2-10x2 (highest agent count variant). Included to evaluate HyperMARL's competitiveness against a method with ensemble critics and diversity loss, in off-policy setting.
SePS 1I
Selective
Although this requires pre-training phase, is strong bascline for parameter sharing approaches. Tested in: Blind-Particle Spread environments, with 15 30 agents.
Param-
Sharing
SEAC 12
Shared
Focuses primarily on experience sharing rather than parameter sharing architecture, falling outside our research scope.
Experi-
ence
CDAS 26
Intrinsic
X Only implemented for off off-policy methods and has been demon- strated to underperform FuPS/NoPS architectures [17], making it less suitable for our primary on-policy comparisons. Shows limited practical performance advantages in comparative studies [12) suggesting other baselines provide rigorous comparison points.
Reward
ROMA/RODE
48
Role-
X
49
based
SNP-PS 24
Architectural Pruning
publicly available implementation, preventing direct, repro- ducible comparison.
G.2.1 Training and Evaluation
Training:
For Dispersion
5.2g
we run 10 seeds and train for 20 million timesteps.
SMAX 5.3) and MaMuloCo (5.2). we run seeds and train for 10 million timesteps, consistent with the basclines.
For Navigation
25

---

For Spread (BPS), we run 5 seeds and train for 20 million timesteps. consistent with baselines.
Evaluation:
For Dispersion
evaluation is performed every 100k timesteps across 32 episodes. following the baselines, evaluation is performed every 120k
For Navigation
timesteps across 200 episodes.
For SMAX (5.3) evaluation is performed every 500k timesteps across 32 episodes.
For MaMuJoCo 5.2)
following the baselines, evaluation is performed every 25 training
episodes over 40 episodes.
G.2.2 Measuring Policy Diversity Details
We measure team diversity using the System Neural Diversity (SND) metric (Equation S 7 details Section[D with censen-Shannon distance. SND ranges from (identical policies across all agents) to (maximum diversity). We collect dataset of observations from OPPO-NoPS and IPPO-FIS PPO- policies checkpointed at 5 and 20 million training steps. Each policy is rolled out for 10,000 episodes, generating 16 million observations. We then sample million observations from this dataset calculate the SND for each method tested.
26

---

Table 7: Mean episode return MAMuJoCo for off-po MATD3 variants. IQM of the mean episode returns with 95% stratified bootstrap CL. Bold indicates the highest IQM score; indicates scores whose confidence intervals overlap with the highest. Although Kaleidoscope employs an ensemble of five critics and an explicit diversity loss, HyperMARI. (using a standard MATD3 setup with two crities) achieves competitive results without these additional mechanisms.
Environment
Ind. Actors, Shared Critic
Kaleldoscope
HyperMARL.
Ant-v2
$270.38 (4329.73, 73, 5719.78)
6160.70 3798.02. 6443.83)
$8866.88 (5840.00, 5920.66) 7057.44 (3508, 70, 8818.11)
HalfCheetah-v2
*6777.04 (3169.11 11, 8233,94)
*6901 (3609.73, 73, 8192.38)
Walker2d-v2
L777 (5 54,4,84, 8103.34)
*6664.32 (5408.95, 8828.11
7057.68 (5976.50, 8166.09)
Swimmer-v2-10x2
+453.74 447,24, 487.86)
+462.48 (444.22, 475.64)
465.91 (410.82, 475.77)
H Detailed Results
H.1
Comparison with Kaleidoscope using Off-Policy - MATD3
Our comparison with Kaleidoscope (28] mentioned Section 5 is conducted using off-policy methods due to its original design. Kaleidoscope incorporates intricate mechanisms (e.g.. learnable masks, an ensemble five critics, a specific diversity loss) and numerous specialised hyperparameters (e.g., for critic ensembling: "critic deque len', "critic fie__oeef,, 'reset interval"; for mask and threshold parameters: 'n masks', 'threshold init scale", 'threshold init bias", 'weighted _makss", ; 'sparsity layer weights', etc.). Porting this full complexity an on-policy PPO backbone would constitute a significant research deviation rather than a direct benchmark of the established method.
Therefore, we utilised Kaleidoscope's original off-cy implementation to ensure a meaningful comparison. We adopted MATD3 as the algorithmic backbone for this evaluation, as was the only publicly available Kaleidoscope variant with tuned hyperparameters for Multi-Agent MuJoCo (MaMuJoCo). The MaMuJoCo tasks were chosen for alignment with our primary on-policy (IPPO) results and Kaleidoscope's 's original evaluation, specifically: Ant-v2, HalfCheetah-v2, Walker2d-v2 (overlapping with our IPPO experiments), and Swimmer-v22110
( which represents the MaMuJoCo variant with the highest number of agents). Comparative results in Table 7 that HyperMARL achieves competitive results with Kaleidoscope, while only using two critics (standard MATD3) and without additional diversity objectives.
H.2 Comparison with Selective Parameter Sharing (SePS) A2C
We benchmark HyperMARL against Selective Parameter Sharing (SePS) a method that shares weights among pre-determined groups of similar agents. SePS identifies these groups by clustering agent trajectories via an autoencoder during a pre-training phase. Following the original authors' setup, we use their provided source code and hyperparameters, evaluating all A2C [32] variants on the Blind Particle Spread (BPS) benchmarks (v1-4), which scale up to 30 agents. For HyperMARL, we sweep the learning rates (Ie-4, 3e-4) and hypernetwor hidden dimension (16, 64) per scenario.
As shown in Table 8 HyperMARL demonstrates competitive performance against both NoPS and SePS in these challenging settings, which feature up to 30 agents and five distinct agent roles. Crucially, HyperMARL achieves this result without resorting to agent-specific weights (like NoPS) or requiring pre-training and clustering pipeline (like SePS).
Table Mean episode reward in Blind Particle Spread for A2C variants. We report the IQM of the final total reward (with 95% confidence intervals) across 5 seeds on the BPS environments. Bold indicates the highest IQM score; indicates scores whose 95% confidence intervals overlap with the highest score. HyperMARL achieves results competitive with SePS without requiring a pre-training phase or explicit agent clustering.
Eavtronmentt
NoPS
FuPS-ID
SePS
HyperMARI.
BPS-1 upents, groupe, 5-5-5)
(-227777
-228.2 -313.3)
+2201 0:22144
-166
(-22.,
-178.5
BPS-2 (30 agents. 3 groups, 10-10-10)
4.15.4 (-459.4, -36.3)
.479.7 (-507.11 -411.7
+-407 (-453.7. 7.53777788.66
.297.8 (-43377 376.6)
BPS-3 (30 agents, 5 groups. 6 6-6-6-6-6)
-40.44 (-421.0, -381.0
(-14655. -534.)
+-422.1
(-466.2 (37.6)
+.17.77 (-48.1. 331.1)
BPS-4 (30 agents, 5 unoven grops, 2-2-2-15-9)
110.8 (-136.6,
(-1641.1
(-57.6, -36.00
27

---

H.3 Dispersion Detailed Results
H.3.1
Interval Estimates Dispersion
IQM
Mean
IQM
Mean
Lin. Hyper.
Lin. Hyper. NOPS
NOPS
0..4 0.0 0.4 D. D.D
0.4 0.8
050
Mean Episode Return
Mean Episode Retum
(a) IPPO
(b) MAPPO
Figure 18: Performance of IPPO and MAPPO on Dispersion after 20 million timesteps. We show the Interquartile Mean (IQM) of the Mean Episode Return and the 95% Stratified Bootstrap Confidence Intervals (CI) using (T. Hypernetworks achieve comparable performance to NoPS, while FuPS struggle with specialisation.
0.75
IQM Mean Episode Return
0.50
0.25
0.00
-0.25
-0.50
0
Training Steps
MLP Hypernetwork
Figure 19: We see that even if we run MAPPO- FuPS on Dispersion for 40 million timesteps (double the timesteps MLP Hypernetwork), it converges to suboptimal performance.
H.4 Detailed MAMujoco Plots
28

---

MAPPO-FUPS
MAPPO (Independent Actors)
HyperMARL (MAPPO)
HAPPO
1QM Mean Episode Return
7000
6000
6000
5000
1QM Mean Episode
4000
4000
3000
2000
2000
1000
1000
0
Training Steps (M)
Training Steps (M)
(a) Humanoid-v2 17x1
(h) Walker2d-v2 2x3
1QM Mean Episode Return
7000
1QM Episode Return
6000
6000
5000
5000
4000
3000
3000
2000
2000
1000
Training Steps (M)
Training Steps (M)
(c) HalfChectah-v2 2x3
(d) Ant-v2 4x2
Figure 20: Performance Recurrent IPPO and MAPPO on MaMoJoCo. HyperMARL, performs comparably to these baselines, and is the only method with shared actors to demonstrate stable learning in the notoriously difficult 17-agent Humamoid environment.
29

---

H.5
Detailed Navigation Plots
(a) 8 Agents, Same Goal
(b) 8 Agents, Different Goals
(c) 8 Agents, Four Goals
Britee Stppri
Trntec Seanio)
Britee Strur
(d) Agents, Same Goal
(c) 4 Agents, Different Goals
(f) 4 Agents, Two Goals
(g) Agents, Same Goal
(h) 2 Agents, Different Goals
FuPS
NoPS
MLP Hyper.
Lin. Hyper.
DiCo
Figure 21: Sample efficiency of IPPO variants in the VMAS Navigation environment. Plots show IQM and 95% CI (shaded regions) of mean episode return against training steps for different agent counts (rows: 8, 4, 2 agents) and goal configurations (columns, where applicable: Same, Different, Specific Goal Counts). Legend shown at bottom applies to all subplots.
H.6 Interval Estimates SMAX
H.7 Additional Ablations
30

---

IQM
Mean
IQM
Meari
HYBorMARL HyperMARL (YperMMARL (IPPO ANN
RYpErMARE TIPPO TIPP RNNN MAPPO IPPO RNN (FUPS)
IAPO RAIN (PUPS)
0.960.991.02 1.05 0.900.99..21.05
0.720.800.880.99
0.80 0.89 0.96
Episode
Fpisode
2x33
(h) 3s557.
IOM
Meon
10M
Mean
HyperMARL
HyperMARL (IPPO [IPPO BNIN
HyperMARE MARL (IPPO ANN IPPO ANN (FUPS)
IPPO ANN (PUPS)
0:15 0.30 0.45
0.15 0.30 0.45
0.000 160 240.24 0.06 0. 0.24
Mean Episade Return
Mean Episode Return
(c) SMACv2 10 Units
(d) SMACv2 20 Units
Figure 22: Performance comparison in SMAX environments after 10 million timesteps. We show the Interquartile Mean (IQM) of the Mean Win Rate and the 95% Stratified Bootstrap Confidence Intervals (CI). HyPerMARL performs comparably FuPS baselines across all environments, demonstrating its effectiveness in tasks requiring homogeneous behaviours and using recurrency.
10M Mean Epioade Raturn
-37
Training Steps (M)
FuPS
HyperMARL w/o RF
HyperMARLS
HyperMARL w/o GD
HyperMARL w/ One-Hot
HyperMARL
Figure 23: Ablation results comparing HyperMARL with its variants Dispersion. The results highlight that gradient decoupling is essential for maintaining HyperMARL's performance.
31

---

I
Hyperparameters
Table 9: Hyperparameters, Training and Evaluation for Specialisation and Synchronisation Game
Hyperparameter
Value
Environment Configuration
Number of agents
2. 4. 8, 16
Maximum steps per episode
10
Training Protocol
Number of seeds
10
Training steps
10,000
Evaluation episodes
100
Evaluation interval
1,000 steps
Batch size
32
Model Architecture
Hidden layer size
8, 16, 32, 64 ReLU
Activation function
Output activation
Softmax
Optimization
Learning rate
0.01 SGD
Optimizer
Model Parameter Count
2 Agents
NoPS: 60
FuPS: 58
FuPS+ID: 74
FuPS+ID (No State): 42
4 Agents
NoPS: 352
FuPS: 240
FuPS+ID: 404
FuPS+IDD (No State): 148
8 Agents
NoPS; 2400
FuPS: 2344
FuPS+ID: 2600
FuPS+ID (No State): 552
16 Agents
NoPS: 17728
FuPS: 17488
FuPS+ID: 18512
FuPS+ID (No State): 2128
32

---

Table 10: IPPO and MAPPO Hyperparameters Dispersion
Hyperparameter
Value
LR
0.000)5
GAMMA
0.99
VF VF_ FFCOEF
0.5
CLIP_EPS
0.2
ENT COEF
0.01
NUM ENVS
16
NUM STEPS
128
GAE LAMBDA
0.95
NUM LUMDAPESE
9765
"AISSDDSS
32
EVAL_ ENTERVAT
100000
MAX GRAD MAX_GRAD NORM
0.5
UPDATE_EPORE EPOCHS
4
NUM MINIBATCHES
TOTAL_ TIMESTEPS
20000000
ANNEAL LR
false
ACTOR LAYERS
[64. 64]
CRITIC LAYERS
[64. 64]
ACTIVATION
relu
SEEDS
ACTION ACTION_SPACE, SPACE_TYPE
discrete
Table 11: MLP Hypernet Hyperparameters in Dispersion
Parameter
IPPO
MAPPO
HYPERRETTEEMEEDEEEE EMBEDDING DIM
8
EMBEDDING DIM Sweep
[4. 16, 64)
[4, 8, 16, 64]
HYPERNETTHIDDEN DIMS
64
64
Table 12: Dispersion Settings Value
Setting
n_food
n_agents
max_steps
100
food radius
0.08
share_ reward
false
penalise ______ytt_________________e__ee_ time
continuous_action _ctioos
false
Table 13: IPPO Hyperparameters for Navigation
Hyperparameters
Value
LR
0.00005
NUM ENVS
600
NUM_STEPS STEPS
100
TOTAL TIMESTEPS
10
UPDATE EPOCHS
45
NUM MINIBATCHES
GAMMA
0.9
GAF_ MAF_LAMBDA
0.9
CLIP EPS
0.2
ENT COFF
VF COFF
1.0
MAX GRAD_NORM
ACTIVATION
tanh
ANNEAL_LR
Falsc
ACTOR LAYERS
[256, 256]
CRITIC LAYERS
[256, 256]
ACTION SPACE TYPE TYPE
continuous
33

---

Table 14: MLP Hypernet Hyperparameters in Navigation
Parameter
IPPO
MAPPO
HYPERNET EMBEDDING DIM
EMBEDDING DIM Sweep
[4, 16, 64]
[4, 8, 16, 64]
HYPERNET HIDDEN RYPERNETTHIDDEN DIMS
64
64
Table 15: DiCo Algorithm SND des Hyperparameter Goal Configuration
Number of Agents
SND des
0
agents same goal
2
1.2 (From DiCo paper)
All agents different goals
[-1,1.2.2.4] -1 (Best)
[-1,1.2,4.8] -1 (Best)
[-1,1.2] + (Best)
Some agents share goals
[-1,2.4.1.2] + (Best)
Table 16: Parameter Sweeps for IPPO Variants in Navigation Tasks with Four and Eight Agents
Parameter Sweeps
CLIP_EPS
0.2, 0.1
LR
5e-5, 5e-4, 2.5e-4
Algorithm
Setting
Selected Values
IPPO-FuPS
Agents (Same Goals)
0.2, 5c-5
Agents (Different Goals)
0.1, 5e-5
8 Agents (Four Goals)
0.1, Se-5
Agents (Same Goals)
0.2, 5e-5
4 Agents (Different Goals)
0.2, 5e-5
4 Agents (Two Goals)
0.2, 5e-5
IPPO-Linear Hypernetwork
8 Agents (Same Goals)
0.2, 5e-5
Agents (Different Goals)
0.1,
8 Agents (Four Goals)
0e1,
Agents (Same Goals)
0,2, 5e-5
4 Agents (Different Goals)
0.1,
Agents (Two Goals)
0.1,
IPPO-MLP Hypernetwork
8 Agents (Same Goals)
0.2, 5e-5
Agents (Different Goals)
0e-,
8 Agents (Four Goals)
0e1,
Agents (Same Goals)
4 Agents (Different Goals)
0.1,
Agents (Two Goals)
IPPO-NoPS
Agents (Same Goals)
0.1,
8 Agents (Different Goals)
0.2,
Agents (Four Goals)
0.1,
4 Agents (Same Goals)
0e1,
Agents (Different Goals)
0,2, 5e-5
4 Agents (Two Goals)
0e1,
IPPO-Dico
Agents (Same Goals)
0.2,
8 Agents (Different Goals)
0.1,
Agents (Four Goals)
0.1, 2.5ee4
4 Agents (Same Goals)
0.2, 5e-5
Agents (Different Goals)
0.1, 2.5e-4
4 Agents (Two Goals)
0.1,
34

---

Table 17: Environment Settings for Navigation Task
Parameter
Value
oLagents
2,4,8
agents, with same same_goal
aggentss: agents
max max_stepp steps
100
collisions
False
split_goals
False
observe all goals
True
shared rew
False
lidar range
0.35
agent radius
0.1
continuous_actinn
Truc
Table 18: Default algorithm and model hyperparameters for the Ant-v2-4x2 environment (from [54)
Parameter
Value
Algorithm Parameters
action_aggen _agggeggagiion
prod
actor_num_mint iai__mnni_bbaaa batch
clip_param
0.1
critic_epock
5
critic_num num mini batch
1
entropy_com _cef
0
fixed_order
true
gae_ gae_lambda
0.95
gamma
0.99
huber jebter
10.0
max_grad_norm
10.0
ppo_epoch
5
share_paran
false
use_ clipped_value. loss
true
use_gae
true
use_huber_loss use_max_graas norm
true
true
use seciicy_attive_eee value less__osseeee
true
1
Model Parameters
activation func critic Ir data_chunk__ gain
relu
0.0005
10
0.01
hidden_siieee
[128, 128, 128)
initialization_metlle
orthogonal_
Ir opti_eps recurrent
0.0005
le-05
1
std x x_coef std_y_coo
1
0.5
use saaaiverrecurret__peee
true
use_naive_res spclicy
false
recurrent_poliee
false
eeight_decar
0
35

---

Table 19: Default algorithm and model hyperparameters for the Humanoid-v2-177x environment
(from (44).
Parameter
Value
Algorithm Parameters
action_ageen __aggreggation
prod
actor_muue @atr_umm_mig laii_mmniaaaaaaaaall
1
clip_param
0.1
critic_epoch
5
critic_num_ntt Liimmmmiii_ie batch
I
entropy_com _cce
0
fixed_order
false
gae_ _a_la
0.95
gamma
0.99
huber_delta
10.0
max_grad_nomm
10.0
ppo_epoch
5
share_paree
false
wwwppppddvaluueeeaaeeee loss
true
use_gae use _ue_huber___e loss
true
use_may grad norm
true
use _se__olliee hol__polcy____ policyyactivee active masks value_loss_coed
Model Parameters
activation_fum func
relu
critic_Ir Ir
0.0005
sirta_hunn_leee
10
gain
0.01
hidden_sizze
[128, 128, 128]
initialization_metteee
orthogonal
Ir opti_eps recurrent n std x_ceeee std_y_coef use crrauurenoommalzaainn use_naive percurrent_polleey
0.0005
le-05
1
1
0.5
false
use_ use_recurrent_polic
false
weight_decay
0
36

---

Table 20: Default algorithm and model hyperparameters for the Walker2d-v2-2x3 environment (from
Parameter
Value
Algorithm Parameters
action_agee __aggregaaion
prod
actor fri__nn__minn lap_mmni__bbbbb batch
2
clip_param
0.05
critic_epoch
5
critic_num_ntt aaananummmnnni__aaaaa batch
2
entropy_com coef
0
fixed order
false
gae lambda
0.95
gamma
0.99
huber_delta
10.0
max_grad_norm
10.0
ppo_epoch
5
share_param
false
gge_cliped__vauee
true
use use huber loss use max_ _aad norm
true
true
true
policyoactivee use policy active masks value_loss_cooe
true
1
Model Parameters
activation_ fune
relu
critic_Ir
0.001
data_chunk length
10
gain
0.01
hidden_sizes
128, 128, 128
cnttialization_mettaee
orthogonal_
Ir
0.001
opti_eps recurrent n std x___f std_y_coef
le-05
1
1
0.5
use feature feafffue__normaaleeee
normalization use wwanaaaaivuurraeeaa
true
false
use see_recurrent_policy
false
weight_decay
0
37

---

Table 21: Default algorithm and model hyperparameters for the Se-CCheetah-2222:5 environment
(from 54
Parameter
Value
Algorithm Parameters
action_aggen aaa_ggggggation
prod
actor_ mum_mini
clip_param
0.05
critic_epoch
15
critic_num_minne facuu_mmni_baatee batch
1
entropy_coet
0.01
_ee__order
false
gae lambaa
0.95
gamma
0.99
huber_dell
10.0
max_grad_ _ord_nnomm
10.0
ppo_epoct
15
share_param
false
true
use_gae
true
use_huber_loss
true
use_max_grad_nnne
true
use_ use_policy_acee masks
true
value_loss
Model Parameters
activation_fune
relu
critic_Ir
0.0005
data_chunn length
10
gain
0.01
hidden sizes
128, 128, 128
cotpalization_mmttiee
orthogonal
Ir
0.0005
opti_eps recurrent_n
1e-05
1
std_x_coef
1
std_y_coef
0.5
true
use seee_navve___e sndiverreuurent eccccrent_polccyy
false
use recurrent_por pooicypolycyc
false
weight decay
0
Table 22: HyperMARL Hyperparameters Across MaMuloCo Environments
Parameter
Humanoid
Walker2d v2-2x3
HalfCheetah
Ant
Sweeps
v2-17x1
v2-2x3
v2-4x2
AGENT _DM
64
64
64
64
None
HNET HEST_HIDDENDDDD _DMMS
64
64
64
64
None
clip_ param
0.075
0.0375
0.0375
0.075
[0.1.0.075.0005.000375]
38

---

Table 23: Recurrent IPPO and MAPPO Hyperparameters in SMAX (from JaxMARL paper)
Hyperparameter
IPPO Value
MAPPO Value
LR
0.004
0.002
NUM ENVS
128
128
NUM STEPS
128
128
GRU HIDDEN DIM
128
128
FC DIM SIZE
128
128
T TIMESTEPS
1e7
1e7
UPDATE EPOCHS
4
4
NUM MINIBATCHES
4
GAMMA
0.99
0.99
LAMBDA
0.95
0.95
CLIP_EPS
0.05
0.2
SCALE CLIP EPS
False
False
ENT COEF
0.01
0.0
VF COEF
0.5
0.5
MAX GRAD GRAD_NORM
0.25
0.25
ACTIVATION
relu
relu
SEED
30.1.42,72858,230065
30,1,42,
30.1142,,2888823306668
ANNEAL LR
True
True
OBS WITH W AG
True
39

---

Table 24: Hyperparameter Sweeps and Final Values for Different Maps in SMAX. H- refers to
HyperMARL MLP Hypernetworss.
Map
Algorithm
LR Range
Chosen LR
HNET Embedding Dim
HNET Hidden Dims
IPPO
0.004
0.004
0.002
2532.
H-IPPO
0.004
0.004
32
H-MAPPO
0.002
0.002
16
IPPO
0.004
0.004
MАPPO
0.002, 0.005, 0.0003
0.002
3s5z
H-IPPO
0.004
0.004
H-MAPPO
0,002, 0.005, 0.0003
0.0003
64
16
IPPO
0.005, 0.001, 0.0003, 0.004
0.001
MАPPO
0,002, 0.005, 0.0003
0.0003
smacv2. units
H-IPPO
0.005, 0.001, 0.0003, 0.004
0.005
H-MAPPO
0.002, 0.005, 0.0003
0.0003
0.002, 0.005, 0.0003
0.005
MАPPO
0.002, 0.005, 0.0003
0.0003
smacv2__00 _unutts
H-IPPO
0.002, n.005, 0.0003
0.005
64
64
H-MAPPO
0.002, 0.005. 0.0003
0.0003
Note: HNET Embedding Dim refers the hypernetwor embedding dimension, chosen from the range (4, 16.
64). HNET Hidden Dims refers to the hidden layer dimensions the hypernetwork, chosen from the range (16,
32, 64).
Computational Resources
Table 25: Computational Resources by Experiment Type
Experiment Category
Hardware Configuration
Execution Time
Total Hours
Specialisation, Synchronisation & Dispersion
cores on AMD EPYC 7H12
2-6 hours per run (agent- (agent-count count dependent)
250
64-Core Processor
Navigation
8 cores on AMD EPYC 7H12
hours per run
Experiments
64-Core Processor NVIDIA RTX A4500
MaMuJoCo
8 cores AMD EPYC 7HI2
8-24 hours per (scenario & algorithm dependent)
1,680
Experiments
64-Corc Processor NVIDIA RTX A4500
SMAX
8 cores AMD EPYC 7H12
2-5 hours per run
160
Experiments
64-Core Processor NVIDIA RTX A4500
40

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sections and 6.1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Sections A and
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
41

---

Justification:
Please see Section E.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please Sections 4 Hand T
We also provide all the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
42

---

Answer: [Yes]
Justification:
We provide all the code and scripts to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodemsssseeeeee //nips.com http://bbb///iiiidedeeee/eaaaaaa cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please see Sections 5 and App.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Following best practices, report statistical significance in all results in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
43

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Read and confirmed.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See App.
B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
44

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
45

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
46

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
47


Reviews:
[{"review_id": "AviISNLT74", "reviewer": "Reviewer_bHoX", "summary": "This paper addresses parameter sharing in cooperative multi-agent reinforcement learning (MARL). The authors propose HyperMARL, which uses hypernetworks to generate agent-specific parameters by decoupling observation- and agent-conditioned gradients. The method is evaluated across different scenarios including homogeneous, heterogeneous, and mixed behavior tasks with 2-20 agents, showing competitive performance against fully-shared and non-shared approaches.", "questions": "1. Could the authors provide comprehensive comparisons with current parameter sharing methods on standard benchmarks to better demonstrate HyperMARL's advantages? These methods directly address parameter sharing in MARL and would provide a more complete evaluation context.\n2. Could the authors expand the related work section to include a broader discussion of hypernetwork applications across machine learning domains beyond reinforcement learning?\n3. Could the authors provide the training costs for HyperMARL and baseline methods (trainable parameters count / training time / training memory)? (As HyperMARL introduces additional trainable parameters and training costs compared to the FuPS baseline, although this may be acceptable.)\n4. In Figure 8, HyperMARL performs comparably to FuPS while incurring higher computational costs. What specific advantages does HyperMARL offer over FuPS in homogeneous tasks that would justify its adoption despite these added costs?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper addresses an important challenge in MARL regarding parameter sharing and policy specialization.\n2. The proposed decoupling of observation- and agent-conditioned gradients through hypernetworks is conceptually novel and well-motivated.\n3. The writing is clear and accessible. \n\n**Weaknesses**\n1. My main concern is with the evaluation：\n    While this work critiques previous efforts for adding complexity through altered objectives, manual preset diversity levels, or sequential updates, the proposed HyperMARL also introduces significantly heavier training complexity (memory-wise). The trade-off between performance gains and computational costs is not clearly justified.\n    Specifically, such trade-off should be made clearer by:\n    - Providing comprehensive comparisons of the proposed HyperMARL against current baselines on standard benchmarks with results reported consistently in the main text. Currently, the evaluation appears fragmented - HyperMARL is compared against DiCo on one benchmark (MPE) while comparison with Kaleidoscope is conducted on a different benchmark (MaMuJoCo) with results relegated to the appendix. This inconsistent evaluation approach makes it difficult to assess the method's overall effectiveness.\n    - Moreover, several important parameter sharing methods such as [1-3] are entirely omitted from the evaluation, despite addressing the exact same research question of parameter sharing in MARL. These methods would serve as highly relevant baselines for a thorough evaluation.\n    - Including a thorough comparison of computational costs (parameters, training time, memory usage) across all methods to justify the additional complexity of HyperMARL.\n2. The related work section could be enhanced.\n   - Recent efforts on parameter sharing appear to be overlooked [3].\n   - The discussion of hypernetworks should be expanded beyond reinforcement learning to position this work within the broader ML literature, where hypernetworks have been applied to areas such as federated learning, continual learning, and multi-task learning [4-5].\n\n----\n\n[1] \"Scaling multi-agent reinforcement learning with selective parameter sharing\", ICML 2021 \\\n[2] \"Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning\", AAMAS 2023 \\\n[3] \"GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning\", ICML 2025 \\\n[1] \"A brief review of hypernetworks in deep learning\", Artificial Intelligence Review. \\\n[2] \"Personalized federated learning using hypernetworks\" ICML 2021.", "was_revised": true, "final_justification": "I thank the authors for the discussion. Overall, I find this submission to have clear strengths and weaknesses:\n- As noted in my original review, the idea of decoupling observation- and agent-conditioned gradients through hypernetworks is conceptually novel and well-motivated.\n- However, my concerns regarding performance remain unresolved after the discussion. It is difficult to fully assess the effectiveness of the proposed method because the selection of baselines varies across benchmarks. While it is understandable that including every baseline in every benchmark may not always be feasible, the absence of a single benchmark that provides a comprehensive comparison with all or most of the baselines mentioned in the paper limits the ability to evaluate the method. This is an important consideration and is critical for readers to understand when and where the method is most applicable.\n- My concerns regarding the cost are mostly addressed. Through Appendix F, I see that while HyperMARL does introduce extra complexity compared to the baselines, this added complexity is acceptable as long as it is clearly reported (as the authors have done!) and balanced against improved performance (which unfortunately lack sufficient support).\n\nBased on the above justification, I will update my rating to a 3.", "rebuttal": "We thank reviewer `bHoX` for their review and for encouraging comments regarding the paper’s motivation, important problem setting, novel decoupling of observation- and agent-conditioned gradients and clear writing. We address your comments as follows:\n\n## 1. **Evaluations & Benchmarks (W1,Q1)** \n\n### **Experiment Design** \n\nWe thank the reviewer for this point. Our primary goal was to ensure the most rigorous comparison for each scenario by using baselines in the environments where they were originally tuned and shown to be most effective (e.g., DiCo in Navigation; HAPPO and Kaleidoscope in MaMuJoCo; FuPS in SMAX). By using original code and hyperparameters, we ensure fairness, reproducibility, and adherence to best evaluation practices [4]. We acknowledge that this can appear fragmented; to address this, we will add a paragraph to our existing discussion of this methodological choice (line 209) in Section 5 to clarify this further. \n\n### **Additional Baselines**\n\nWe thank the reviewer for proposing further baseline methods [1-3] to add to our five existing baselines. \n\n**New Baseline and Environment:** Based on the reviewer's recommendation, we have added **an additional baseline --  SePS [1]**, a parameter sharing method that uses an autoencoder during pre-training to cluster agents based on their transitions and shares parameters between these agent groups. We use their original source code and hyperparameters and evaluate on all their Blind Particle Spread settings (v1-4), which include up to **30 agents**.  \n\nIQM final total reward (with 95% confidence intervals) across 5 seeds on the BPS environments from [1], Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.\n\n| Alg | BPS-1 (15 agents, 3 groups)  | BPS-2 (30 agents, 3 groups)  | BPS-3 (30 agents, 5 groups) | BPS-4 (30 agents, 5 uneven groups) | \n|-----------|----|----|----|----|\n| NOPS | **-205.4** (-242.2, -197.8) | **-405.1** (-468.5, -379.9) | *-420.3 (-439.9, -407.4) | **-413.5** (-449.8, -399.1) |\n| FUPS-ID | *-219.8 (-249.7, -207.1) | *-442.6 (-467.4, -383.7) | -764.7 (-1326.7, -515.3) | -705.9 (-933.3, -591.1) |\n| SEPS | *-216.3 (-233.5, -191.8) | *-413.0 (-464.9, -385.1) | **-403.1** (-423.7, -352.9) | *-418.3 (-436.0, -393.5) |\n| HyperMARL | *-205.6 (-223.3, -175.1) | *-421.0 (-436.7, -401.0) | *-414.0 (-442.8, -374.2)  | *-428.5 (-462.7, -376.3) |\n\n**Takeaway**: The results highlight HyperMARL's competitive performance. In all four scenarios, its 95% confidence intervals overlap with SePS and NoPS, indicating it is statistically on par with these baselines. Notably, SePS relies on a **pre-training clustering phase**, whereas HyperMARL learns end-to-end without pre-training.\n\nWe thank the reviewer for the suggestion and for the opportunity to further expand our experimental evaluation, now up to **22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS)**. We will include this in the updated manuscript.  \n\nOther baselines you mentioned:\n- As mentioned in Table 4 in the App., SNP-PS [2] does not have any publicly available source code. Furthermore, we already have Kaleidoscope as a baseline, which outperformed SNP-PS. \n- GradPS [3] was made available online after 5 June (ICML camera-ready deadline) **after the NeurIPS deadline** (15 May), with no previously available preprint. Furthermore, there is no publicly available source code. We will include this work in the related work discussion.  \n\n\n## 2. **Computational Costs & Trade-offs (Q3)** \n\nWe thank the reviewer for raising this point. We analysed and discussed these points in the \"F.3 Scalability and Parameter Efficiency\" and “F.4 Speed and Memory Usage” sections in the Appendix, and will ensure key trade-offs are also mentioned in the main text. \n\n- **Parameter Scaling (F.3):** There is a direct trade-off between scalability and versatility. While our MLP-based hypernetworks have more initial parameters than FuPS, they scale nearly constantly with the number of agents. In contrast, NoPS sees its parameter count grow linearly with respect to the number of agents. This makes our approach far more parameter-efficient for large-scale MARL, especially when the task could require specialisation. FuPS+ID parameters scale better than our hypernetworks, but these approaches fail to solve tasks that require specialisation, as we showed in Sections 3.2 and 5. We also mention that techniques like chunked hypernetworks or low-rank approximations could be used in future work to further improve the parameter efficiency of our method. In Figure 15, we also see that if we scale FuPS+ID to match the number of trainable parameters in our MLP hypernetworks, our approach still outperforms these methods, showing the benefit is architectural and not just from a higher parameter count.\n- **Runtime & Memory (F.4):** HyperMARL offers a practical middle ground. It is more computationally intensive than the simple FuPS architecture, but it is more efficient than NoPS in terms of speed and memory. Furthermore, NoPS would also incur data transfer and synchronisation costs not fully captured in our single-GPU benchmarks.\n\n\n## 3. **Additional Related Work (W2, Q2)**\n\nThank you for suggesting additional related work. As mentioned, GradPS [3] was published after the submission deadline, but thanks for the recommendation, and we will include a discussion of this in the updated manuscript. For the other papers mentioned, our focus was reinforcement learning and so we focused on hypernetworks in this setting (not in other fields like federated learning). Based on your recommendation, we will also position our work within the broader hypernetwork literature in the updated manuscript. \n\n## 4. **HyperMARL vs FuPS in Homogeneous Tasks like SMAX (Q4)**\n\nThese experiments were presented to validate Hypermarl’s adaptability claim, i.e. that it can learn specialised or homogeneous behaviour when required. Matching FuPS performance in homogeneous tasks shows that HyperMARL can also adapt to these settings, i.e. does not enforce specialisation in a detrimental manner. The challenge is that in most settings, you do not already know if the task requires homogeneous, specialised or mixed behaviour a priori, therefore, it is useful to have a method that can adapt to varying problem classes and a versatile architecture that is effective across these settings. To validate these claims was the core focus of the experimental section's  **Q1** and **Q2** in Section 5. In summary, this confirmed that our method **does not degrade performance in homogeneous regimes while improving performance in heterogeneous ones** (compared to FuPS).\n\n\n## **Conclusion**\n\nIn summary, based on your feedback we added:\n- A **new baseline** SePS [1] and environment, Blind Particle Spread. Extending our evaluation to **22 scenarios, up to 30 agents and six baselines** (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). Our approach is competitive with this baselines, without requiring a *pre-training* phase. \n- Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4). \n\nDuring the rebuttal, we have also added:\n- **Sensitivity Analysis** showing which parameters of our method influence performance the most.\n- An **analysis of the learned agent embeddings** showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise. \n- A comparison with **dual-channel networks**, showing our approach is more effective.\n\nWe hope these clarifications and additional experiments address your concerns, and we thank the reviewer for the opportunity to improve our manuscript. If there are no further issues, we respectfully ask reviewer `bHoX` to provide an updated assessment of our work. \n\nReferences:\n\n1] Christianos, F., Papoudakis, G., Rahman, M.A. and Albrecht, S.V., 2021, July. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR. \n\n2] Kim, W. and Sung, Y., 2023. Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.00912.\n\n3] Qin, H., Liu, Z., Lin, C., Ma, C., Mei, S., Shen, S. and Wang, C., GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning. In Forty-second International Conference on Machine Learning.\n\n4] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25(318):1–63, 2024."}, {"review_id": "Vp3QrHHvKD", "reviewer": "Reviewer_1sWF", "summary": "This paper inroduces HyperMARL, a novel method for multi-agent reinforcement learning that uses agent-conditioned hypernetworks to generate per-agent parameters on the fly. The goal is to enable diverse (specialised), homogeneous, or mixed behaviours without modifying the standard RL objective or requiring manually preset diversity levels. The key idea is to decouple observation-conditioned and agent-conditioned gradients, mitigating cross-agent gradient interference that plagues standard parameter-sharing approaches.", "questions": "## Questions\n1. How does HyperMARL's training time compare (e.g. wall-clock or steps to convergence) to FuPS, NoPS, and DiCo in the largest tested tasks?\n2. How sensitive is HyperMARL to hypernetwork design choices (e.g. embedding dimension, number of MLP layers)?\n3. Can the authors include results comparing with recent heterogeneous-agent methods such as HetGPPO on the same tasks?", "limitations": "## Limitations\n- **Baseline selection**: While strong baselines like DiCo, HAPPO, and Kaleidoscope are included, some very recent or robotics-focused heterogeneous MARL baselines (e.g. HetGPPO) are missing from the comparison.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- **Interesting method for enhancing agent diversity while preserving performance in homogeneous tasks**: HyperMARL achieves NoPS-level policy diversity while maintaining the efficiency of parameter sharing, which is well demonstrated across diverse benchmarks (Dispersion, Navigation, MAMuJoCo, SMAX).\n- **Effectively tackles gradient conflicts**: The paper convincingly shows how decoupling observation and agent-ID gradients reduces policy gradient variance and cross-agent interference. Ablations confirm this is essential for performance.\n- **Clear and easy to follow**: The research questions are well posed (\"Can shared policies adapt without added complexity?\"), the theoretical motivation is thorough (gradient conflict analysis), and the method is described with clear architectural diagrams and derivations. The empirical section answers the research questions systematically.\n\n## Weaknesses\n- **Missing computational comparison (e.g. training time)**: While parameter efficiency is discussed (Appendix A mentions scaling), there is no direct measurement or table comparing training times or wall-clock convergence with standard parameter-sharing or NoPS baselines. Including such data would strengthen the empirical evaluation.\n- **Lack of comparison with state-of-the-art heterogeneous baselines in complex robotic settings**: For example, the paper does not compare HyperMARL with HetGPPO or other recent methods explicitly tailored for large-scale heterogeneous agents in robotics. Even though HyperMARL performs well on MAMuJoCo (17-agent Humanoid), comparing with specialized heterogeneous-agent methods would clarify its relative advantages or limitations.\n- **Sensitivity to hyperparameters**: Hypernetworks are known to be sensitive to hyperparameter choices (e.g. embedding size, MLP depth). The paper briefly notes that hyperparameter sweeps for some baselines (e.g. DiCo) were challenging but does not analyze how much HyperMARL itself depends on careful tuning. A section on its hyperparameter robustness would be valuable.", "was_revised": true, "final_justification": "I keep my initial score since the authors answered my concerns. the quality of the paper is the same after rebuttal", "rebuttal": "We thank reviewer `1sWF` for their detailed and constructive review. We appreciate the reviewer's positive feedback on our novel insight into gradient decoupling, the diversity of our experimental evaluation, and our paper's clarity. We address your points as follows:\n\n## 1. **Computational Comparison (W1,Q1)** \n\nWe thank the reviewer for raising this important point. Our paper provides a detailed analysis of these computational trade-offs in Appendix F.3 and F.4, and we summarise the key findings here:\n\n- **Parameter Scaling (F.3):** There is a direct trade-off between scalability and versatility. While our MLP-based hypernetworks have more initial parameters than FuPS, they scale nearly constantly with the number of agents. In contrast, NoPS sees its parameter count grow linearly with respect to the number of agents. This makes our approach far more parameter-efficient for large-scale MARL, especially when the task could require specialisation. FuPS+ID parameters scale better than our hypernetworks, but these approaches fail to solve tasks that require specialisation, as we showed in Sections 3.2 and 5. We also mention that techniques like chunked hypernetwork or low-rank approximations could be used in future work to further improve the parameter efficiency of our method. In Figure 15, we also see that if we scale FuPS+ID to match the number of trainable parameters in our MLP hypernetworks, our approach still outperforms these methods, showing the benefit is architectural and not just from a higher parameter count.\n- **Runtime & Memory (F.4):** HyperMARL offers a practical middle ground. It is more computationally intensive than the simple FuPS architecture, but it is more efficient than NoPS in terms of speed and memory. Furthermore, NoPS would also incur data transfer and synchronisation costs not fully captured in our single-GPU benchmarks.\n\n## 2. **Comparison with HetGPPO - Multi-Robot Heterogeneous Methods with Communication (W2, Q3)**\n\nWe thank the reviewer for this suggestion. Our work focuses on behavioural heterogeneity, where physically identical agents learn diverse or homogenous policies. This is distinct from methods like HetGPPO, which are designed for structural heterogeneity (e.g., agents with different sensors or actuators) and often incorporate explicit communication.\n\nWhile HetGPPO is an important baseline in its own domain, we benchmarked against the most relevant state-of-the-art methods for our specific problem (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope, and  SePS). To clarify this distinction, we will update our related work section to include a discussion of HetGPPO.  \n\n## 3. **Sensitivity to Hyperparameters (Q2)** \n\n**New Sensitivity Analysis:** We thank the reviewer for raising this important point. Based on your recommendation, we run a sensitivity analysis on the hypernetworks parameters, in the 20-agent SMAX scenario with MAPPO, which also includes a GRU, a challenging setting for optimisation.\n\n**Findings**: \n1. Similarly to FuPS+ID, and in general on-policy RL methods like PPO, learning rate is an important parameter for performance. \n2. Agent embedding size can be an important parameter for the performance of HyperMARL, with a smaller agent embedding size outperforming a larger one in this case. This could be the homogenous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence behaviours. \n3. Other parameters, such as the width, have limited impact beyond a modest size.\n\nIQM and 95% CI of mean win rate across 5 seeds. **Bold** indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.\n\n| Method | Embedding Dim | Hidden Dims | LR 0.0001 | LR 0.0003 | LR 0.0005 |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| FuPS+ID | N/A | N/A | 0.1067 (0.0513, 0.2128) | 0.3053 (0.2041, 0.4769) | *0.4213 (0.3623, 0.4800) |\n| HyperMARL| 4 | 16 | 0.0799 (0.0250, 0.1190) | 0.2946 (0.1143, 0.3778) | **0.4455** (0.3443, 0.5692) |\n| | 4 | 64 | 0.1233 (0.0750, 0.2162) | 0.3338 (0.2692, 0.4286) | *0.3765 (0.3281, 0.4107) |\n| | 4 | 128 | 0.1147 (0.0270, 0.1842) | *0.3787 (0.3509, 0.4603) | 0.3327 (0.2931, 0.3548) |\n| | 4 | 64, 64 | 0.1781 (0.1628, 0.2075) | *0.3498 (0.2807, 0.4500) | 0.1944 (0.1268, 0.3077) |\n| HyperMARL | 64 | 16 | 0.1109 (0.0256, 0.2093) | 0.3198 (0.2353, 0.3934) | *0.3483 (0.2581, 0.4308) |\n| | 64 | 64 | 0.1360 (0.0303, 0.2821) | 0.1191 (0.0517, 0.1964) | 0.1155 (0.0794, 0.1500) |\n| | 64 | 128 | 0.1193 (0.0750, 0.1579) | 0.1036 (0.0517, 0.1594) | 0.1106 (0.0455, 0.1587) |\n| | 64 | 64, 64 | 0.1353 (0.0541, 0.1818) | 0.1283 (0.0526, 0.2258) | 0.0860 (0.0345, 0.1343) |\n\n**Takeaway:** Agent embedding size can be an important hyperparameter in our method, one that could correspond to the task's diversity requirements.\n\nWe thank the reviewer for recommending this analysis, and we will include a discussion of this in the updated manuscript.\n\n## **Conclusion**\n\nIn summary, based on your feedback we added:\n- A **sensitivity analysis** using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment. \n- Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4). \n\nDuring the rebuttal, we have also added:\n- A **new baseline** SePS and environment, Blind Particle Spread. Extending our evaluation to **22 scenarios, up to 30 agents and six baselines** (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline. \n- A **learned agent embeddings analysis** showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.\n- A comparison with **dual-channel networks**, showing our approach is more effective.\n\nWe would like to thank the reviewer again for their thoughtful review and hope these clarifications and additional experiments have fully addressed the points raised. If you have any follow-up questions or comments, please let us know, and we will be happy to discuss further."}, {"review_id": "pG0VCe5K9R", "reviewer": "Reviewer_uZHN", "summary": "This paper addresses a central challenge in cooperative multi-agent reinforcement learning (MARL): enabling policies to adaptively learn specialized, homogeneous, or mixed behaviors without sacrificing sample efficiency. The authors identify an issue in standard parameter-sharing methods (like FuPS+ID), arguing that the coupling of agent observations and agent IDs within a single network exacerbates cross-agent gradient interference and hinders specialization.\n\nTo solve this, they propose HyperMARL, an architecture that uses an agent-conditioned hypernetwork to generate agent-specific parameters for the policy and critic networks. This design decouples the observation-conditioned gradients from the agent-conditioned ones. The core benefit, as shown theoretically and empirically, is a reduction in policy gradient variance, which mitigates gradient conflicts and facilitates adaptive learning. The authors conduct experiments across a diverse set of MARL benchmarks (VMAS, MAMuJoCo, SMAX), demonstrating that HyperMARL achieves competitive or superior performance compared to baselines, including No-Parameter-Sharing (NoPS), Full-Parameter-Sharing (FuPS), and state-of-the-art diversity-promoting methods like DiCo, HAPPO, and Kaleidoscope, while maintaining high behavioral diversity.", "questions": "The following questions aim to clarify some key aspects of the method and its implications. I would be willing to raise my score if the authors can provide satisfactory answers.\n\n**Q1:** The Nature of Learned Agent Embeddings: For the MLP Hypernetwork, the paper uses learned agent embeddings. Could the authors provide some analysis of these learned embeddings? For instance, do agents that are required to learn similar (or complementary) policies end up with closer embeddings in the latent space? Visualizing the embedding space (e.g., using t-SNE) could provide valuable insights into whether the hypernetwork is learning a meaningful representation of inter-agent relationships.\n\n**Q2:** Disentangling the Effects of Gradient Decoupling vs. Parameter Specialization: HyperMARL introduces two concurrent benefits: (1) it decouples observation- and agent-conditioned gradients, and (2) it provides each agent with its own specialized set of parameters. While these two are intertwined in the proposed architecture, could the authors speculate on which factor is more critical? For example, would an architecture that provides specialized parameters but without explicit gradient decoupling (if one could be designed) perform as well? This question aims to pinpoint the core ingredient of HyperMARL's success.\n\n**Q3:** Performance in Mixed Cooperative-Competitive Settings: The experiments focus on fully cooperative MARL. How would HyperMARL be expected to perform in mixed-motive or general-sum settings where agents might have conflicting objectives? In such scenarios, the concept of \"cross-agent gradient interference\" becomes more complex, as some interference might be adversarial rather than simply a result of differing specialization needs. Does the gradient decoupling mechanism remain as beneficial?\n\n**Q4:** Comparison with Role-Based Approaches: How does HyperMARL compare conceptually to explicit role-based learning methods (e.g., ROMA)? Both aim to induce specialization. It seems HyperMARL learns these roles implicitly through the agent embeddings and generated weights, potentially offering more flexibility. A brief discussion contrasting these two paradigms would strengthen the paper's positioning in the literature.", "limitations": "The authors have commendably included a limitations section in the appendix. To build upon that, the following points can be highlighted:\n\n**L1:**  Assumption of Fixed Agent Population: The current framework assumes a fixed number of agents n throughout training, as the hypernetwork architecture (especially the linear variant) and agent embeddings are tied to this number. The model may not generalize to scenarios with a varying or unknown number of agents without retraining or specific architectural modifications (e.g., attention-based hypernetworks).\n\n**L2:**  Increased Computational Cost at Inference: Generating network weights on the fly with a hypernetwork introduces an additional computational step at inference time compared to a standard FuPS model. While faster than a NoPS approach (which may involve significant data transfer overhead in a real distributed system), this added latency could be a concern for time-critical applications. The analysis in Appendix F.4 is good, but this trade-off deserves to be highlighted as a limitation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- S1: Novel Insight: The paper provides a clear and compelling diagnosis of a problem in parameter-shared MARL. The empirical study in Section 3 and Figure 4, which isolates the detrimental effect of coupling observation and agent-ID inputs, is a contribution in itself. This insight provides motivation for the proposed architectural design.\n- S2: Clarity and Presentation: The paper is well-written and easy to follow. Figure 1 provides an visual summary of the core idea. The logical flow, from problem identification to solution and validation, is clear. The appendix is detailed and provides sufficient information.\n\nWeaknesses\n- W1: Scalability in Terms of Parameter Count: The primary weakness of the HyperMARL approach, particularly the MLP variant, is the significant parameter overhead. The hypernetwork must generate the entire weight set for each agent's policy (and critic) network. While the paper commendably analyzes scalability with the number of agents (Appendix F.3) and shows near-constant scaling for MLP hypernetworks, it doesn't fully address the scalability with respect to the size of the target network. For very deep or complex policy architectures (e.g., those based on Transformers), the output dimension of the hypernetwork could become prohibitively large, making the approach computationally expensive and difficult to train. The suggested remedies (chunking, low-rank) are mentioned but not empirically explored.\n- W2: Complexity of Hypernetwork Training and Initialization: The stability and performance of HyperMARL likely depend on careful initialization and training of the hypernetwork itself. The paper mentions a principled initialization scheme (Section 4.2) to match standard network initializations, which is a good practice. However, training a network to output the weights of another network is inherently a more complex and potentially less stable optimization problem than standard policy training. The paper could benefit from a more detailed discussion or analysis of the training dynamics and sensitivity to the hypernetwork's own hyperparameters (e.g., its depth, width, learning rate).", "was_revised": true, "final_justification": "I have raised my score from 3 (Borderline Reject) to 4 (Borderline Accept). The authors' rebuttal was effective, particularly the new analysis of learned agent embeddings. This provided the compelling evidence for meaningful specialization that was previously missing, significantly strengthening the paper's core contribution and serving as the key factor for my score change. Nonetheless, the evaluation scope is confined to fully cooperative settings, leaving its performance in more complex mixed-motive scenarios as an important open question. Therefore, while the paper is now technically much stronger, its limited scope makes a borderline accept the most appropriate rating. The reasons to accept now outweigh the reasons to reject.", "rebuttal": "We thank reviewer `uZHN` for their thoughtful and constructive review. We found your points valuable for helping improve the quality of our manuscript. We also thank the reviewer for acknowledging our diverse experimental evaluation, novel insight concerning gradient decoupling in MARL, and for recognising our clarity and presentation. We address your comments as follows:\n\n## 1. **Scalability with Target Network Size (W1)**\n\nWe thank the reviewer for this point. The parameter overhead of the hypernetwork does indeed scale with the size of the target network. While we argue this is often an acceptable trade-off for the smaller actor-critic networks commonly used in MARL and the near-constant scaling with respect to the number of agents might be a viable trade-off for some use cases (Section F.3), this could be problematic for much larger target networks such as Transformers. However, as noted in our limitations and your review, this is a known characteristic of hypernetworks, and there is a clear and established path for scaling to such architectures using parameter-efficient techniques like chunking or low-rank weight approximations.\n\n## 2. **Complexity of Hypernetwork Training and Sensitivity to Hyperparameters (W2)**\n\n**New Sensitivity Analysis:** We thank the reviewer for raising this important point. Based on your recommendation, we run a sensitivity analysis on the hypernetworks parameters, in the 20-agent SMAX scenario with MAPPO, which also includes a recurrent GRU, a challenging setting for optimisation.\n\n**Findings**: \n1. Similarly to FuPS+ID, and in general on-policy RL methods like PPO, learning rate is an important parameter for performance. \n2. Agent embedding size can be an important parameter for the performance of HyperMARL, with a smaller agent embedding size outperforming a larger one in this case. This could be the homogenous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence behaviours. \n3. Other parameters, such as the width, have limited impact beyond a modest size.\n\nIQM and 95% CI of mean win rate across 5 seeds. **Bold** indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.\n\n| Method | Embedding Dim | Hidden Dims | LR 0.0001 | LR 0.0003 | LR 0.0005 |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| FuPS+ID | N/A | N/A | 0.1067 (0.0513, 0.2128) | 0.3053 (0.2041, 0.4769) | *0.4213 (0.3623, 0.4800) |\n| HyperMARL| 4 | 16 | 0.0799 (0.0250, 0.1190) | 0.2946 (0.1143, 0.3778) | **0.4455** (0.3443, 0.5692) |\n| | 4 | 64 | 0.1233 (0.0750, 0.2162) | 0.3338 (0.2692, 0.4286) | *0.3765 (0.3281, 0.4107) |\n| | 4 | 128 | 0.1147 (0.0270, 0.1842) | *0.3787 (0.3509, 0.4603) | 0.3327 (0.2931, 0.3548) |\n| | 4 | 64, 64 | 0.1781 (0.1628, 0.2075) | *0.3498 (0.2807, 0.4500) | 0.1944 (0.1268, 0.3077) |\n| HyperMARL | 64 | 16 | 0.1109 (0.0256, 0.2093) | 0.3198 (0.2353, 0.3934) | *0.3483 (0.2581, 0.4308) |\n| | 64 | 64 | 0.1360 (0.0303, 0.2821) | 0.1191 (0.0517, 0.1964) | 0.1155 (0.0794, 0.1500) |\n| | 64 | 128 | 0.1193 (0.0750, 0.1579) | 0.1036 (0.0517, 0.1594) | 0.1106 (0.0455, 0.1587) |\n| | 64 | 64, 64 | 0.1353 (0.0541, 0.1818) | 0.1283 (0.0526, 0.2258) | 0.0860 (0.0345, 0.1343) |\n\n**Takeaway:** Agent embedding size can be an important hyperparameter in our method, one that could correspond to the task's diversity requirements.\n\nWe thank the reviewer for recommending this analysis, and we will include a discussion of this in the updated manuscript.\n\n\n## 3. **Analysis of Learned Agent Embeddings (Q1)**\n\nWe thank the reviewer for this suggestion. We conducted an analysis of the learned embeddings as per the reviewer's recommendation. We chose the 4-agent Navigation environment since we could test the embeddings in the same settings with two different objectives: one where all agents have the *same goal position* (from Fig. 7a) and the one where agents have *different goal* locations (Fig. 7b). \n\n**New Agent Embedding Analysis:** We compute the cosine distance between agent embeddings. At initialisation, due to the orthogonal initialisation, agent embeddings have a distance of 1 from each other. \n\n**Findings**: When the task requires agents to learn the same behaviour (go to the same goal), the embeddings move closer together (-0.118, p-value using one-sample t-test = 0.005, i.e. we reject that embeddings have the same distance as at init). When agents have different goals, the embeddings remain near orthogonal (+0.010, p-value using one-sample t-test = 0.331, i.e. we cannot reject that embeddings maintain their initial distance). This confirms that the hypernets have learned useful and adaptive representations. \n\n\n**Results:** The table below shows mean cosine distances (± standard deviation) across 5 random seeds:\n| Settings | At Init | After Training | Change from Init |\n|----------------|-------------------|----------------|------------------|\n| Same Goals | 1.000 | 0.882 ± 0.042 | -0.118  |\n| Different Goals | 1.000 | 1.010 ± 0.017 | +0.010  |\n\nWe believe this provides further motivation for our approach and will include this and our t-SNE plots in the updated manuscript. Thanks again for the recommendation. \n\n## 4. **Disentangling the Effects of Gradient Decoupling vs. Parameter Specialisation (Q2)**\n\nWe believe both gradient decoupling and parameter specialisation are important, but gradient decoupling is a more critical factor. Our ablation studies in Fig. 9 provide strong evidence of the importance of the gradient decoupling. In the \"w/o GD\" (without gradient decoupling) variant, agents still have specialised parameters, but we re-couple the gradients by also passing the ID to the policy, and this hurts performance. This demonstrates that simply providing specialised parameters is insufficient without the architectural decoupling that HyperMARL provides. We will make this clearer in the updated manuscript. \n\n## 5. **HyperMARL in Mixed/Competitive Settings (Q3)**\n\nWe thank the reviewer for the suggestion and agree that this would be an interesting angle to our work, especially with regard to the conflicting objectives. While parameter sharing has traditionally been investigated in cooperative tasks, it would be interesting to study this in other game settings. Unfortunately, due to limited resources and time, we are not able to robustly measure HyperMARL performance in these settings. We believe this might be an interesting direction, and will include it in potential avenues of future work. \n\n\n## 6. **Comparison with Role-Based Approaches (Q4)**\n\nRole-based methods, like ROMA and RODE [1, 2], assign agents to subtasks/roles but rely on mutual information (MI) objectives for diversity, and have had limited practical success [3]. We will include a more detailed comparison in the updated paper. \n\n## 7. **Assumption of Fixed Agent Population (L1)**\n\nIt is correct that we only consider the setting where the team in training and evaluation is fixed and doesn't change, i.e. no generalisation to an unknown number of agents. As noted by the reviewer,  the linear hypernetwork variant is tied to the number of agents, but not the MLP variant. \n\nWe actually consider this setting an ideal one for our MLP hypernetworks, because, unlike traditional FuPS+ID, which cannot handle a different number of agents due to the dimension one-hot encoded agent IDs, MLP hypernets with learned agent embeddings (of a fixed size) enable adding new agents to the team with some fine-tuning of the agent ID embeddings. Rather than explicitly a limitation, since all baselines are either incompatible or have the same property, we see this as an interesting area for future work. We will mention this in the updated manuscript. \n\n## 8. **Trade-off between Computational Cost (at Inference) and Adaptability (L2)**\n\nWe thank the reviewer for raising this important point. HyperMARL's adaptivity does come at a higher inference cost than FuPS -- as we showed in F.4, and this trade-off would need to be balanced depending on the scenario. We will explicitly mention this in the limitations section in the updated manuscript. \n\n\n## **Conclusion**\n\nIn summary, based on your feedback we added:\n- A **sensitivity analysis** using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment. \n- A **learned agent embeddings analysis** showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.\n- Discussed the importance of gradient decoupling vs parameter specialisation. \n\nDuring the rebuttal, we have also added:\n- A **new baseline** SePS and environment, Blind Particle Spread. Extending our evaluation to **22 scenarios, up to 30 agents and six baselines** (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline. \n- A comparison with dual-channel networks, showing our approach is more effective.\n\nWe would like to thank the reviewer again for their thoughtful review and for the opportunity to improve our manuscript. We hope these clarifications and additional experiments address your concerns. If there are no further issues, we respectfully ask reviewer `uZHN` to provide an updated assessment of our work.\n\n\nReferences:\n\n1] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, 407 pages 9876–9886, 2020.\n\n2] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.\n\n3] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In 34th Conference on Neural Information Processing Systems, 2020."}, {"review_id": "P30NYN2Seu", "reviewer": "Reviewer_z98m", "summary": "The paper proposes HyperMARL, which decouples the observation-conditioned and agent ID-conditioned gradients via hypernetworks to enable better coordination for both specialist and generalist behaviors. The authors first show that full parameter sharing leads to conflicting updating gradients of agents. Then they propose HyperMARL that uses agent ID-conditioned hypernetworks to generate weights and biases for agent policy and critic networks. Experiments on Dispersion, Navigation, MAMuJoCo, and SMAX validate the proposed HyperMARL when compared with various baselines.", "questions": "1.\tCould the authors compare hypernetworks with a dual-channel network, where one channel is for agent observation and one channel is for agent ID?\n2.\tCould HyperMARL work well in complicated tasks such as SMACv2 where the observation provides key decision information? May hypernetworks limit the learning ability of the agent network?\n3.\tDoes HyperMARL support value-factorization MARL algorithms such as QMIX and QPLEX?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1.\tThe finding of conflicting gradients among agents by full parameter sharing and agent ID is important.\n2.\tThe method is well-motivated and reasonable. Through the hypernetwork, the gradient flow is explicitly decoupled as shown in Equation (4).\n3.\tThis paper is well-written and well-organized.\n\nWeaknesses\n\n1.\tUsing agent ID or embedding only to generate weights and biases of agent policy and critic networks may limit the representation ability in complex tasks.\n2.\tThe widely used SMAC and SMACv2 environments are not tested. For the SMAX, its dynamic seems to be simplified and ease the learning of hypernetworks for policy and critic weights.\n3.\tSome role-based MARL works, such as RODE, are not compared in the experiments.", "was_revised": true, "final_justification": "Although I recommend accepting, this paper could benefit from comparing with role-based approaches and conducting experiments in SMAC.", "rebuttal": "We thank reviewer `z98m` for their review and for encouraging comments regarding our novel insight with respect to gradient decoupling in MARL, our strong motivation for using agent-conditioned hypernetworks and our clarity. We address your comments as follows: \n\n## 1. **Using ID or Embeddings … may limit representation ability (W1)**\n\nWe would like to clarify that the agent IDs or embeddings inputs are not the main components of the hypernetworks that influence their representation ability, but rather the functional form of the hypernetwork. Linear hypernetworks can only capture linear relationships between agents’ embeddings and their corresponding weights. To enhance expressiveness, we introduced MLP hypernetworks for most of our experiments, allowing our method to adapt to more complex tasks. Furthermore, using learned embeddings means that the conditioning can automatically adapt to the task. We will make this property clearer in the updated paper. \n\n## 2. **Using SMAC and SMACv2 vs SMAX and Could HyperMARL work well in... SMACv2 (W2, Q2)**\n\nWe evaluate on SMAX, an accelerated approximate reimplementation of SMAC, which includes v1 and v2 style scenarios. We benchmark on both v1 (e.g. 2s3z,3s5z) and v2 style maps from SMAX (10 and 20 units). Although some dynamics are different in SMAX, it features a more sophisticated heuristic AI opponent, and some maps are harder in this version [1]. We believe these settings provide a strong and appropriate testbed for our method. \n\nFurthermore, we included SMACv2-style scenarios in SMAX that also have this “meaningful partial observability” property of SMACv2-style maps. Our approach performed well in this case (Fig. 8c and d) showing that the hypernetworks do not limit the agent's network. \n\n## 3. **Comparison with role-based approaches, such as ROMA, in the Experiments (W3)**\n\nRole-based methods, like ROMA and RODE [3, 4], assign agents to subtasks/roles but rely on mutual information (MI) objectives for diversity. Our primary reason for not including these methods in the experiments is that they have been shown to have limited empirical performance (Table 1 in [2]), which we discussed in Table 4 in the Appendix. Furthermore, we have six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS), which we feel serve as a good testbed. Nonetheless, we will expand on our discussion of role-based methods in the related work, in the updated manuscript. \n\n## 4. **Comparison with Dual-channel Network Architecture**\n\n**New Ablation:** We thank the reviewer for their thoughtful suggestion. Following the reviewer's advice, we implemented and evaluated a **dual-channel network** architecture. This baseline processes the agent's observation and ID in separate channels before concatenating their outputs. We tested two variants: one using one-hot encoded agent IDs and another using learned agent embeddings. We evaluated this using IPPO on the Dispersion task from our paper. Our results below show that HyperMARL outperforms dual-channel networks. \n\n**Results:** IQM and 95% CI of mean test episode return across 5 seeds, **Bold** indicates the highest IQM score. \n\n| Method | IQM, 95% CI |\n|---|---|\n|  HyperMARL (MLP hypernetwork) | **0.7783** (0.7768, 0.7802) |\n|  Dual-Channel (one-hot ID) | -0.4749 (-0.9187, 0.3984) |\n|  Dual -Channel (with Agent Embedding) | -0.2934 (-0.9187, 0.5450) |\n\n**Takeaway:** HyperMARL outperforms dual-channel architectures.\n\n## 5. **Compatibility with value-factorisation/decomposition methods (e.g. QMIX, QPLEX) (Q3)**\n\nYes, HyperMARL is compatible with these methods as it simply generates policy or critic networks and could do so in any setting or method. In Table 5 in the App, we showed that HyperMARL also works well in off-policy methods such as MATD3. We believe these results would also hold for value-decomposition methods. However, due to limited time resources and time during the rebuttal, we are unable to run additional experiments using HyperMARL for value-decomposition methods. We will discuss this as a possible future work direction in the updated manuscript.  \n\n\n## **Conclusion**\n\nIn summary, based on your feedback we added:\n- A comparison with **dual-channel networks**, showing our approach is more effective.\n- Discussed the compatibility of our method with value-decomposition methods and discussed the representations that can be learned.  \n\n\nDuring the rebuttal, we have also added:\n- A **new baseline** SePS and environment, Blind Particle Spread. Extending our evaluation to **22 scenarios, up to 30 agents and six baselines** (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline. \n- A **sensivitiy analysis** using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment. \n- A **learned agent embeddings analysis** showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.\n\nWe would like to thank the reviewer again for their thoughtful review and hope these clarifications and additional experiments address your concerns. If there are no further issues, we respectfully ask reviewer `z98m` to provide an updated assessment of our work.\n\nReferences:\n\n1] Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson Juto, G., Willi, T., Hammond, R., Khan, A., Schroeder de Witt, C. and Souly, A., 2024. Jaxmarl: Multi-agent rl environments and algorithms in jax. Advances in Neural Information Processing Systems, 37, pp.50925-50951.\n\n2] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent\nreinforcement learning. In 34th Conference on Neural Information Processing Systems, 2020.\n\n3] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, 407 pages 9876–9886, 2020.\n\n4] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020."}, {"review_id": "2edrNGxIOY", "reviewer": "Reviewer_QQ2N", "summary": "In this work, the authors focus on the potential limitations in parameter sharing methods. Based on experimental evidence, it is elaborated that coupling agent IDs with observations can be detrimental, blamed by gradient interference among agents. To overcome the key difficulty with FuPS, an agent-conditioned hypernetworks based parameter sharing method is proposed, which explicitly decouples observation- and agent-conditioned gradients. By mitigating cross-agent interference, HyperMARL empirically reduces policy gradient variance, facilitates shared-policy adaptation, and learns heterogeneous, homogeneous, or mixed behaviors, as demonstrated in various challenging MARL benchmarks.", "questions": "1. Is it possible that the specialization of the policy may lead to redundant diversification policies? For example, in a neural network, a function can be fitted with two nodes, or it can be fitted with MLP, but the latter may cause serious overfitting problems. Will the strengthening of specialization lead to a decrease in adaptability? How to know the appropriate level of specialization is achieved?\n\n2. (Christianos, Filippos, et al. \"Scaling multi-agent reinforcement learning with selective parameter sharing.\" International Conference on Machine Learning. PMLR, 2021). The above paper seems to have similar ideas, and their encoder-decoder can also be considered as a type of hypernetworks. What are the similarities and differences between this work and previous work? Furthermore, in the previous work, the encoder-decoder was trained before RL, while in this paper, it is trained simultaneously. During simultaneous training, will there be conflicts between RL learning and policy specialization learning? What are the advantages of simultaneous training? Based on the decoupling idea mentioned in this article, it seems that it would be better to decouple these two optimization goals?\n\n3. In Figure 3 (page 4), why ‘FuPS+ID’ method fails even in synchronization environment? According to the explanation in subsection 3.2, it should not fail in this environment? In Figure 4, what is ‘PG-FuPS+ID’ (with no definition)? Is ‘PG-FuPS+ID’ equal to ‘FuPS+ID’? In line 105 (page 4), ‘Table 3’ (mistake?)? In line 108 to 109, why authors claim that the fully policy sharing method also has scalability issue in synchronization environment? \n\n4. In subsection 3.2 (line 117 ‘Importance of Observation and ID Decoupling’), authors show the result of ‘FuPS+ID’ and ‘FuPS+ID (No State)’. What about ‘FuPS’? Is it an orthogonal gradient or a more conflicting gradient? If it is orthogonal gradient: why is it not performing well? If not: is it possible that adding ID could reduce the conflicting gradients, that is, ID is actually beneficial? Based on the existing results, it seems that we cannot easily draw a conclusion about \"Importance of Observation and ID Decoupling\"?\n\n5. In section 4, how are the agent-specific parameters ($\\theta $) integrated into original RL networks? Are all the parameters of the policy network ($\\pi $) generated by the hypernetworks? Or is it just the last layer that is replaced, and the other layers are still RNN? Why use ID as input to the hypernetworks? Can the inputs to the hypernetworks and policy network be swapped (i.e. use observation as input to the hypernetworks, since the purpose here seems to be just for decoupling)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written. The organization of this paper is clear, and the proposed idea is easy to follow. The limitations of parameter sharing are elaborated with clear definition of research problems. The experimental verification on the impact of the coupling of observations and agent IDs brings certain level of new opinion and knowledge. \n\n2. The experiment is comprehensive. The proposed method is tested in diverse benchmarks where heterogeneous, homogeneous, or mixed behaviors are required. The ablation and computation complexity are also considered. \n\n3. Even though simple, the methodology is effective. The theory behind has been carefully discussed with verified hypernetwork gradients. The key idea of decoupling is well introduced and proved.\n\nWeaknesses:\n\n1. Some discussion of related work seems to be missing. Part of this work seems similar to “Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing”. Maybe adding further discussion in the related work section could be helpful. \n\n2. In methodology section (Section 4), how the agent-specific parameters ($\\theta $) are integrated into original RL networks seems not to be clear. \n\n3. The argument in Section 3 seems unsatisfactory. There seems to be some small mistakes.\n\nQuality: 2.9 $\\approx $ 3 (From introduction to conclusion section, the discussion of parameter sharing is comprehensive.); \n\nClarity: 3.3 $\\approx $ 3 (Overall, this paper is well-written with clear definition of research problem. It is easy to follow the idea in this paper.); \n\nSignificance: 2.4 $\\approx $ 2 (Parameter sharing is a well-discussed topic. It seems that part of this work is considered by previous work?); \n\nOriginality: 2.4 $\\approx $ 2 (I am more likely to view this paper as an extension of previous parameter sharing work?); \n\nRating: 4.5 $\\approx $ 5 (Overall, this is an interesting paper. If authors can address my concerns, I will confirm the score to 5.);\n\nConfidence: 3.6 $\\approx $ 4 (The paper is good and easy to follow. But I am not quite sure about the key difference between it and previous work.).", "was_revised": true, "final_justification": "Based on the rebuttal and subsequent discussion, I believe the authors have addressed most of my concerns. I would like to confirm my score as 5 and recommend accepting.", "rebuttal": "We thank reviewer `QQ2N` for their thorough and thoughtful review. We appreciate the reviewer's positive feedback on our novel insight into gradient decoupling, our comprehensive experimental evaluation, and our paper's clarity. We address your points as follows:\n\n## **1. SePS[1] - Related work (W1, Q2)**\n\nWe thank the reviewer for this point. SePS [1] is a parameter sharing method that uses an autoencoder during *pre-training* to cluster agents based on their transitions and shares parameters between these agent groups. This differs from our work, as we 1) do not use a *pre-training* phase to collect transitions and cluster agents, 2) do not maintain $K$ per-agent networks, but rather generate these networks on the fly based on agent embeddings, and 3) we use agent-conditioned hypernetworks that explicitly decouple observation- and agent-conditioned gradients (Section 4.3). This decoupling helps mitigate gradient interference between agents. Although we briefly mentioned SePS in our related work, we will improve our discussion of this work. \n\n**New Baseline and Environment:** Furthermore, since this work was mentioned as closely related by two reviewers, we conducted additional experiments comparing HyperMARL to SePS. We use their original source code and hyperparameters and evaluate on all their Blind Particle Spread settings (v1-4), which include up to **30 agents**.  \n\nIQM final total reward (with 95% confidence intervals) across 5 seeds on the BPS environments from [1], Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.\n\n| Alg | BPS-1 (15 agents, 3 groups)  | BPS-2 (30 agents, 3 groups)  | BPS-3 (30 agents, 5 groups) | BPS-4 (30 agents, 5 uneven groups) | \n|-----------|----|----|----|----|\n| NOPS | **-205.4** (-242.2, -197.8) | **-405.1** (-468.5, -379.9) | *-420.3 (-439.9, -407.4) | **-413.5** (-449.8, -399.1) |\n| FUPS-ID | *-219.8 (-249.7, -207.1) | *-442.6 (-467.4, -383.7) | -764.7 (-1326.7, -515.3) | -705.9 (-933.3, -591.1) |\n| SEPS | *-216.3 (-233.5, -191.8) | *-413.0 (-464.9, -385.1) | **-403.1** (-423.7, -352.9) | *-418.3 (-436.0, -393.5) |\n| HyperMARL | *-205.6 (-223.3, -175.1) | *-421.0 (-436.7, -401.0) | *-414.0 (-442.8, -374.2)  | *-428.5 (-462.7, -376.3) |\n\n**Takeaway**: The results highlight HyperMARL's competitive performance. In all four scenarios, its 95% confidence intervals overlap with SePS and NoPS, indicating it is statistically on par with these baselines. Notably, SePS relies on a **pre-training clustering phase**, whereas HyperMARL learns end-to-end without pre-training.\n\n## 2. **Agent-Specific Parameters Integrated into the Agent’s Networks (W2, Q5)**\n\nA hypernetwork generates the agent-specific parameters $\\theta^i$ that form all the weights (and biases) of policy $\\pi_{\\theta^i}$ (can also be seen in Fig. 1).\n\nFor RNNs,  HyperMARL maintains the same architecture as for feedforward methods, generating only the actor and critic feedforward weights, not the GRU weights (we also mention this in G.2, but we will highlight this in the main text). The RNN hidden state is then fed into the networks as input, as is common in recurrent architectures in RL. \n\n## 3. **Could specialisation lead to redundant diversification policies? Could specialisation lead to a decrease in adaptability? (Q1)**\n\nThank you for these insightful questions. Our opinion is that enforced specialisation, for example enforcing a diversity loss/objective, could lead to redundant diversification when not necessary to solve a task, since the mutual information objective could simply enforce different agent policies to be different, irrespective of the task. This could also hamper adaptability to tasks that do not require specialisation. This is the exact reason we do not *enforce specialisation*, but rather *enable* it, by learning end-to-end when specialised or homogeneous behaviours are required. To validate this, we experimentally test our approach in a wide array of settings, requiring homogeneous, heterogeneous, or mixed behaviours.\n\n## 4. **FuPS+ID in the Synchronisation Environment (Q4)**\n\nFuPS+ID performs well in the Synchronisation environment with a few agents, but as the number of agents increases, this environment becomes more challenging and FuPS+ID performance drops. This is because the probability of getting the maximum reward (all agents selecting the same action) during exploration is $\\frac{n!}{n^n}$, where $n$ is the number of agents e.g. $n=8$, $Pr(X = max_reward)=0.002$. This means that as $n$ grows large, it is harder for agents to synchronise. Agents get rewards for partial matches as described in section E.1 in the Appendix, but this is still a challenging environment. We will make this clearer in the updated manuscript.   \n\n## 5. **FuPS in the Gradient Conflict Section (3.2) (Q5)**\n\nThe primary goal of Section 3.2 was to investigate why FuPS+ID struggles to specialise, despite being a universal approximator ($\\pi_{\\theta}(a^i|o^i,id^i)$ should be able to approximate $\\pi_{{\\theta}_i}$). As you note, our analysis in this section omits the plain FuPS baseline because it has a more fundamental limitation: a shared policy without agent IDs cannot learn distinct agent behaviours. We prove this formally for the 2-player Specialisation Game in Appendix E.3. The proof and empirical results shown in Table 3 show that agent IDs can be beneficial. \n\nHowever, our central claim is that the method of providing this ID is critical. The FuPS+ID (No State) ablation, which outperforms standard FuPS+ID by ignoring observations, demonstrates that naively coupling the observation and ID inputs introduces destructive gradient interference. Therefore, the conclusion about the \"Importance of Observation and ID Decoupling\" still holds -- while IDs are necessary for specialisation, they must be decoupled from observation processing to be most effective. We will clarify this in the paper. \n\n## 6. **Can the inputs to the hypernetworks and policy network be swapped (i.e. use observation as input to the hypernetworks, …) (Q5)?**\n\nThis is an interesting idea, but we believe this will likely not work. Swapping the inputs would re‑entangle ID and observations. The agent‑conditioned Jacobian in Eq. (4) would become observation‑dependent (not deterministic with respect to mini-batch samples), removing the variance/decoupling benefit that HyperMARL relies on. Furthermore, this would result in other challenges as an entirely new policy would be generated every timestep.  The agents would also effectively be blind since the policy is not conditioned on observations, and therefore would likely not perform well in most environments (even if the weights were generated conditioned on the observations).\n\n## 7. **Minor Wording (Q3)**\n- \"PG-FuPS+ID\" - Thanks for pointing this out. Apologies, \"PG-FuPS+ID\" is the same as \"FuPS+ID\"; PG stands for policy gradient. We will fix this minor wording error in the updated manuscript. \n- Lines 108-109 - Thanks for the clarifying question. Our claim regarding \"scalability\" refers not to FuPS's performance degradation, but to the widening performance gap between the FuPS and NoPS architectures as the number of agents increases, i.e. for larger values of $n$, the gap between FuPS and NoPS increases (NoPS performing better in Specialisation, FuPS in Synchronisation). We will clarify the wording accordingly.\n\n\n## **Conclusion**\n\nIn summary, based on your feedback we added:\n- A **new baseline** SePS and environment, Blind Particle Spread. Extending our evaluation to **22 scenarios, up to 30 agents and six baselines** (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline. \n- Clarified the integration of agent-specific parameters into RL networks, the interplay between specialisation and redundant diversification, and other clarifications.  \n- Clarified the FuPS results in the Synchronisation Environment and the Gradient Conflict Section.  \n\nDuring the rebuttal, we have also added:\n- A **sensitivity analysis** using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment. \n- Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4). \n- A **learned agent embeddings analysis** showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.\n- A comparison with **dual-channel networks**, showing our approach is more effective.\n\nWe would like to thank the reviewer again for their thorough review and questions. We hope these clarifications and additional experiments have fully addressed the points raised, and if you have any follow-up questions or comments, please let us know, and we will be happy to discuss further.\n\n\nReferences:\n\n1] Christianos, F., Papoudakis, G., Rahman, M.A. and Albrecht, S.V., 2021, July. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR. \n\n2] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989."}]

Author rebuttal:

Meta-review: Parameter-sharing (PS) techniques are widely used in MARL agent networks due to sample efficiency. However, it may lead to a lack of policy diversity due to cross-agent gradients. The authors find that cross-agent gradients can be exacerbated by the coupling of observations and agent IDs. To address this problem, the authors propose HyperMARL, which uses hypernetworks for dynamic-agent-specific parameters, without introducing new RL objectives or sparsity. The authors verify the effectiveness of HyperMARL by comparing it against multiple baselines across multiple benchmarks.

The strengths of this work are summarized as follows.
1. Parameter-sharing techniques are important for MARL. 
2. The idea that using hypernetworks for agent-id specific parameters, thus to decouple agent-wise gradient conflicts, is novel.
3. The paper is well written.
4. Multiple experiments are conducted. The learned agent embeddings demonstrate that the model learns meaningful representations.

The weaknesses of this work are summarized as follows. 
1. Some related works are missing. The authors promised that they will discuss them in the new revision.
2. The evaluation is limited. It does not have at least one benchmark that incorporates all of the relevant baselines (Reviewer bHoX, z98m).
3. SMAC/SMACv2 are not tested (Reviewer z98m)

Please include the new experimental results and the promised changes in the paper if it is accepted for publication.
Decision: Accept (poster)