Title: HyperMARL: Adaptive Hypernetworks for Multi-Agent RL
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Parameter Sharing

Specialisation

Behavioural Diversity
Abstract: Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of *coupling agent IDs with observations*. Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates -- raising a fundamental question: *can shared policies adapt without these intricacies?* We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and *decouple* observation- and agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, **HyperMARL**, avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL.

HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

Abstract

Adaptive cooperation in multi-agent reinforcement learning (MARL) requires policies to express homogeneous, specialised, or mixed behaviours, yet achieving this adaptivity remains a critical challenge. While parameter sharing (PS) is standard for efficient learning, it notoriously suppresses the behavioural diversity required for specialisation. This failure is largely due to cross-agent gradient interference, a problem we find is surprisingly exacerbated by the common practice of coupling agent IDs with observations . Existing remedies typically add complexity through altered objectives, manual preset diversity levels, or sequential updates - raising a fundamental question: can shared policies adapt without these intricacies? We propose a solution built on a key insight: an agent-conditioned hypernetwork can generate agent-specific parameters and decouple observationand agent-conditioned gradients, directly countering the interference from coupling agent IDs with observations. Our resulting method, HyperMARL , avoids the complexities of prior work and empirically reduces policy gradient variance. Across diverse MARL benchmarks (22 scenarios, up to 30 agents), HyperMARL achieves performance competitive with six key baselines while preserving behavioural diversity comparable to non-parameter sharing methods, establishing it as a versatile and principled approach for adaptive MARL. The code is publicly available at https://github.com/KaleabTessera/HyperMARL .

1 Introduction

Specialist and generalist behaviours are critical to collective intelligence, enhancing performance and adaptability in both natural and artificial systems [52, 43, 44, 23, 50]. In Multi-Agent Reinforcement Learning (MARL) [2], this translates to a critical need for policies that can flexibly exhibit specialised, homogeneous, or mixed behaviours to meet diverse task demands [26, 6].

Optimal MARL performance thus hinges on being able to represent the required behaviours. While No Parameter Sharing (NoPS) [30] enables specialisation by using distinct per-agent networks, it suffers from significant computational overhead and sample inefficiency [11]. Conversely, Full Parameter Sharing (FuPS) [45, 18, 15], which trains a single shared network, improves efficiency but typically struggles to foster the behavioural diversity necessary for many complex tasks [24, 17, 26].

This failure of FuPS, particularly for diverse behaviours, was hypothesised to be gradient interference among agents, whereby their updates negatively impact each other's learning [11, 54]. We not only empirically validate this hypothesis but also demonstrate a critical insight: this conflict is significantly exacerbated by the common practice of coupling observations with agent IDs within a shared network (Fig. 1 for coupling, Sec. 3.2 for results).

Hypernetwork

ID Encoder

Agent-specific

weights

Loss

Loss

Grads

Obs-Cond

Grads

Agent-Cond Grads

Decoupled

Coupled

Balancing FuPS efficiency with the capacity for diverse behaviours therefore remains a central open problem in MARL. Prior works have explored intrinsic-rewards [26, 22], role-based allocations [48, 49], specialised architectures [24, 28, 6], sequential updates [54], and sharing parameters within clusters of agents [11]. However, these remedies introduce their own intricacies: they often alter the learning objective, require prior knowledge of optimal diversity levels, necessitate maintaining agent-specific parameters or require sequential updates. This raises a fundamental question: Can we design a shared MARL architecture that flexibly supports both specialised and homogeneous behaviours-without altered learning objectives, manual preset diversity levels, or sequential updates?

Guided by our observation-ID coupling insight, we propose HyperMARL , a novel agent-conditioned hypernetwork [19] architecture. HyperMARL generates per-agent weights on the fly (Fig. 1) and explicitly decouples observation- and agent-conditioned gradients (Sec. 4.3). This choice is motivated by hypernetworks' proven effectiveness at resolving gradient conflicts in multi-task RL [33] and continual learning [47]. Our work establishes their effectiveness for the problem of cross-agent interference in MARL. Indeed, HyperMARL empirically attains lower policy gradient variance than FuPS, and we show this decoupling is critical for specialisation (Sec. 5.2, 6.1), confirming its role in mitigating interference.

We validate HyperMARL on diverse MARL benchmarks - including Dispersion and Navigation (VMAS) [5], Multi-Agent MuJoCo (MAMuJoCo) [37], SMAX [41], and Blind-Particle Spread (BPS) [11] - across environments with two to thirty agents that require homogeneous, heterogeneous, or mixed behaviours. HyperMARL consistently matches or outperforms NoPS, FuPS, and diversitypromoting methods such as Diversity Control (DiCo) [6], Heterogeneous-Agent Proximal Policy Optimisation (HAPPO) [54], Kaleidoscope [28] and Selective Parameter Sharing (SePS) [11], while achieving NoPS-level behavioural diversity while using a shared architecture.

Our contributions are as follows:

We identify that cross-agent gradient interference in shared policies is critically exacerbated by the common practice of coupling agent IDs with observations (Sec. 3.2).

We propose HyperMARL (Sec. 4), an agent-conditioned hypernetwork architecture, to test the hypothesis that explicitly decoupling these gradients enables adaptive (diverse, homogeneous, or mixed) behaviours without the complexities of prior remedies (e.g., altered objectives, preset diversity levels, or sequential updates).

Our extensive evaluation (Sec. 5) across 22 diverse scenarios (up to 30 agents) shows HyperMARL achieves competitive returns against six strong baselines, while achieving NoPS-level behavioural diversity. We further show this decoupling is empirically linked to reduced policy gradient variance and is critical for specialisation (Sec. 5.2; Sec. 6.1).

2 Background

We formulate the fully cooperative multi-agent systems addressed in our work as a Dec-POMDP [34]. A Dec-POMDP is a tuple, ⟨ I , S , { A i } i ∈ I , R, { O i } i ∈ I , O, T, ρ 0 , γ ⟩ , where I is the set of agents of size

P2

A

B

P1

A

(0.5,0.5)

(1,1)

B

(1,1)

(0.5,0.5)

Two-Player Payoff

matrix

+1

+1

+1

...

...

N

-player Interaction

P2

A

B

P1

A

(1,1)

(0.5,0.5)

B

(0.5,0.5)

(1,1)

Two-Player Payoff

matrix

+1

+1

+1

...

...

N

-player Interaction

(a)

Specialisation Game

(b)

Synchronisation Game

n = | I | , S is the set of global states with an initial state distribution ρ 0 , A i is the action space for agent i where A = × i A i is the joint action space, R : S × A → R is the shared reward function, O i is the observation space for agent i with the joint observation space O = × i O i , O : O × A × S → [0 , 1] is the probability of joint observation o ∈ O , i.e. O ( o , a , s ) = Pr( o t | s t , a t -1 ) , T : S × A × S → [0 , 1] is the state transition function i.e. T ( s, a , s ′ ) = Pr ( s t +1 | s t , a t ) and γ is the discount factor.

In this setting, each agent i receives a partial observation o i t ∈ O i . These observations are accumulated into an action-observation history h i t = ( o i 0 , a i 0 , . . . , o i t -1 , a i t -1 , o i t ) . Each agent i acts based on their decentralised policies π i ( a i | h i ) . The joint history and joint action are defined as follows h t = ( h 1 t , . . . , h n t ) and a t = ( a 1 t , . . . , a n t ) . The goal is to learn an optimal joint policy π ∗ = ( π 1 ∗ , ..., π n ∗ ) that maximizes the expected discounted return as follows, 1 π ∗ = arg max π E s 0 ∼ ρ 0 , h ∼ π [ G ( h )] , where G ( h ) = ∑ ∞ t =0 γ t R ( s t , a t ) .

Specialised Policies and Environments. We say an environment is specialised if its optimal joint policy contains at least two distinct, non-interchangeable agent policies ( Def. 1 in App. C). Under this mild condition, tasks such as Dispersion (5.2) or our Specialisation Game (E.1) require agents to learn complementary roles rather than identical behaviours.

3 Are Independent or Fully Shared Policies Enough?

Standard independent ( NoPS ) and fully parameter-shared ( FuPS ) policies face inherent trade-offs in MARL. NoPS allows for uninhibited agent specialisation but can be sample inefficient and computationally expensive. FuPS, often conditioned with an agent ID (FuPS+ID), is more efficient but can struggle when agents must learn diverse behaviours [11]. This section investigates the limitations of these common policy architectures.

To probe these limitations, we introduce two illustrative environments: the Specialisation Game , rewarding distinct actions, and the Synchronisation Game , rewarding identical actions. Both are inspired by prior work [10, 17, 5, 35] and extended here to N -agent and temporal settings where agents observe prior joint actions (see Appendix E for full definitions).

3.1 Limitations of Fully Shared and Independent Policies

FuPS without agent IDs provably cannot recover optimal pure Nash equilibria in the non-temporal 2-player Specialisation Game (Proof E.3, App. E). In practice, however, FuPS is often conditioned with agent IDs, and MARL policies must handle complexities beyond static, two-player interactions. We therefore evaluate standard architectures in the temporal n -player versions of these games 2 We compare three standard architectures trained with REINFORCE [51]: 1) NoPS : independent

2

4

8

16

Number of Agents / Actions

0.0

0.2

0.4

0.6

0.8

Avg. Eval. Reward

(a) Avg. evaluation reward

5

10

15

Number of Agents / Actions

-1.00

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

1.00

Avg. Gradient Conflict

Complete

Alignment (+1)

Orthogonal (0)

Complete

Conflict (-1)

(b) Avg. gradient conflict

NoPS

FuPS+ID

FuPS+ID (No State)

HyperMARL

policies ( π θ i ( a i | o i ) ); 2) FuPS : a single shared policy ( π θ ( a i | o i ) ); and 3) FuPS+ID : a shared policy incorporating a one-hot agent ID ( π θ ( a i | o i , id i ) ). All use single-layer networks, 10-step episodes and 10 , 000 training steps (further details in Table 9 in App. I).

Empirical Performance. Table 1 shows that neither NoPS nor FuPS consistently achieves the highest mean evaluation rewards. NoPS excels in the Specialisation Game but is outperformed by FuPS (optimal) and FuPS+ID in the Synchronisation Game. Furthermore, the performance gaps widen as the number of agents increases (notably at n = 8 and n = 16 ), highlighting the scalability challenges of both fully independent and fully shared policies.

3.2 Why FuPS+ID Fails to Specialise: The Problem of Gradient Conflict

Despite being a universal approximator [20], FuPS+ID often struggles to learn diverse policies in practice (Table 1, [11, 54]). A key reason is gradient conflict : when a single network processes both observation o and agent ID id i , updates intended to specialise agent i (based on id i ) can conflict with updates for agent j (based on id j ), particularly if they share similar observations but require different actions. This obstructs the emergence of specialised behaviours (conflict measured via inter-agent gradient cosine similarity, App. E.4).

Importance of Observation and ID Decoupling. To investigate the effect of entangled observation and ID inputs, we introduce an ablation: FuPS+ID (No State) , where the policy π θ ( a i | id i ) conditions only on the agent ID, ignoring observations. Surprisingly, FuPS+ID (No State) outperforms standard FuPS+ID in the Specialisation Game for all tested N (Figure 3a), even when N ≤ 4 (where observation spaces are small, suggesting the issue is not merely observation size). Figure 3b reveals why: FuPS+ID (No State) shows near-zero gradient conflict (nearly orthogonal gradients), whereas standard FuPS+ID exhibits negative cosine similarities (conflicting gradients).

These results show that naively coupling observation and ID inputs in shared networks can lead to destructive interference, hindering specialisation. While discarding observations is not a general solution (most tasks require state information), this finding motivates designing architectures that can leverage both state and agent IDs, while minimising interference. Section 4 introduces HyperMARL

(Figure 1), which explicitly decouples observation- and agent-conditioned gradients through agentconditioned hypernetworks, leading to improved performance over FuPS variants and reduced gradient conflict compared to standard FuPS+ID (Figure 3).

4 HyperMARL

We introduce HyperMARL , an approach that uses agent-conditioned hypernetworks to learn diverse or homogeneous policies end-to-end , without modifying the standard RL objective or requiring manual preset diversity levels. By operating within a fully shared paradigm, HyperMARL leverages shared gradient information while enabling specialisation through the decoupling of observation- and agent-conditioned gradients. We present the pseudocode in Sec. F.1, with additional scaling (F.3) and runtime (F.4) details.

4.1 Hypernetworks for MARL

As illustrated in Figure 1, for any agent i with context e i (i.e., either a one-hot encoded ID or a learned embedding), the hypernetworks generate the agent-specific parameters:

where h π ψ and h V φ are the hypernetworks for the policy and critic, respectively. The parameters θ i and ϕ i define each agent's policy π θ i and critic V ϕ i , dynamically enabling either specialised or homogeneous behaviours as required by the task.

Linear Hypernetworks Given a one-hot agent ID, 1 i ∈ R 1 × n , a linear hypernetwork h π ψ generates agent-specific parameters θ i as follows 3 :

where W ∈ R n × m is the weight matrix (with m the per-agent parameter dimensionality and n is the number of agents), and b ∈ R 1 × m is the bias vector. Since 1 i is one-hot encoded, each θ i corresponds to a specific row of W plus the shared bias b . If there is no shared bias term, this effectively replicates training of separate policies for each task (in our case, for each agent) [3], since there are no shared parameters and gradient updates are independent.

MLP Hypernetworks for Expressiveness To enhance expressiveness, MLP Hypernetworks incorporate hidden layers and non-linearities:

where g π ψ 2 is an MLP processing the agent context e i , and f π ψ 1 is a final linear output layer.

Unlike linear hypernetworks with one-hot agent IDs, MLP hypernetworks do not guarantee distinct weights for each agent. Additionally, they increase the total number of trainable parameters, necessitating a careful balance between expressiveness and computational overhead.

4.2 Agent Embeddings and Initialisation

The agent embedding e i is a one-hot encoded ID for Linear Hypernetworks. For MLP Hypernetworks, we use learned agent embeddings, orthogonally initialised and optimised end-to-end with the hypernetwork. HyperMARL's hypernetworks are themselves initialised such that the generated agent-specific parameters ( θ i , ϕ i ) initially match the distribution of standard direct initialisation schemes (e.g., orthogonal for PPO, preserving fan in/out), promoting stable learning.

4.3 Gradient Decoupling in HyperMARL

A core difficulty in FuPS is cross-agent gradient interference [11, 54]. HyperMARL mitigates this by generating each agent's parameters through a shared hypernetwork, thereby decoupling agent-conditioned and observation-conditioned components of the gradient.

Hypernetwork gradients. Consider a fully cooperative MARL setting with a centralised critic, we can formulate the policy gradient for agent i as follows [2, 25]:

where h t and a t are the joint histories and joint actions for all agents, θ i denotes the parameters of agent i , and A ( h t , a t ) = Q ( h t , a t ) -V ( h t ) is the advantage function.

Decoupling. In HyperMARL each agent's policy weights are produced by the hypernetwork h π ψ : θ i = h π ψ ( e i ) , so we optimise a single parameter vector ψ . Applying the chain rule and re-ordering the expectations:

Agent-conditioned factor J i . This Jacobian depends only on the fixed embedding e i and the hypernetwork weights ψ , therefore, it is deterministic with respect to mini-batch samples (as e i and ψ are fixed per gradient step), separating agent identity from trajectory noise.

Observation-conditioned factor Z i . The expectation averages trajectory noise per agent i for its policy component π θ i , prior to transformation by J i and aggregation.

The crucial structural decoupling in Equation (4) ensures HyperMARL first averages noise per agent (via factor Z i ) before applying the deterministic agent-conditioned transformation J i . This mitigates gradient interference common in FuPS+ID [11, 54], where observation noise and agent identity become entangled (see Equation (12) in App. F.2). This is the MARL analogue of the task/state decomposition studied by [42, Eq. 18] in Meta-RL. Section 5.2 empirically verifies the predicted variance drop, and ablations in Section 6.1 demonstrate that disabling decoupling degrades performance, underscoring its critical role.

5 Experiments

Our empirical evaluation of HyperMARL assesses whether agent-conditioned hypernetworks can enable adaptive (specialised or homogeneous) policies without altered RL objectives, preset diversity levels, or sequential updates. We structure our experiments to directly answer two key research questions: Q1: Specialised Policy Learning: Can HyperMARL effectively learn specialised policies via a shared hypernetwork? Q2: Effectiveness in Homogeneous Tasks : Is HyperMARL competitive in environments that necessitate homogeneous behaviours?

To address these questions, HyperMARL is evaluated against six representative modern baselines across a carefully selected suite of MARL benchmarks (22 scenarios, up to 30 agents). All experiments use at least 5 seeds (details in App. G.2.1).

5.1 Experimental Setup

Environments. HyperMARL is evaluated across 22 scenarios from five diverse MARL environments (Dispersion [5], Navigation [5], MAMuJoCo [37], SMAX [41], BPS [11]) (Table 2). These were specifically chosen to rigorously test performance across varying complexities, agent counts (2 to 30), and distinct behaviours (heterogeneous, homogeneous, or mixed). Full details in Appendix G.1.

Baselines. We evaluate HyperMARL against modern parameter sharing (PS) and diversity-promoting baselines. Core PS comparisons use FuPS+ID and NoPS . For specialisation, we include privileged baselines: DiCo [6] (shared and non-shared weights, preset diversity levels); HAPPO [54] (shared critic, sequential actor updates); Kaleidoscope [28] (learnable masks, critic ensembles, diversity loss); and SePS (pre-training phase, agent clustering).

FuPS+ID

NoPS

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Training Steps (M)

0.75

0.50

0.25

0.00

0.25

0.50

0.75

IQM Mean Episode Return

(a) IPPO

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Training Steps (M)

0.75

0.50

0.25

0.00

0.25

0.50

0.75

IQM Mean Episode Return

(b) MAPPO

FuPS+ID

Methods

0.00

0.01

0.02

0.03

0.04

0.05

Actor Gradient Variance

×10 -4

IPPO

MAPPO

FuPS+ID

NoPS

Methods

0

2

4

6

JS Distance

×10 -4

IPPO

MAPPO

(d) Policy Diversity

We use IPPO/MAPPO [13, 53] as the underlying algorithm for all methods except Kaleidoscope and SePS (see App. H.1 and App. H.2 for these results).

Adhering to best evaluation practices [36], we use original codebases/hyperparameters and environments for which baselines were tuned . HyperMARL uses identical observations and generates architectures of equivalent capacity to baselines. Training and evaluation (App. G.2.1) and hyperparameters (App. I) follow each baseline's original setup. We detail our baseline and environment selection criteria in Table 6, with architecture details in App. G.2.

Measuring Policy Diversity. To measure the diversity of the policies we System Neural Diversity (SND) [7] (Equation 5) with Jensen-Shannon distance (details in App. G.2.2).

5.2 Q1: Specialised Policy Learning

Learning Diverse Behaviour (Dispersion) Figures 4a and 4b show that FuPS variants (IPPO-FuPS, MAPPO-FuPS - ( · )) can struggle to learn the diverse policies required by Dispersion (even when running for longer - Fig. 19), while their NoPS counterparts (IPPO-NoPS, MAPPO-NoPS-( · )) converge to the optimal policy, corroborating standard FuPS limitations to learn diverse behaviour. In contrast, HyperMARL (both linear and MLP variants) ( · , · ) match NoPS performance, suggesting that a shared hypernetwork can effectively enable agent specialisation. SND policy diversity measurements (Fig. 4d) confirm FuPS variants achieve lower behavioural diversity than NoPS, while HyperMARL notably matches NoPS-level diversity.

Gradient Variance. HyperMARL (IPPO and MAPPO variants) also exhibits lower mean policy gradient variance than FuPS+ID across actor parameters (Fig. 4c). This aligns with their ability to learn diverse behaviours and supports the hypothesis that its gradient decoupling mechanism (Sec. 4.3) enhances training stability.

Diversity at Complexity and Scale (MAMuJoCo). In the challenging MAMuJoCo heterogeneous control tasks (Table 3), HyperMARL (MLP variant) is broadly competitive. Notably, unlike HAPPO and MAPPO (independent actors), HyperMARL uses a shared actor and parallel updates, and yet manages strong performance, even in the 17-agent Humanoid-v2 notoriously difficult heterogeneous task[54] (Fig. 5), matching methods that employ independent actors and sequential updates.

Adaptability (Navigation). Navigation tasks [5] assess adaptability to homogeneous, heterogeneous, and mixed goals (some agents have the same goals, others different). We compare HyperMARL with baselines including DiCo [6]. While using

0

2

4

6

8

10

Training Steps (M)

0

1000

2000

3000

4000

5000

6000

7000

IQM Mean Episode Return

MAPPO (FuPS+ID)

MAPPO (Independent Actors)

HyperMARL (MAPPO)

HAPPO

DiCo's optimal preset diversity for n=2 agents, we note that identifying appropriate diversity levels

Scenario

HAPPO

FuPS+ID

Ind. Actors

HyperMARL (Ours)

Humanoid-v2 17x1

6501.15

*

(3015.88, 7229.79)

566.12 (536.36, 603.01)

6188.46

*

(5006.13, 6851.74)

6544.10

(3868.00, 6664.89)

4748.06

*

(4366.94, 6230.81)

4574.39

*

(4254.21, 5068.32)

4747.05

*

(3974.76, 6249.58)

5064.86

(4635.10, 5423.42)

HalfCheetah-v2 2x3

6752.40

*

(6130.42, 7172.98)

6771.21

*

(6424.94, 7228.65)

6650.31

*

(5714.68, 7229.61)

7063.72

(6696.30, 7325.36)

6031.92

*

(5924.32, 6149.22)

6148.58

(5988.63, 6223.88)

6046.23

*

(5924.62, 6216.57)

5940.16

*

(5485.77, 6280.59)

0.026

0.028

0.030

FuPS

NoPS

DiCo

2 Agents

0.022

0.024

0.026

0.028

4 Agents

0.020

0.022

0.024

0.026

8 Agents

Average Reward

0.024

0.027

0.030

FuPS

NoPS

DiCo

2 Agents

0.023

0.024

0.026

0.027

4 Agents

0.017

0.020

0.022

0.025

8 Agents

Average Reward

(a) Shared Goals

(b) Unique Goals

0.025

0.026

0.027

FuPS

NoPS

DiCo

4 Agents (2 Goals)

0.017

0.020

0.022

0.025

8 Agents (4 Goals)

Average Reward

(c) Mixed Goals (Half Agents Share

Goals)

for DiCo with larger teams ( n > 2 ) via hyperparameter sweeps proved challenging (see Tables 15 and 16).

Across all tested goal configurations (shared, unique, and mixed), HyperMARL consistently achieves strong performance (Figure 6). It generally matches or outperforms NoPS and FuPS, and outperforming DiCo. Interestingly, unlike in sparse-reward tasks like Dispersion, FuPS remains competitive with NoPS and HyperMARL in Navigation scenarios requiring diverse behaviours for smaller teams ( n ∈ { 2 , 4 } ), likely due to Navigation's dense rewards. However, HyperMARL distinguishes itself as the strongest method for n=8 agents, highlighting its effectiveness in handling more complex coordination challenges.

5.3 Q2: Effectiveness in Homogeneous Tasks

SMAX. Finally, we evaluate HyperMARL (MLP) on SMAX, where recurrent FuPS is the established baseline [41, 53, 17]. Figure 7 shows while some FuPS variants might exhibit marginally faster initial convergence on simpler maps, HyperMARL achieves comparable final performance on all maps, using the same GRU backbone for partial observability. These results highlight two points: (i)

MAPPO RNN (FuPS)

IPPO RNN (FuPS)

HyperMARL (IPPO RNN)

HyperMARL (MAPPO RNN)

0

2

4

6

8

10

Environment Steps (M)

0.0

0.2

0.4

0.6

0.8

1.0

IQM Mean Win Rate

(a) 2s3z

0

2

4

6

8

10

Environment Steps (M)

0.0

0.2

0.4

0.6

0.8

1.0

IQM Mean Win Rate

(b) 3s5z

0

2

4

6

8

10

Environment Steps (M)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

IQM Mean Win Rate

(c) SMACv2 10 Units

0

2

4

6

8

10

Environment Steps (M)

0.0

0.1

0.2

0.3

0.4

0.5

IQM Mean Win Rate

(d) SMACv2 20 Units

0

2

4

6

8

10

Training Steps (M)

0

2000

4000

6000

IQM Mean Episode Return

FuPS+ID

HyperMARL

HyperMARL w/o GD

(a) Humanoid w/o GD

0

5

10

15

20

Training Steps (M)

-0.5

0.0

0.5

IQM Mean Episode Return

FuPS+ID

HyperMARL

HyperMARL w/o GD

(b) Dispersion w/o GD

0

2

4

6

8

10

Training Steps (M)

0

2000

4000

6000

IQM Mean Episode Return

FuPS+ID

HyperMARL

HyperMARL w/o RF

(c) Humanoid w/o RF

0

5

10

15

20

Training Steps (M)

-0.5

0.0

0.5

IQM Mean Episode Return

FuPS+ID

HyperMARL

HyperMARL w/o RF

(d) Dispersion w/o RF

HyperMARL is fully compatible with recurrent architectures essential under partial observability, and (ii) it has no intrinsic bias toward specialisation and can converge to homogeneous behaviour when it is optimal (also shown with strong same-goal Navigation performance (Fig. 6a)), even with large observation spaces and many agents.

Summary. Our empirical results confirm HyperMARL effectively addresses both research questions. For Q1 (Specialisation) , across Dispersion, MAMuJoCo, and Navigation, HyperMARL learned specialised policies, matched NoPS-level diversity and performance where FuPS+ID struggled, and scaled to complex, high-agent-count heterogeneous tasks. For Q2 (Homogeneity) , HyperMARL demonstrated competitive performance against strong FuPS baselines in SMAX and shared-goal Navigation, confirming its versatility.

6 Ablations and Embedding Analysis

6.1 Ablations: Gradient Decoupling and Initialisation Scaling

We ablate two critical components of HyperMARL: gradient decoupling (Sec. 4.3) and initialisation scaling (Sec. 4.2). In HyperMARL w/o GD , the hypernetwork is conditioned on [ o t , e i t ] , coupling observation and agent-ID gradients. In HyperMARL w/o RF , we remove the reset fan-in/out scaling that aligns the scale of generated parameters ( θ i , ϕ i ) with standard initialisers.

Gradient decoupling is essential; initialisation scaling grows with complexity. Figure 8 shows that removing GD consistently degrades performance across both Humanoid-v2 (17 agents) and Dispersion , showing that GD is an essential component of HyperMARL. Removing RF reveals a task-dependent effect: it is critical on Humanoid-v2 , consistent with hypernetwork initialisation results [8], but has a minor impact on Dispersion , indicating that principled initialisation becomes more vital with increased complexity. We provide additional ablations in App. H.7.

6.2 Analysis of Learned Agent Embeddings

Recall from 4.2 that each agent i possesses an embedding e i , which serves as input to the hypernetworks. For the MLP hypernetworks, these embeddings are learned end-to-end and are orthogonally initialised . Consequently, at initialisation (step 0), the pairwise cosine distance between any two distinct agent embeddings is 1 . 0 , mirroring the separability of one-hot IDs.

We probe how these embeddings e i adapt in the 4-agent Navigation task (same task as in Figure H.5) under two objectives with identical dynamics: (i) same goal (all agents navigate to a single shared target) and (ii) different goals (each agent navigates to its own unique target). At the end of training, we compute the mean pairwise cosine distance between agents' embeddings (lower values imply greater similarity) and compare it to the orthogonal baseline of 1 . 0 . (Cosine distance = 1 -cosine similarity; 0 = identical direction, 1 = orthogonal,

Same Goals

Different Goals

0.825

0.850

0.875

0.900

0.925

0.950

0.975

1.000

1.025

Mean Cosine Distance

Initialisation: 1.0

2 = opposite). Figure 9 shows the per-seed distributions, with the 1 . 0 initialisation value as a dashed line.

Agent embeddings adapt to task demands. When behaviour should be homogeneous ( same goal ), embedding directions become significantly more aligned, the mean pairwise cosine distance contracts to 0 . 882 ± 0 . 042 (one-sample t -test vs 1 . 0 : p = 0 . 0079 ). Conversely, when behaviour must differ ( different goals ), the embeddings maintain their separation , remaining near their orthogonal initialisation at 1 . 010 ± 0 . 017 . These results demonstrate that the hypernetwork actively modulates the agent embeddings based on the task, promoting alignment for homogeneity while preserving separability for specialisation. We also conduct a sensitivity analysis on HyperMARL's hyperparameters in App F.5 and see that agent embedding size can be an important hyperparameter, one that could correspond to the task's diversity requirements.

7 Related Work

Hypernetworks in RL and MARL. Hypernetworks are effective in single-agent RL for metalearning, multi-task learning, and continual learning [3, 4, 42, 21]. In MARL, QMIX [40] used hypernetworks (conditioned on a global state) to mix per-agent Q-values; however, each agent's own network remained a standard GRU. Parallel work, CASH [16], conditions a hypernetwork on local observations and predefined capability descriptors to target zero-shot generalisation across heterogeneous action spaces. By contrast, we use agent-conditioned hypernetworks in homogeneous action spaces, conditioning only on agent IDs/learned embeddings, and we explicitly decouple agent-conditioned from observation-conditioned gradients - a mechanism absent in CASH - which we find critical for specialisation.

Variants of Parameter Sharing. While Full Parameter Sharing (FuPS) is the most common approach, several other variants exist. Selective Parameter Sharing (SePS) [11] shares weights between similar groups of agents, identified by clustering agent trajectories during a pre-training phase. Pruning methods (SNP-PS, Kaleidoscope) [24, 28] split a single network into agent-specific subnetworks using learned agent masks. AdaPS [27] combines clustering and pruning masks to create shared policies for different groups of agents. Concurrent to our work, GradPS [38] identifies neurons with conflicting gradient updates, clones these neurons, and assigns each clone to a group of agents with low gradient conflict. Unlike these works, HyperMARL does not rely on pre-training (SePS), clustering algorithms requiring a preset number of clusters (SePS, GradPS, AdaPS), intricate pruning hyperparameters (SNP-PS, Kaleidoscope), auxiliary diversity losses (Kaleidoscope), or gradient conflict thresholds (GradPS).

Learning Diverse Policies. Shared parameters often limit policy diversity [11, 24, 17, 26]. Proposed solutions include: maximising mutual information between agent IDs and trajectories [26], role-based methods that assign distinct roles to agents [48, 49], best-response [39] methods and approaches that use structural modifications or constraints to induce diversity [24, 6, 28, 27]. Outside FuPS/NoPS, HAPPO [54] uses a non-shared centralised critic with individual actors updated sequentially to learn heterogeneous behaviours. In contrast to these works, HyperMARL does not alter the learning objective, use sequential updates or require preset diversity levels.

8 Conclusion

We investigated why standard parameter sharing fails at behavioural diversity, identifying that cross-agent gradient interference is critically exacerbated by coupling agent IDs with observations. We hypothesised that explicitly decoupling these gradients would enable adaptivity without prior complexities, and confirmed this using our HyperMARL approach. Our results show this decoupling enables adaptive behaviours (up to 30 agents) and is linked to reduced policy gradient variance. These findings establish gradient decoupling via HyperMARL as a versatile, principled approach for adaptive MARL. We discuss limitations in App. A, most notably parameter count, which can be remedied by parameter-efficient hypernetworks (e.g., chunked variants [47, 9]).

References

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems , 2021.

Stefano V Albrecht, Filippos Christianos, and Lukas Schäfer. Multi-agent reinforcement learning: Foundations and modern approaches . MIT Press, 2024.

Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in metareinforcement learning. In Conference on Robot Learning , pages 1478-1487. PMLR, 2023.

Jacob Beck, Risto Vuorio, Zheng Xiong, and Shimon Whiteson. Recurrent hypernetworks are surprisingly strong in meta-rl. Advances in Neural Information Processing Systems , 36, 2024.

Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems , 2022.

Matteo Bettini, Ryan Kortvelesy, and Amanda Prorok. Controlling behavioral diversity in multi-agent reinforcement learning. In Forty-first International Conference on Machine Learning , 2024.

Matteo Bettini, Ajay Shankar, and Amanda Prorok. System neural diversity: Measuring behavioral heterogeneity in multi-agent learning. arXiv preprint arXiv:2305.02128 , 2023.

Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations , 2020.

Vinod Kumar Chauhan, Jiandong Zhou, Ghadeer Ghosheh, Soheila Molaei, and David A Clifton. Dynamic inter-treatment information sharing for individualized treatment effects estimation. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics , volume 238 of Proceedings of Machine Learning Research , pages 3529-3537. PMLR, 02-04 May 2024.

Filippos Christianos, Georgios Papoudakis, and Stefano V. Albrecht. Pareto actor-critic for equilibrium selection in multi-agent reinforcement learning. Transactions on Machine Learning Research (TMLR) , 2023.

Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning , pages 1989-1998. PMLR, 2021.

Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In 34th Conference on Neural Information Processing Systems , 2020.

Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533 , 2020.

Dominik Maria Endres and Johannes E Schindelin. A new metric for probability distributions. IEEE Transactions on Information theory , 49(7):1858-1860, 2003.

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems , 29, 2016.

Kevin Fu, Pierce Howell, Shalin Jain, and Harish Ravichandar. Learning flexible heterogeneous coordination with capability-aware shared hypernetworks. arXiv preprint arXiv:2501.06058 , 2025.

Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In International Conference on Machine Learning , pages 6863-6877. PMLR, 2022.

Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16 , pages 66-83. Springer, 2017.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. Continual model-based reinforcement learning with hypernetworks. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 799-805. IEEE, 2021.

Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 4992-5001. PMLR, 18-24 Jul 2021.

R Kassen. The experimental evolution of specialists, generalists, and the maintenance of diversity. Journal of evolutionary biology , 15(2):173-190, 2002.

Woojun Kim and Youngchul Sung. Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.00912 , 2023.

Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Haifeng Zhang, David Mguni, Jun Wang, Yaodong Yang, et al. Settling the variance of multi-agent policy gradients. Advances in Neural Information Processing Systems , 34:13458-13470, 2021.

Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems , 34:3991-4002, 2021.

Dapeng Li, Na Lou, Bin Zhang, Zhiwei Xu, and Guoliang Fan. Adaptive parameter sharing for multi-agent reinforcement learning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6035-6039. IEEE, 2024.

Xinran Li, Ling Pan, and Jun Zhang. Kaleidoscope: Learnable masks for heterogeneous multi-agent reinforcement learning. arXiv preprint arXiv:2410.08540 , 2024.

Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory , 37(1):145-151, 1991.

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems , 30, 2017.

Kevin R McKee, Joel Z Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems , 36(1):21, 2022.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PmLR, 2016.

Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypernetworks. arXiv preprint arXiv:2010.04104 , 2020.

Frans A Oliehoek and Christopher Amato. A concise introduction to decentralized POMDPs . Springer, 2016.

Martin J Osborne and Ariel Rubinstein. A course in game theory . MIT press, 1994.

Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research , 25(318):1-63, 2024.

Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems , 34:12208-12221, 2021.

Haoyuan Qin, Zhengzhu Liu, Chenxing Lin, Chennan Ma, Songzhu Mei, Siqi Shen, and Cheng Wang. Gradps: Resolving futile neurons in parameter sharing network for multi-agent reinforcement learning. In Forty-second International Conference on Machine Learning , 2025.

Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V. Albrecht. Generating teammates for training robust ad hoc teamwork agents via best-response diversity. Transactions on Machine Learning Research (TMLR) , 2023.

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research , 21(178):1-51, 2020.

Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments and algorithms in jax. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems , pages 2444-2446, 2024.

Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning , pages 9301-9312. PMLR, 2021.

Chris R Smith, Amy L Toth, Andrew V Suarez, and Gene E Robinson. Genetic and genomic analyses of the division of labour in insect societies. Nature Reviews Genetics , 9(10):735-748, 2008.

James Surowiecki. The Wisdom of Crowds . Doubleday, New York, 2004.

Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning , pages 330-337, 1993.

Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii , 5(3):64-72, 1969.

Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations , 2020.

Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning , pages 9876-9886, 2020.

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523 , 2020.

Katherine Y Williams and Charles A O'Reilly III. Demography and. Research in organizational behavior , 20:77-140, 1998.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8:229-256, 1992.

Anita Williams Woolley, Ishani Aggarwal, and Thomas W Malone. Collective intelligence and group performance. Current Directions in Psychological Science , 24(6):420-424, 2015.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems , 35:24611-24624, 2022.

Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneousagent reinforcement learning. Journal of Machine Learning Research , 25(32):1-67, 2024.

A Limitations

Hypernetworks generate weights for target networks, which can lead to high-dimensional outputs and many parameters for deep target networks, particularly when using MLP-based hypernetworks. While HyperMARL uses more parameters than NoPS and FuPS for few agents, it scales almost constantly with agent count, an attractive property for large-scale MARL. Parameter efficiency could be improved through chunking techniques [47, 9], or low-rank weight approximations. This parameter overhead is often acceptable in RL/MARL given typically smaller actor-critic networks, and HyperMARL's favorable agent scaling (see App. F.3).

B Broader Impact

This paper presents work whose goal is to advance the field of Multi-Agent Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

C Specialised Policies and Environments

Specialisation plays a key role in MARL, yet remains under-defined, so we define specialised environments and specialised policies .

Definition 1. An environment is specialised if the following both hold:

̸

Distinct Agent Policies. The optimal joint policy π ∗ consists of at least two distinct agent policies, i.e., ∃ i, j ∈ I such that π i = π j .

Non-Interchangeability. Any permutation σ of the policies in π ∗ , denoted as π σ , results in a weakly lower expected return:

with strict inequality if the joint policies are non-symmetric (i.e., swapping any individual policy degrades performance).

For example, consider a specialised environment such as a football game, optimal team performance typically requires players in distinct roles (e.g., "attackers," "defenders"). Permuting these roles (i.e., exchanging their policies) would typically lead to suboptimal results. Here, agents develop specialised policies by learning distinct, complementary behaviours essential for an optimal joint policy. While agents with heterogeneous capabilities (e.g., different action spaces) are inherently specialised, homogeneous agents can also learn distinct policies. Such environments are analysed in Sections E.1 and 5.2.

D Measuring Behavioural Diversity

D.1 Quantifying Team Diversity

We quantify policy diversity using System Neural Diversity (SND) [7], which measures behavioural diversity based on differences in policy outputs:

where n is the number of agents, O is a set of observations typically collected via policy rollouts, π i ( o t ) and π j ( o t ) are the outputs of policies i and j for observation o t , and D is our distance function between two probability distributions.

In contrast to [7], we use Jensen-Shannon Distance (JSD) [14, 29] as D , rather than the Wasserstein metric [46]. As shown in Appendix D.2, JSD is a robust metric for both continuous and discrete cases, and provides a more reliable measure of policy distance.

D.2 Finding a Suitable Distance Function for Policy Diversity

The choice of distance function D in Equation 5 is crucial for accurately measuring policy diversity. In MARL, policies are often represented as probability distributions over actions, making the choice of distance function non-trivial.

[6] use the Wasserstein metric for continuous policies [46] as distance function D , while [31] use the total variation distance for discrete policies. For discrete policies, Wasserstein distance would require a cost function representing the cost of changing from one action to another, which might not be trivial to come up with. On the other hand, although well-suited for discrete policies, TVD might miss changes in action probabilities because it measures the largest difference assigned to an event (i.e. action) between two probability distributions.

We consider a simple example to illustrate this point. Suppose we have two policies π 1 and π 2 with action probabilities as shown in Figure 10. π 1 stays constant, while π 2 changes gradually over timesteps. We see that even as π 2 changes over time, the TV D ( π 1 , π 2 ) between π 1 and π 2 remains constant. This is because TVD only measures the largest difference between the two distributions, and does not consider the overall difference between them. On the other hand, the Jensen-Shannon distance (JSD) [14], which is the square root of the Jensen-Shannon divergence, does not have this problem as it is a smooth distance function. Furthermore, it satisfies the conditions for being a metric - it is non-negative, symmetric, and it satisfies the triangle inequality.

For continuous policies, as shown in Figure 11, JSD exhibits similar trends to the Wasserstein distance. Since JSD is a reasonable metric for both continuous and discrete probability distributions, we will use it as the distance metric for all experiments and propose it as a suitable distance function for measuring policy diversity in MARL.

We also summarise the properties of the various distance metrics in Table 4.

A1

A2

A3

Actions

0.0

0.2

0.4

0.6

0.8

1.0

Probability

0.05

0.85

0.10

1  (Constant)

0

1

2

3

4

Time Step

0.0

0.2

0.4

0.6

0.8

1.0

0.85

0.66

0.47

0.29

0.10

0.05

0.05

0.05

0.05

0.05

0.10

0.29

0.47

0.66

0.85

2  (Changing over Time)

Action 1

Action 2

Action 3

0

1

2

3

4

Time Step

0.625

0.650

0.675

0.700

0.725

0.750

0.775

0.800

Distance

Distances between Policies

D

(

1 ,

2 )

TVD

JSD

0

1

2

3

4

5

Mean

0.0

0.2

0.4

0.6

0.8

1.0

Probability Density

1  (Constant)

1

0

1

2

3

4

5

Mean

0.0

0.2

0.4

0.6

0.8

1.0

Probability Density

2  (Converging to

1 )

t=0

t=1

t=2

t=3

t=4

0

1

2

3

4

Time Steps

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Distance

Distances between Policies

D

(

1 ,

2 )

JSD

Method

Kinds of Actions

Metric

Smooth

Wasserstein Distance [46]

Metric

Yes

W

(

p, q

) =

(

inf

γ

∈

Γ(

p,q

)

∫

R

×

R

|

x

-

y

|

dγ

(

x, y

)

)

1

/p

Total Variation Distance

Discrete

Metric

No

TV

(

p, q

) =

1

2

∑

x

|

p

(

x

)

-

q

(

x

)

|

Both

Divergence

Yes

JSD

(

p

∥

q

) =

1

2

D

KL

(

p

∥

m

) +

1

2

D

KL

(

q

∥

m

)

, m

=

1

2

(

p

+

q

)

Jensen-Shannon Distance [14]

Both

Metric

Yes

√

JSD

(

p

∥

q

)

E Specialisation and Synchronisation Games

To study the challenges of specialisation and coordination in an isolated setting, we introduce the Specialisation and Synchronisation Games, drawing inspiration from a version of the XOR game [17], VMAS's Dispersion [5] and coordination and anti-coordination games in game theory [35]. These environments encourage agents to take distinct actions (Specialisation) or take identical actions (Synchronisation). Despite their deceptively simple payoff structure, these games present substantial learning challenges - non-stationary reward distributions driven by others' adapting behaviours and in their temporal extension, the joint observation spaces grows exponentially with the number of agents.

E.1 Specialisation and Synchronisation Games Description

Specialisation Game. Agents are encouraged to choose distinct actions. In the simplest setting, it is a two-player matrix game where each agent selects between two actions ( A or B ). As shown in Figure 2a, agents receive a payoff of 1 . 0 when their actions differ (creating two pure Nash equilibria on the anti-diagonal) and 0 . 5 when they match. This structure satisfies Definition 1, since optimal joint policies require complementary, non-identical strategies. There is also a symmetric mixed-strategy equilibrium in which each agent plays A and B with probability 0 . 5 .

Synchronisation Game. Conversely, agents are encouraged to coordinate and choose identical actions. The payoff matrix inverts the Specialisation game's structure, agents receive 1 for matching actions and 0 . 5 for differing ones. This creates two pure Nash equilibria along the diagonal of the payoff matrix (Figure 2b), and incentivises uniform behaviour across agents.

N -Agent Extension. Both games naturally scale to n agents and n possible actions. In Specialisation, unique actions receive a payoff of 1 . 0 , while selecting the same action receives payoffs of 1 k , where k is the number of agents choosing that action. In contrast, in Synchronisation, agents receive maximum payoffs ( 1 . 0 ) only when all actions match. For partial coordination, rewards follow a hyperbolic scale, 1 n -k +1 , encouraging agents to align their choices. Visualisations in Figure 2 and detailed reward profiles appear in Figure 12.

E.2 Generaln Payoff Definitions

Both games generalise naturally to n agents and n possible actions. We show the reward profiles for n = 5 agents in Figure 12.

Let a = ( a 1 , . . . , a n ) ∈ { 1 , . . . , n } n and k a = ∣ ∣ { j : a j = a } ∣ ∣ be the joint action profile and the count of agents choosing action a , respectively.

Temporal Extension. To model sequential decision-making, we extend each normal-form game into a repeated Markov game, where the state at time t is the joint action at time t -1 . At each step t all agents observe a t -1 , choose a t i , and receive the original Specialisation or Synchronisation payoff. This repeated setup isolates how agents adapt based on past joint actions, exposing temporal patterns of specialisation and coordination.

E.2.1 Specialisation Game

The reward is formulated as follows:

E.2.2 Synchronisation Game

The reward is formulated as follows:

so that r i sync = 1 . 0 when k a i = n (all agents select the same action), and otherwise follows a hyperbolic scale encouraging consensus.

1

2

3

4

5

Number of Agents Choosing Same Action

0.0

0.2

0.4

0.6

0.8

1.0

Reward

1.00

0.20

0.50

0.25

0.33

0.33

0.25

0.50

0.20

1.00

Reward Profiles (n=5)

Specialisation

Synchronisation

E.3 Proof that FuPS cannot represent the optimal policy in the two-player Specialisation Game

Theorem 1. A stochastic, shared policy without agent IDs cannot learn the optimal behaviour for the two-player Specialisation Game.

Proof. Let π be a shared policy for both agents, and let α = P ( a i = 0) represent the probability of any agent choosing action 0. The expected return of π for each agent is:

Thus, E [ R ( π )] ≤ 0 . 75 < 1 for all α ∈ [0 , 1] , with the maximum at α = 0 . 5 . Therefore, a shared policy cannot achieve the optimal return of 1, confirming the need for specialised behaviour to optimise rewards.

E.4 Measuring Agent Gradient Conflict

We measure gradient conflicts , via the cosine similarity between agents' gradients cos ( g ( i ) t , g ( j ) t ) = ⟨ g ( i ) t ,g ( j ) t ⟩ ∥ g ( i ) t ∥∥ g ( j ) t ∥ , where g ( i ) t = ∇ θ L ( i ) ( θ t ) .

E.5 Empirical Results in N-player Specialisation and Synchronisation Normal-Form Game

To assess this limitations of FuPS and NoPS in practice, we compare three REINFORCE [51] variants in both games with n = 2 , 4 , 8 , 16 , 32 agents: NoPS (No Parameter Sharing), FuPS (Fully Parameter Sharing), and FuPS+ID (FuPS with one-hot agent IDs). All policies use single-layer neural networks with controlled parameter counts (see Appendix I for details).

4

8

16

32

Number of Agents / Actions

0.0

0.2

0.4

0.6

0.8

1.0

Avg. Evaluation Reward

(Mean ± Std. Error)

Specialisation Game

4

8

16

32

Number of Agents / Actions

Synchronisation Game

PG-NoPS

PG-FuPS

PG-FuPS+ID

Algorithm 1 HyperMARL

1: Input: Number of agents n , number of training iterations K , MARL algorithm parameters (e.g., MAPPO-specific hyperparameters)

2: Initialise:

3: Hypernetwork parameters ψ, φ {Ensure θ i and ϕ i follow standard initialization schemes, e.g., orthogonal}

4: Agent embeddings { e i } n i =1 {One-hot or orthogonally initialised learnable parameters}

5: Output: Optimized joint policy π

6: for each training iteration k = 0 , 1 , . . . , K -1 do

7: for each agent i = 1 , . . . , n do

8: θ i ← h π ψ ( e i ) {Policy parameters}

9: ϕ i ← h V φ ( e i ) {Critic parameters}

10: end for

11: Interact with environment using { π θ i } n i =1 to collect trajectories D

12: Compute shared loss L from D , using { V ϕ i } n i =1 {Standard RL loss function}

13: Update ψ , φ , and e by minimizing L {Optimise parameters.}

14: end for

15: Return π = ( π 1 , . . . , π n )

F HyperMARL Details

F.1 HyperMARL Pseudocode

In Algorithm 1, we present the pseudocode for HyperMARL, with HyperMARL-specific steps highlighted in blue. HyperMARL leverages hypernetworks to dynamically generate the parameters of both actor and critic networks. The weights of the hypernetworks and the agent embeddings are automatically updated through automatic differentiation (autograd) based on the computed loss. Additionally, Figure 1 provides a visual representation of the HyperMARL architecture.

F.2 Variance of the HyperMARL Gradient Estimator

Unbiased estimator. Following from Equation 4, we can write the unbiased estimator for HyperMARLs gradient as follows:

where B trajectories { τ ( b ) } B b =1 are sampled i.i.d. and ̂ Z i is the empirical analogue of the observationconditioned factor.

Assumptions. (A1) trajectories are i.i.d.; (A2) all second moments are finite; (A3) ψ, θ, e i are fixed during the backward pass.

Variance expansion. Since each J i is deterministic under (A3), we may factor them outside the covariance:

(11)

Equation (11) makes explicit that all trajectory-induced covariance is captured Cov( ̂ Z i , ̂ Z j ) , while the agent-conditioned Jacobians J i remain trajectory noise-free.

Mini-batch update and covariance. Let ̂ Z i be the unbiased mini-batch estimate of Z i and ̂ g HM = ∑ i J i ̂ Z i the stochastic update. Because every J i is deterministic (wrt. to mini-batch),

(derivation in Appendix F.2). Equation (12) shows that HyperMARL first averages noise within each agent ( ̂ Z i ) and only then applies J i . FuPS+ID, by contrast, updates the shared weights θ with every raw sample A ∇ θ log π θ [ h, id] , leaving observation noise and agent ID entangled and making it susceptible to gradient interference [11, 54].

F.3 Scalability and Parameter Efficiency

Hypernetworks generate weights for the target network, which can lead to high-dimensional outputs and many parameters for deep target networks. This challenge is amplified in MLP-based hypernetworks, which include additional hidden layers. Figure 14 shows scaling trends:

NoPS and linear hypernetworks : Parameter count grows linearly with the number of agents.

FuPS : More efficient, as growth depends on one-hot vector size.

MLP hypernetworks : Scale better with larger populations, since they only require embeddings of fixed size for each new agent.

To reduce parameter count, techniques like shared hypernetworks, chunked hypernetworks [47, 9], or producing low-rank weight approximations, can be used. While naive implementations are parameterintensive, this might be less critical in RL and MARL which commonly have smaller actor-critic networks. Moreover, HyperMARL's near-constant scaling with agents suggests strong potential for large-scale MARL applications.

FuPS + One Hot

Linear Hypernets+ One Hot

MLP Hypernets + Embed

NoPS

4

16

64

256

1024

Number of Agents

10K

100K

1M

10M

Number of Parameters

Scaling of IPPO on Dispersion

To isolate the effects of parameter count, we scaled the FuPS networks (Figure 15) to match the number of trainable parameters in HyperMARL. Despite generating 10x smaller networks, HyperMARL consistently outperforms FuPS variants, showing its advantages extend beyond parameter count.

0.0

0.3

0.6

IPPO-FuPS

MAPPO-FuPS

IPPO-MLP Hypernetwork

MAPPO-MLP Hypernetwork

IQM

0.0

0.3

0.6

Mean

Mean Episode Return

F.4 Speed and Memory Usage

We examine the computational efficiency of HyperMARL compared to NoPS and FuPS by measuring inference speed (Figure 16a) and GPU memory usage (Figure 16b) as we scale the number of agents. The benchmarks were conducted using JAX on a single NVIDIA GPU (T4) with a recurrent (GRUbased) policy architecture. All experiments used fixed network sizes (64-dimensional embeddings and hidden layers) with a batch size of 128 and 64 parallel environments, allowing us to isolate the effects of varying agent count. Each measurement represents the average of 100 forward passes per configuration, with operations repeated across 10 independent trials.

FuPS + One Hot

MLP Hypernets

Linear Hypernets

NoPS

4

16

64

256

512

Number of Agents

0

10

20

30

40

50

60

70

Forward Pass Time (ms)

Recurrent IPPO - Forward Pass Speed vs Number of Agents

(batch˙size=128)

(a) Forward pass inference time

0

100

200

300

400

500

Number of Agents

0

2000

4000

6000

8000

10000

12000

Recurrent IPPO - GPU Memory Usage vs Number of Agents

FuPS + One Hot

MLP Hypernet

Linear Hypernets

NoPS

(b) GPU memory usage

Figure 16: Computational efficiency scaling with number of agents. HyperMARL offers a balance between NoPS and FuPS. Notably, in real-world deployments, NoPS incurs additional data transfer and synchronisation costs not reflected here, further widening the efficiency gap.

The results demonstrate that HyperMARL offers a balance between the extremes represented by NoPS and FuPS. In practice, NoPS incurs additional data transfer and update costs, widening the efficiency gap.

F.5 Sensitivity Analysis

We run a sensitivity analysis on the HyperMARL's hyperparameters in the 20-agent SMAX scenario with recurrent MAPPO, a challenging setting for optimisation. The results are shown Table 5.

Findings.

Similarly to FuPS, and in general, deep RL methods like PPO, the learning rate is an important parameter for performance.

Agent embedding size is an important hyperparameter. In these results, a smaller embedding size outperformed a larger one. This could be due to the homogeneous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence similar behaviours. This suggests the embedding size could correspond to the task's diversity requirements.

Other parameters, such as the width, have limited impact beyond a modest size.

G Experiment Details

G.1 Environments

(a)

Dispersion

.

(b)

Navigation

.

(c)

MAMuJoCo

.

(d)

SMAX

.

L

L

L

AGENTS

(e)

BPS

.

Step 0

0E

20

15

10

5

10

15

20

25

30

Dispersion (VMAS) [5] : A 2D environment where four agents collect unique food particles. This task requires specialised heterogeneous behaviours and resembles the Specialisation Game from Section E.1.

Navigation (VMAS) [5] : Agents navigate in a 2D space to assigned goals, receiving dense rewards based on proximity. Configurations include shared goals ( homogeneous ), unique goals ( heterogeneous ), and mixed goals, where some agents share goals while others have unique ones.

Multi-Agent MuJoCo (MAMuJoCo) [37] : A multi-agent extension of MuJoCo, where robot body parts (e.g., a cheetah's legs) are modelled as different agents. Agents coordinate to perform efficient motion, requiring heterogeneous policies [54].

SMAX (JaxMARL) [41] : Discrete action tasks with 2 to 20 agents on SMACv1- and SMACv2-style maps. FuPS baselines have been shown optimal for these settings [53, 17] indicating homogeneous behaviour is preferred here.

Blind-Particle Spread (BPS) [11] Blind-Particle Spread (BPS) is a modified variant of the MultiAgent Particle Environment (MPE) Simple Spread task [30] with 15 to 30 agents. The environment contains landmarks of different colours, and each agent is assigned a colour. Agents are 'blind' as they cannot observe their own assigned colour or the colours of other agents. They must infer the correct landmark to navigate towards based only on their observations and rewards. The number of colours thus dictates the number of distinct roles or behaviours the team must learn to successfully complete the task.

G.2 HyperMARL Architecture Details

For the Dispersion and Navigation results (Sec. 5.2) we use feedforward architectures, where we use HyperMARL to generate both the actor and critic networks. For the MAPPO experiments in Section 5.2, for fairness in comparisons with HAPPO and MAPPO, we maintain the centralised critic conditioned on the global state and only use HyperMARL to generate the weights of the actors. For the recurrent IPPO experiments in Section 5.3, HyperMARL only generates the actor and critic feedforward weights, not the GRU weights.

G.2.1 Training and Evaluation

· Training:

-For Dispersion (5.2), we run 10 seeds and train for 20 million timesteps.

-For Navigation (5.2), SMAX (5.3), and MaMuJoCo (5.2), we run 5 seeds and train for 10 million timesteps, consistent with the baselines.

-For Blind-Particle Spread (BPS), we run 5 seeds and train for 20 million timesteps, consistent with baselines.

· Evaluation:

-For Dispersion (5.2), evaluation is performed every 100k timesteps across 32 episodes.

-For Navigation (5.2), following the baselines, evaluation is performed every 120k timesteps across 200 episodes.

-For SMAX (5.3), evaluation is performed every 500k timesteps across 32 episodes.

-For MaMuJoCo (5.2), following the baselines, evaluation is performed every 25 training episodes over 40 episodes.

G.2.2 Measuring Policy Diversity Details

We measure team diversity using the System Neural Diversity (SND) metric (Equation 5 [7], details Section D) with Jensen-Shannon distance. SND ranges from 0 (identical policies across all agents) to 1 (maximum diversity). We collect a dataset of observations from IPPO-NoPS and IPPO-FuPS policies checkpointed at 5 and 20 million training steps. Each policy is rolled out for 10,000 episodes, generating 16 million observations. We then sample 1 million observations from this dataset to calculate the SND for each method tested.

H Detailed Results

H.1 Comparison with Kaleidoscope using Off-Policy - MATD3

Our comparison with Kaleidoscope [28], mentioned in Section 5, is conducted using off-policy methods due to its original design. Kaleidoscope incorporates intricate mechanisms (e.g., learnable masks, an ensemble of five critics, a specific diversity loss) and numerous specialised hyperparameters (e.g., for critic ensembling: 'critic_deque_len', 'critic_div_coef', 'reset_interval'; for mask and threshold parameters: 'n_masks', 'threshold_init_scale', 'threshold_init_bias', 'weighted_masks', 'sparsity_layer_weights', etc.). Porting this full complexity to an on-policy PPO backbone would constitute a significant research deviation rather than a direct benchmark of the established method.

Therefore, we utilised Kaleidoscope's original off-policy implementation to ensure a meaningful comparison. We adopted MATD3 as the algorithmic backbone for this evaluation, as it was the only publicly available Kaleidoscope variant with tuned hyperparameters for Multi-Agent MuJoCo (MaMuJoCo). The MaMuJoCo tasks were chosen for alignment with our primary on-policy (IPPO) results and Kaleidoscope's original evaluation, specifically: Ant-v2, HalfCheetah-v2, Walker2d-v2 (overlapping with our IPPO experiments), and Swimmer-v2-10x2 ( which represents the MaMuJoCo variant with the highest number of agents). Comparative results in Table 7 show that HyperMARL achieves competitive results with Kaleidoscope, while only using two critics (standard MATD3) and without additional diversity objectives.

H.2 Comparison with Selective Parameter Sharing (SePS) - A2C

We benchmark HyperMARL against Selective Parameter Sharing (SePS) [11], a method that shares weights among pre-determined groups of similar agents. SePS identifies these groups by clustering agent trajectories via an autoencoder during a pre-training phase. Following the original authors' setup, we use their provided source code and hyperparameters, evaluating all A2C [32] variants on the Blind Particle Spread (BPS) benchmarks (v1-4), which scale up to 30 agents. For HyperMARL, we sweep the learning rates (1e-4, 3e-4) and hypernetwork hidden dimension (16, 64) per scenario.

As shown in Table 8, HyperMARL demonstrates competitive performance against both NoPS and SePS in these challenging settings, which feature up to 30 agents and five distinct agent roles. Crucially, HyperMARL achieves this result without resorting to agent-specific weights (like NoPS) or requiring a pre-training and clustering pipeline (like SePS).

H.3 Dispersion Detailed Results

H.3.1 Interval Estimates Dispersion

0.4 0.0

0.4

0.8

FuPS+ID

NoPS

IQM

0.4

0.0

0.4

0.8

Mean

Mean Episode Return

(a) IPPO

0.000.250.500.75

FuPS+ID

NoPS

IQM

0.00 0.25 0.50 0.75

Mean

Mean Episode Return

(b) MAPPO

0

10

20

30

40

Training Steps (M)

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

IQM Mean Episode Return

FuPS

MLP Hypernetwork IQM

H.4 Detailed MAMujoco Plots

MAPPO-FuPS

MAPPO (Independent Actors)

HyperMARL (MAPPO)

HAPPO

0

2

4

6

8

10

Training Steps (M)

0

1000

2000

3000

4000

5000

6000

7000

IQM Mean Episode Return

(a) Humanoid-v2 17x1

0

2

4

6

8

10

Training Steps (M)

0

1000

2000

3000

4000

5000

6000

IQM Mean Episode Return

(b) Walker2d-v2 2x3

0

2

4

6

8

10

Training Steps (M)

0

1000

2000

3000

4000

5000

6000

7000

IQM Mean Episode Return

(c) HalfCheetah-v2 2x3

0

2

4

6

8

10

Training Steps (M)

1000

2000

3000

4000

5000

6000

IQM Mean Episode Return

H.5 Detailed Navigation Plots

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(a) 8 Agents, Same Goal

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(b) 8 Agents, Different Goals

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(c) 8 Agents, Four Goals

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(d) 4 Agents, Same Goal

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(e) 4 Agents, Different Goals

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

IQM Mean Reward

(f) 4 Agents, Two Goals

0

2

4

6

8

10

Training Steps (M)

0.000

0.005

0.010

0.015

0.020

0.025

0.030

IQM Mean Reward

(g) 2 Agents, Same Goal

0

2

4

6

8

10

Training Steps (M)

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

0.030

IQM Mean Reward

(h) 2 Agents, Different Goals

FuPS

NoPS

DiCo

H.6 Interval Estimates - SMAX

H.7 Additional Ablations

0.960.991.021.05

IPPO RNN (FuPS)

MAPPO RNN (FuPS)

HyperMARL (IPPO RNN)

HyperMARL (MAPPO RNN)

IQM

0.960.991.021.05

Mean

Mean Episode Return

(a) 2s3z

0.720.800.880.96

IPPO RNN (FuPS)

MAPPO RNN (FuPS)

HyperMARL (IPPO RNN)

HyperMARL (MAPPO RNN)

IQM

0.80 0.88 0.96

Mean

Mean Episode Return

(b) 3s5z

0.15 0.30 0.45

IPPO RNN (FuPS)

MAPPO RNN (FuPS)

HyperMARL (IPPO RNN)

HyperMARL (MAPPO RNN)

IQM

0.15 0.30 0.45

Mean

Mean Episode Return

(c) SMACv2 10 Units

0.080.160.240.32

IPPO RNN (FuPS)

MAPPO RNN (FuPS)

HyperMARL (IPPO RNN)

HyperMARL (MAPPO RNN)

IQM

0.08 0.16 0.24

Mean

Mean Episode Return

(d) SMACv2 20 Units

0

5

10

15

20

Training Steps (M)

-0.75

-0.50

-0.25

0.00

0.25

0.50

0.75

IQM Mean Episode Return

FuPS

HyperMARL-S

HyperMARL w/ One-Hot

HyperMARL w/o RF

HyperMARL w/o GD

HyperMARL

I Hyperparameters

Note: HNET Embedding Dim refers to the hypernetwork embedding dimension, chosen from the range {4, 16, 64}. HNET Hidden Dims refers to the hidden layer dimensions of the hypernetwork, chosen from the range {16, 32, 64}.

J Computational Resources

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Sections 3, 5 and 6.1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Sections A and F.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please see Section E.3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please see Sections 4 and 1. We also provide all the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all the code and scripts to reproduce results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see Sections 5 and App. I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Following best practices, we report statistical significance in all results in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Section J.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Read and confirmed.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See App. B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors focus on the potential limitations in parameter sharing methods. Based on experimental evidence, it is elaborated that coupling agent IDs with observations can be detrimental, blamed by gradient interference among agents. To overcome the key difficulty with FuPS, an agent-conditioned hypernetworks based parameter sharing method is proposed, which explicitly decouples observation- and agent-conditioned gradients. By mitigating cross-agent interference, HyperMARL empirically reduces policy gradient variance, facilitates shared-policy adaptation, and learns heterogeneous, homogeneous, or mixed behaviors, as demonstrated in various challenging MARL benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written. The organization of this paper is clear, and the proposed idea is easy to follow. The limitations of parameter sharing are elaborated with clear definition of research problems. The experimental verification on the impact of the coupling of observations and agent IDs brings certain level of new opinion and knowledge. \n\n2. The experiment is comprehensive. The proposed method is tested in diverse benchmarks where heterogeneous, homogeneous, or mixed behaviors are required. The ablation and computation complexity are also considered. \n\n3. Even though simple, the methodology is effective. The theory behind has been carefully discussed with verified hypernetwork gradients. The key idea of decoupling is well introduced and proved.\n\nWeaknesses:\n\n1. Some discussion of related work seems to be missing. Part of this work seems similar to “Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing”. Maybe adding further discussion in the related work section could be helpful. \n\n2. In methodology section (Section 4), how the agent-specific parameters ($\\theta $) are integrated into original RL networks seems not to be clear. \n\n3. The argument in Section 3 seems unsatisfactory. There seems to be some small mistakes.\n\nQuality: 2.9 $\\approx $ 3 (From introduction to conclusion section, the discussion of parameter sharing is comprehensive.); \n\nClarity: 3.3 $\\approx $ 3 (Overall, this paper is well-written with clear definition of research problem. It is easy to follow the idea in this paper.); \n\nSignificance: 2.4 $\\approx $ 2 (Parameter sharing is a well-discussed topic. It seems that part of this work is considered by previous work?); \n\nOriginality: 2.4 $\\approx $ 2 (I am more likely to view this paper as an extension of previous parameter sharing work?); \n\nRating: 4.5 $\\approx $ 5 (Overall, this is an interesting paper. If authors can address my concerns, I will confirm the score to 5.);\n\nConfidence: 3.6 $\\approx $ 4 (The paper is good and easy to follow. But I am not quite sure about the key difference between it and previous work.).\n\nQUESTIONS:\n1. Is it possible that the specialization of the policy may lead to redundant diversification policies? For example, in a neural network, a function can be fitted with two nodes, or it can be fitted with MLP, but the latter may cause serious overfitting problems. Will the strengthening of specialization lead to a decrease in adaptability? How to know the appropriate level of specialization is achieved?\n\n2. (Christianos, Filippos, et al. \"Scaling multi-agent reinforcement learning with selective parameter sharing.\" International Conference on Machine Learning. PMLR, 2021). The above paper seems to have similar ideas, and their encoder-decoder can also be considered as a type of hypernetworks. What are the similarities and differences between this work and previous work? Furthermore, in the previous work, the encoder-decoder was trained before RL, while in this paper, it is trained simultaneously. During simultaneous training, will there be conflicts between RL learning and policy specialization learning? What are the advantages of simultaneous training? Based on the decoupling idea mentioned in this article, it seems that it would be better to decouple these two optimization goals?\n\n3. In Figure 3 (page 4), why ‘FuPS+ID’ method fails even in synchronization environment? According to the explanation in subsection 3.2, it should not fail in this environment? In Figure 4, what is ‘PG-FuPS+ID’ (with no definition)? Is ‘PG-FuPS+ID’ equal to ‘FuPS+ID’? In line 105 (page 4), ‘Table 3’ (mistake?)? In line 108 to 109, why authors claim that the fully policy sharing method also has scalability issue in synchronization environment? \n\n4. In subsection 3.2 (line 117 ‘Importance of Observation and ID Decoupling’), authors show the result of ‘FuPS+ID’ and ‘FuPS+ID (No State)’. What about ‘FuPS’? Is it an orthogonal gradient or a more conflicting gradient? If it is orthogonal gradient: why is it not performing well? If not: is it possible that adding ID could reduce the conflicting gradients, that is, ID is actually beneficial? Based on the existing results, it seems that we cannot easily draw a conclusion about \"Importance of Observation and ID Decoupling\"?\n\n5. In section 4, how are the agent-specific parameters ($\\theta $) integrated into original RL networks? Are all the parameters of the policy network ($\\pi $) generated by the hypernetworks? Or is it just the last layer that is replaced, and the other layers are still RNN? Why use ID as input to the hypernetworks? Can the inputs to the hypernetworks and policy network be swapped (i.e. use observation as input to the hypernetworks, since the purpose here seems to be just for decoupling)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes HyperMARL, which decouples the observation-conditioned and agent ID-conditioned gradients via hypernetworks to enable better coordination for both specialist and generalist behaviors. The authors first show that full parameter sharing leads to conflicting updating gradients of agents. Then they propose HyperMARL that uses agent ID-conditioned hypernetworks to generate weights and biases for agent policy and critic networks. Experiments on Dispersion, Navigation, MAMuJoCo, and SMAX validate the proposed HyperMARL when compared with various baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tThe finding of conflicting gradients among agents by full parameter sharing and agent ID is important.\n2.\tThe method is well-motivated and reasonable. Through the hypernetwork, the gradient flow is explicitly decoupled as shown in Equation (4).\n3.\tThis paper is well-written and well-organized.\n\nWeaknesses\n\n1.\tUsing agent ID or embedding only to generate weights and biases of agent policy and critic networks may limit the representation ability in complex tasks.\n2.\tThe widely used SMAC and SMACv2 environments are not tested. For the SMAX, its dynamic seems to be simplified and ease the learning of hypernetworks for policy and critic weights.\n3.\tSome role-based MARL works, such as RODE, are not compared in the experiments.\n\nQUESTIONS:\n1.\tCould the authors compare hypernetworks with a dual-channel network, where one channel is for agent observation and one channel is for agent ID?\n2.\tCould HyperMARL work well in complicated tasks such as SMACv2 where the observation provides key decision information? May hypernetworks limit the learning ability of the agent network?\n3.\tDoes HyperMARL support value-factorization MARL algorithms such as QMIX and QPLEX?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a central challenge in cooperative multi-agent reinforcement learning (MARL): enabling policies to adaptively learn specialized, homogeneous, or mixed behaviors without sacrificing sample efficiency. The authors identify an issue in standard parameter-sharing methods (like FuPS+ID), arguing that the coupling of agent observations and agent IDs within a single network exacerbates cross-agent gradient interference and hinders specialization.\n\nTo solve this, they propose HyperMARL, an architecture that uses an agent-conditioned hypernetwork to generate agent-specific parameters for the policy and critic networks. This design decouples the observation-conditioned gradients from the agent-conditioned ones. The core benefit, as shown theoretically and empirically, is a reduction in policy gradient variance, which mitigates gradient conflicts and facilitates adaptive learning. The authors conduct experiments across a diverse set of MARL benchmarks (VMAS, MAMuJoCo, SMAX), demonstrating that HyperMARL achieves competitive or superior performance compared to baselines, including No-Parameter-Sharing (NoPS), Full-Parameter-Sharing (FuPS), and state-of-the-art diversity-promoting methods like DiCo, HAPPO, and Kaleidoscope, while maintaining high behavioral diversity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- S1: Novel Insight: The paper provides a clear and compelling diagnosis of a problem in parameter-shared MARL. The empirical study in Section 3 and Figure 4, which isolates the detrimental effect of coupling observation and agent-ID inputs, is a contribution in itself. This insight provides motivation for the proposed architectural design.\n- S2: Clarity and Presentation: The paper is well-written and easy to follow. Figure 1 provides an visual summary of the core idea. The logical flow, from problem identification to solution and validation, is clear. The appendix is detailed and provides sufficient information.\n\nWeaknesses\n- W1: Scalability in Terms of Parameter Count: The primary weakness of the HyperMARL approach, particularly the MLP variant, is the significant parameter overhead. The hypernetwork must generate the entire weight set for each agent's policy (and critic) network. While the paper commendably analyzes scalability with the number of agents (Appendix F.3) and shows near-constant scaling for MLP hypernetworks, it doesn't fully address the scalability with respect to the size of the target network. For very deep or complex policy architectures (e.g., those based on Transformers), the output dimension of the hypernetwork could become prohibitively large, making the approach computationally expensive and difficult to train. The suggested remedies (chunking, low-rank) are mentioned but not empirically explored.\n- W2: Complexity of Hypernetwork Training and Initialization: The stability and performance of HyperMARL likely depend on careful initialization and training of the hypernetwork itself. The paper mentions a principled initialization scheme (Section 4.2) to match standard network initializations, which is a good practice. However, training a network to output the weights of another network is inherently a more complex and potentially less stable optimization problem than standard policy training. The paper could benefit from a more detailed discussion or analysis of the training dynamics and sensitivity to the hypernetwork's own hyperparameters (e.g., its depth, width, learning rate).\n\nQUESTIONS:\nThe following questions aim to clarify some key aspects of the method and its implications. I would be willing to raise my score if the authors can provide satisfactory answers.\n\n**Q1:** The Nature of Learned Agent Embeddings: For the MLP Hypernetwork, the paper uses learned agent embeddings. Could the authors provide some analysis of these learned embeddings? For instance, do agents that are required to learn similar (or complementary) policies end up with closer embeddings in the latent space? Visualizing the embedding space (e.g., using t-SNE) could provide valuable insights into whether the hypernetwork is learning a meaningful representation of inter-agent relationships.\n\n**Q2:** Disentangling the Effects of Gradient Decoupling vs. Parameter Specialization: HyperMARL introduces two concurrent benefits: (1) it decouples observation- and agent-conditioned gradients, and (2) it provides each agent with its own specialized set of parameters. While these two are intertwined in the proposed architecture, could the authors speculate on which factor is more critical? For example, would an architecture that provides specialized parameters but without explicit gradient decoupling (if one could be designed) perform as well? This question aims to pinpoint the core ingredient of HyperMARL's success.\n\n**Q3:** Performance in Mixed Cooperative-Competitive Settings: The experiments focus on fully cooperative MARL. How would HyperMARL be expected to perform in mixed-motive or general-sum settings where agents might have conflicting objectives? In such scenarios, the concept of \"cross-agent gradient interference\" becomes more complex, as some interference might be adversarial rather than simply a result of differing specialization needs. Does the gradient decoupling mechanism remain as beneficial?\n\n**Q4:** Comparison with Role-Based Approaches: How does HyperMARL compare conceptually to explicit role-based learning methods (e.g., ROMA)? Both aim to induce specialization. It seems HyperMARL learns these roles implicitly through the agent embeddings and generated weights, potentially offering more flexibility. A brief discussion contrasting these two paradigms would strengthen the paper's positioning in the literature.\n\nLIMITATIONS:\nThe authors have commendably included a limitations section in the appendix. To build upon that, the following points can be highlighted:\n\n**L1:**  Assumption of Fixed Agent Population: The current framework assumes a fixed number of agents n throughout training, as the hypernetwork architecture (especially the linear variant) and agent embeddings are tied to this number. The model may not generalize to scenarios with a varying or unknown number of agents without retraining or specific architectural modifications (e.g., attention-based hypernetworks).\n\n**L2:**  Increased Computational Cost at Inference: Generating network weights on the fly with a hypernetwork introduces an additional computational step at inference time compared to a standard FuPS model. While faster than a NoPS approach (which may involve significant data transfer overhead in a real distributed system), this added latency could be a concern for time-critical applications. The analysis in Appendix F.4 is good, but this trade-off deserves to be highlighted as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper inroduces HyperMARL, a novel method for multi-agent reinforcement learning that uses agent-conditioned hypernetworks to generate per-agent parameters on the fly. The goal is to enable diverse (specialised), homogeneous, or mixed behaviours without modifying the standard RL objective or requiring manually preset diversity levels. The key idea is to decouple observation-conditioned and agent-conditioned gradients, mitigating cross-agent gradient interference that plagues standard parameter-sharing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Interesting method for enhancing agent diversity while preserving performance in homogeneous tasks**: HyperMARL achieves NoPS-level policy diversity while maintaining the efficiency of parameter sharing, which is well demonstrated across diverse benchmarks (Dispersion, Navigation, MAMuJoCo, SMAX).\n- **Effectively tackles gradient conflicts**: The paper convincingly shows how decoupling observation and agent-ID gradients reduces policy gradient variance and cross-agent interference. Ablations confirm this is essential for performance.\n- **Clear and easy to follow**: The research questions are well posed (\"Can shared policies adapt without added complexity?\"), the theoretical motivation is thorough (gradient conflict analysis), and the method is described with clear architectural diagrams and derivations. The empirical section answers the research questions systematically.\n\n## Weaknesses\n- **Missing computational comparison (e.g. training time)**: While parameter efficiency is discussed (Appendix A mentions scaling), there is no direct measurement or table comparing training times or wall-clock convergence with standard parameter-sharing or NoPS baselines. Including such data would strengthen the empirical evaluation.\n- **Lack of comparison with state-of-the-art heterogeneous baselines in complex robotic settings**: For example, the paper does not compare HyperMARL with HetGPPO or other recent methods explicitly tailored for large-scale heterogeneous agents in robotics. Even though HyperMARL performs well on MAMuJoCo (17-agent Humanoid), comparing with specialized heterogeneous-agent methods would clarify its relative advantages or limitations.\n- **Sensitivity to hyperparameters**: Hypernetworks are known to be sensitive to hyperparameter choices (e.g. embedding size, MLP depth). The paper briefly notes that hyperparameter sweeps for some baselines (e.g. DiCo) were challenging but does not analyze how much HyperMARL itself depends on careful tuning. A section on its hyperparameter robustness would be valuable.\n\nQUESTIONS:\n## Questions\n1. How does HyperMARL's training time compare (e.g. wall-clock or steps to convergence) to FuPS, NoPS, and DiCo in the largest tested tasks?\n2. How sensitive is HyperMARL to hypernetwork design choices (e.g. embedding dimension, number of MLP layers)?\n3. Can the authors include results comparing with recent heterogeneous-agent methods such as HetGPPO on the same tasks?\n\nLIMITATIONS:\n## Limitations\n- **Baseline selection**: While strong baselines like DiCo, HAPPO, and Kaleidoscope are included, some very recent or robotics-focused heterogeneous MARL baselines (e.g. HetGPPO) are missing from the comparison.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses parameter sharing in cooperative multi-agent reinforcement learning (MARL). The authors propose HyperMARL, which uses hypernetworks to generate agent-specific parameters by decoupling observation- and agent-conditioned gradients. The method is evaluated across different scenarios including homogeneous, heterogeneous, and mixed behavior tasks with 2-20 agents, showing competitive performance against fully-shared and non-shared approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper addresses an important challenge in MARL regarding parameter sharing and policy specialization.\n2. The proposed decoupling of observation- and agent-conditioned gradients through hypernetworks is conceptually novel and well-motivated.\n3. The writing is clear and accessible. \n\n**Weaknesses**\n1. My main concern is with the evaluation：\n    While this work critiques previous efforts for adding complexity through altered objectives, manual preset diversity levels, or sequential updates, the proposed HyperMARL also introduces significantly heavier training complexity (memory-wise). The trade-off between performance gains and computational costs is not clearly justified.\n    Specifically, such trade-off should be made clearer by:\n    - Providing comprehensive comparisons of the proposed HyperMARL against current baselines on standard benchmarks with results reported consistently in the main text. Currently, the evaluation appears fragmented - HyperMARL is compared against DiCo on one benchmark (MPE) while comparison with Kaleidoscope is conducted on a different benchmark (MaMuJoCo) with results relegated to the appendix. This inconsistent evaluation approach makes it difficult to assess the method's overall effectiveness.\n    - Moreover, several important parameter sharing methods such as [1-3] are entirely omitted from the evaluation, despite addressing the exact same research question of parameter sharing in MARL. These methods would serve as highly relevant baselines for a thorough evaluation.\n    - Including a thorough comparison of computational costs (parameters, training time, memory usage) across all methods to justify the additional complexity of HyperMARL.\n2. The related work section could be enhanced.\n   - Recent efforts on parameter sharing appear to be overlooked [3].\n   - The discussion of hypernetworks should be expanded beyond reinforcement learning to position this work within the broader ML literature, where hypernetworks have been applied to areas such as federated learning, continual learning, and multi-task learning [4-5].\n\n----\n\n[1] \"Scaling multi-agent reinforcement learning with selective parameter sharing\", ICML 2021 \\\n[2] \"Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning\", AAMAS 2023 \\\n[3] \"GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning\", ICML 2025 \\\n[1] \"A brief review of hypernetworks in deep learning\", Artificial Intelligence Review. \\\n[2] \"Personalized federated learning using hypernetworks\" ICML 2021.\n\nQUESTIONS:\n1. Could the authors provide comprehensive comparisons with current parameter sharing methods on standard benchmarks to better demonstrate HyperMARL's advantages? These methods directly address parameter sharing in MARL and would provide a more complete evaluation context.\n2. Could the authors expand the related work section to include a broader discussion of hypernetwork applications across machine learning domains beyond reinforcement learning?\n3. Could the authors provide the training costs for HyperMARL and baseline methods (trainable parameters count / training time / training memory)? (As HyperMARL introduces additional trainable parameters and training costs compared to the FuPS baseline, although this may be acceptable.)\n4. In Figure 8, HyperMARL performs comparably to FuPS while incurring higher computational costs. What specific advantages does HyperMARL offer over FuPS in homogeneous tasks that would justify its adoption despite these added costs?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-04"}