Title: Heterogeneity in Multi-Agent Reinforcement Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: multi-agent reinforcement learning

heterogeneity

policy diversity
Abstract: *Heterogeneity* is a fundamental property in multi-agent reinforcement learning (MARL), which is closely related not only to the functional differences of agents, but also to policy diversity and environmental interactions. However, the MARL field currently lacks a rigorous definition and deeper understanding of heterogeneity. This paper systematically discusses heterogeneity in MARL from the perspectives of *definition*, *quantification*, and *utilization*.
First, based on an agent-level modeling of MARL, we categorize heterogeneity into five types and provide mathematical definitions.
Second, we define the concept of heterogeneity distance and propose a practical quantification method.
Third, we design a heterogeneity-based multi-agent dynamic parameter sharing algorithm as an example of the application of our methodology.
Case studies demonstrate that our method can effectively identify and quantify various types of agent heterogeneity. Experimental results show that the proposed algorithm, compared to other parameter sharing baselines, has better interpretability and stronger adaptability.
The proposed methodology will help the MARL community gain a more comprehensive and profound understanding of heterogeneity, and further promote the development of practical algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a systematic framework for understanding and leveraging heterogeneity in MARL. The authors first categorize heterogeneity into five types—observation, response transition, effect transition, objective, and policy—based on a formal agent-level modeling under the Partially Observable Markov Game framework. Each type is given a mathematical definition. To quantify heterogeneity, the paper introduces the notion of heterogeneity distance, which measures functional differences between agents using representation learning techniques such as Conditional Variational Autoencoders, applicable in both model-based and model-free settings. Additionally, a meta-transition abstraction is also proposed to capture overall agent-level heterogeneity. Building on these foundations, the authors develop HetDPS, a dynamic parameter sharing algorithm that clusters agents based on heterogeneity distances and adjusts sharing schemes during training. This improves policy diversity while maintaining sample efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a well-structured and technically sound framework for defining, quantifying, and utilizing heterogeneity in MARL. The theoretical modeling based on the POMG framework is rigorous and covers multiple aspects of heterogeneity. The introduction of heterogeneity distance and its implementation via CVAE-based representation learning is technically well-founded. The experimental evaluation is comprehensive, including both synthetic and benchmark scenarios. The application to dynamic parameter sharing (HetDPS) further demonstrates practical utility. However, the scalability of the heterogeneity distance computation (quadratic in agent number) may limit its applicability to large-scale systems. Moreover, some methodological choices (e.g., reliance on latent space alignment) could benefit from deeper comparative baselines or ablation studies.\n\nThe paper is generally clear and well-organized. Definitions are formally presented, the taxonomy is intuitively motivated, and the figures (e.g., heterogeneity matrices and case studies) are informative. That said, certain sections (e.g., Section 4’s explanation of representation learning-based measurement) may be dense for readers not familiar with CVAE or distribution distance metrics. A clearer separation between theoretical definitions and their practical computation would improve readability.\n\nThe work addresses a foundational gap in the MARL literature: the lack of a rigorous, unified treatment of agent heterogeneity. The proposed framework has potential impact in both theoretical studies (e.g., policy analysis, agent grouping) and practical applications (e.g., efficient training via adaptive parameter sharing). The significance is further enhanced by its compatibility with both model-based and model-free settings. While the primary application (HetDPS) is within parameter sharing, broader implications for general-purpose heterogeneous MARL algorithms could be explored more explicitly.\n\nThe taxonomy of heterogeneity types and the formal notion of heterogeneity distance represent a novel contribution. Prior work has either focused on specific types (e.g., policy diversity) or has lacked rigorous definitions. The combination of formal theory with representation learning for quantification is both original and insightful. The meta-transition concept is also novel and provides a useful abstraction. While the application to parameter sharing is incremental, the underlying methodological contributions are notably innovative.\n\nQUESTIONS:\n1. The proposed method for computing heterogeneity distance incurs a quadratic computational cost with respect to the number of agents, which could limit its applicability in large-scale MARL scenarios. While the paper mentions partial remedies such as parallel computation and agent subset selection, these aspects are not discussed in depth in the main text. \n\nIt would be helpful if the authors could provide a more detailed empirical or theoretical analysis of the computational overhead associated with their method, especially under varying agent counts. Exploring approximate or hierarchical alternatives (e.g., sparse distance updates, local sampling) would enhance the method's practicality. \n\n2. The proposed heterogeneity quantification relies heavily on representation learning via CVAE, but the paper does not provide an analysis of the training stability, sensitivity to hyperparameters, or generalization across tasks.\n\nI think authors need to include an evaluation of CVAE stability, such as training convergence plots or reconstruction quality metrics. In addition, it would be valuable to assess whether learned heterogeneity representations remain meaningful under task shifts or in transfer learning settings.\n\n3. While the paper references related works (e.g., MADPS, AdaPS), the delineation of methodological contributions, especially in relation to prior studies on policy diversity and heterogeneous grouping, could be more explicit. In detail, the paper references several existing methods, such as MADPS (which performs dynamic grouping based on policy distance) and AdaPS (which relies on response transition heterogeneity), but it does not clearly articulate in which aspects the proposed approach is superior or fundamentally different. In particular, the area of policy diversity has been extensively explored, with many prior works addressing behavioral differences among agents. Therefore, the authors need to clarify what is novel about their formulation of heterogeneity distance compared to existing studies in this domain.\n\nI think it is necessary to establish a more structured comparison—e.g., a table summarizing key distinctions in terms of heterogeneity types addressed, theoretical definitions, and application scopes—would clarify the novelty of the proposed approach. Furthermore, incorporating controlled ablation experiments where heterogeneity quantification is removed or substituted could help isolate its contribution.\n\n4. The current empirical validation focuses primarily on particle-based spreading and selected SMAC tasks. It remains unclear how well the proposed method generalizes across different MARL domains or tasks with diverse forms of heterogeneity.\n\nI think it would be better to evaluate the method on additional domains (e.g., mixed cooperative-competitive environments or hierarchical tasks) would help demonstrate its broader applicability. Alternatively, providing a discussion of the method’s limitations in such settings would also be valuable.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive study of heterogeneity in multi-agent reinforcement learning, including its definition, quantification, and utilization. The authors first systematically categorizes heterogeneity into five types: observation, response transition, effect transition, objective, and policy heterogeneity. Further, it introduces a heterogeneity distance and a representation learning-based method to quantify heterogeneity in both model-free and model-based settings, along with the concept of meta-transition heterogeneity to capture comprehensive agent differences. Building on this quantification, a multi-agent dynamic parameter-sharing algorithm is proposed, offering improved interpretability and reduced reliance on task-specific hyperparameters compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors conduct an in-depth study of heterogeneity in multi-agent reinforcement learning. They novelly categorize heterogeneity into five types: observation, response transition, effect transition, objective, and policy heterogeneity. This is a departure from the pure policy heterogeneity definition in previous works. These five types comprehensively summarize the dimensions of heterogeneity in multi-agent reinforcement learning.\n\n2. The authors propose a standard objective to measure heterogeneity distance and introduce two practical solutions to compute it efficiently. First, standardizing the original distributions maps the intractable distributions to measurable distributions. Second, sampling based on agent-environment interactions builds a trajectory sample pool to reduce computational cost.\n\n3. Based on the heterogeneity distance, the authors further propose a Heterogeneity-based Multi-Agent Dynamic Parameter Sharing algorithm, which groups agents into different clusters and periodically quantifies heterogeneity.\n\nWeaknesses:\n\n1. While the authors introduce a heterogeneity distance objective, there are many possible distance metrics that could be used. Comparing multiple distance metrics could improve the generality and robustness of the proposed method.\n\n2. The concept of heterogeneity distance is not entirely novel and has been discussed in prior works [1,2]. However, the authors do not cite these in the paper. The techniques of standardizing original distributions and dynamic parameter sharing bear strong similarity to [1], and the differences should be clearly discussed.\n\n3. The proposed method does not appear to demonstrate significant performance improvement over baseline methods on the SMAC tasks.\n\n4. The operation of standardizing original distributions may be computationally expensive. In general, Wasserstein distance can be computed more efficiently using Fenchel-Rockafellar duality with trajectory sampling.\n\n[1] Hu, T., Pu, Z., Ai, X., Qiu, T., & Yi, J. (2024). Measuring Policy Distance for Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2401.11257.\n\n[2] Bettini, M., Kortvelesy, R., & Prorok, A. (2024, July). Controlling Behavioral Diversity in Multi-Agent Reinforcement Learning. In International Conference on Machine Learning (pp. 3611-3636). PMLR.\n\nQUESTIONS:\n1. What are the differences between the proposed heterogeneity distance in the paper and previous methods?\n\n2. Can the heterogeneity between agents lead to performance improvement?\n\n3. Can other distance metrics affect the performance of the proposed method?\n\n4. Why are some state-of-the-art MARL methods, such as QMIX (with full parameter sharing), not compared in the experimental section?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the idea of heterogeneity in multi-agent reinforcement learning.\n\nThey break this down into five types of heterogeneity: Observation heterogeneity, Response transition heterogeneity, effect transition heterogeneity, objective heterogeneity and policy heterogeneity. They then define a measure for quantifying the distances for heterogeneity, specifically in equation (3). To compute the heterogeneity distances they learn a representation mapping using a conditional variational autoencoder framework. They then use this encoders output for each sample gathered by the agents to calculate the heterogeneity distances for the various types.\n\nThey then have a case study which is multi-agent spread. A scenario in which agents are to move close to randomly generated landmarks which match their colour assignment. They do this with 6 extended versions to show the quantitative results of the different types of heterogeneity.\n\nUltimately this case study shows that fully parameter sharing has low policy-heterogeneity (and lower reward) and no parameter sharing has high policy-heterogeneity (higher reward). This reveals policy evolution differences.\nThe Meta-heterogeneity stays consistent, which reveals environmental characteristics.\n\nThey then develop Heterogeneity-based multi-agent Dynamic Parameter Sharing (HetDPS). This method adapatively groups agents for parameter sharing as the policy learns based on the heterogeneity types. They show its results in Particle based Multi-agent Spreading, as well as in the StarCraft Multi-Agent Challenge (SMAC). They compare to multiple baselines outlined in table 3. Showing that they perform well in 4 of the SMAC environments compared to the other methods, with comparable results to full parameter sharing in the environments that have homogenous agents.\n\nOverall, I believe this is promising work, but needs work to help with clarity of the final contributions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tOutlines and defines the heterogeneity types well.\n\n2.\tOutlines a method to get the heterogeneity distances. It is shown to work in the case study.\n\n3.\tShows the method works on multiple domains and compares to multiple other methods. With it being better in quite a number of these domains.\n\nWeaknesses:\n\n1.\tThe paper is quite difficult to follow with most of the relevant information being in the appendix. Especially for HetDPS.\n\n2.\tThe results are difficult to interpret, and could definitely use some more explanation. I’m not entirely sure what the Meta-Het Matrix is showing in Figure 6 for example, I assume it’s the different units from HetDPS’s measurement? If so, can it be made clear which unit type it is, so the heterogeneity is more obvious.\n\n3.\tSome of the graphs, specifically Figures 8 and 9, aren’t very clear as to what is to be interpreted.\n\nQUESTIONS:\n1. Will the methods described to find heterogeneity of the different types help to identify lazy agents and help them be reassigned?\n2. Do you think these methods to find heterogeneity can help agents match conventions in a setting strict coordination via one of multiple symmetrical but different conventions is required? (for example, different countries have different rules for driving on different sides of the roads.)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper focuses on different aspects of MARL related to homogeneity, by defining this notion for the multiple components of MARL, by proposing a way to measure such homogeneity and, finally, by leveraging this measure for parameter sharing. The performance of the proposed approach is evaluated both in a particle-based environment and in SMAC.

STRENGTHS AND WEAKNESSES:
Strengths:
- The proposed approach is motivated in a formal way
- The figures are well-designed and informative

Weaknesses:
- The definitions in Section 3 are highly redundant and straightforward (except perhaps for Definition 3), this space could have been used for more insightful aspects of the work
- Proposition 1 is very similar to existing results in the literature, see e.g. Theorem 3.3 in [R1], which shows the same result for Wasserstein distances (which is also the distance used in the paper)
- The proposed method should be introduced in more details

[R1] Lin, Z., & Ruszczyński, A. (2023). An integrated transportation distance between kernels and approximate dynamic risk evaluation in Markov systems. SIAM Journal on Control and Optimization, 61(6), 3559-3583.

QUESTIONS:
- Could the authors be more specific on how the measure of heterogeneity is used in parameter sharing?
- HetDPS is claimed to achieve "optimal or comparable results", but that does not seem to be the case in 15a_3c (Figure 5), where HetDPS seems to alternate between optimal and sub-optimal behaviour between seeds. In what sense is this behaviour optimal?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03