Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper looks at the intersection of representation learning and MARL. The authors aim to design network architectures that are permutation invariant to reordering of agents in the input representation layer of a MARL network and also consider how to develop some output units that are only equivariant to permutations. The authors propose two architectures towards this end (DPN and HPN) with the idea of keeping the backbone intact, allowing for reuse across a variety of different MARL algorithms. The authors demonstrate significant gains in sample efficiency using HPN and DPN across popular MARL environments and algorithms including on SMAC, MPE, and GRF. 
strength_and_weaknesses: Strengths: 
- This paper is well aligned with the conference by demonstrating the value of improved representation learning in complex MARL domains. 
- The PI and PE inductive biases are indeed very common characteristics of MARL problems, making correctly encoding them a fundamental research direction. 
- The paper is clearly written and the proposed approach is clearly justified theoretically.
- The experiments are very well done. The domains considered are large-scale and commonly used, relevant baselines are considered, important ablations are considered, and the approach is shown to be largely agnostic of the choice of MARL backbone. 

Weaknesses: 
- The DPN and HPN architectures are not tremendously novel and utilize well known concepts from the representation learning literature.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written in my opinion. The source code is provided. The experiments are very high quality. The novelty is not tremendously high, but I think it is a high impact use case. 
summary_of_the_review: As a reviewer, I found this paper to be a delight. It is clearly written, clearly motivated, and offers a focused yet potentially high impact solution. The experiments are also of extremely high caliber, really providing me with exactly what I would look for as a reviewer.  Ultimately, my consideration was more between strong accept and accept than it was considering that this paper should not be accepted. I believe it will make a nice contribution to the conference and believe it is likely to have a high impact on this subject area. My only concern, which is why I went with accept, is that the novelty of the proposed approach is really not that high in light of past work considering these topics in representation learning. However, I definitely think that this paper makes a strong contribution to the MARL community. 

Update After Author Feedback: 

I agree with the characterization provided by the authors of the source of novelty in their contribution and believe it is a nice addition to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies permutation invariance (PI) and permutation equivariance (PE) for multi-agent reinforcement learning. Different from existing works which achieve PI and PE via shared embedding of graph nets, the authors proposed a hypernet-based architecture, which is referred to as Hyper Policy Network (HPN). With HPN, the authors built PI input and PE output modules which could be plugged into existing MARL architectures without modifying the backbone network. The proposed approach is evaluated on SMAC, Google Research Football and MPE. The results show that the proposed approach outperforms baseline in various tasks. 

strength_and_weaknesses: ### Strength: 

1. While PI and PE for RL and MARL are widely studied, the idea of using hyper-net to output weight for different input and thus achieve PI and PE are new and interesting. 

2. The reviewer appreciates that the proposed approach could be applied to existing architecture without changing the backbone. This is an advantage over existing graph-based PE and PI MARL approaches. 

3. It is impressive that the proposed approach outperforms baselines on many super-hard SMAC tasks. 

4. The paper is well-written and easy to follow.    


### Weakness / Questions:

1. The reviewer has concerns on the assumption that the structural information of observations and the actions’ PI / PE property are available.  This is privileged information that is not used by baselines such as QMIX. This information may not be available in many real-world applications. For instance, suppose the observations are images or some feature vectors that are not interpretable. How do we apply the proposed approach? 

2. The reviewer has some concern regarding the scalability. Specifically, suppose there are 100 agents, does the hyper-net need to generate 100 different weights? 

3. While the paper highlights ‘break the curse of dimensionality’ and scalability as main contributions, the experiments only report results with few agents. For instance, in MPE, only six agents are considered. In contrast, baseline approaches such as PIC could actually scale to hundreds of agents. Results with more agents are needed to support the claims. 

clarity,_quality,_novelty_and_reproducibility: ### Clarity:
The paper is well-written and easy to follow. 

### Quality:
Overall, the paper is technically sound, but empirical results on settings with more agents are needed to support the claims about scalability. 

### Novelty:
Using hyper-nets to achieve PI and PE is novel and interesting.  

### Reproducibility:
The code is provided and the hyper-parameters are detailed in the appendix. The reviewer thinks there is a high chance one could reproduce the results. 


summary_of_the_review: In summary, the reviewer appreciates the proposed HPN, which nicely leverages the structural information of observation and builds the helpful inductive bias to achieve better performance. However, the reviewer thinks the limitations, such as assumption of the structural information of observation, should be more carefully discussed. In addition, empirical results on tasks with more agents are needed to support the claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on breaking the curse of dimensionality in multi-agent state space by exploiting the permutation invariance and permutation equivariance inductive biases. To achieve this, they proposed two implementations: Dynamic Permutation Network and Hyper Policy Network. A key property of this framework is that they only need to modify the input/output layer, leaving the backbone module unchanged. This property makes the work easy to add to other works. Finally, extensive experiments in SMAC, Google Football and MPE show some strength of this framework.
strength_and_weaknesses: **Strengths**

1. The problem this paper considers is important. How to handle the exponentially increasing dimensionality of state space in MARL is critical and this paper provides a new method to solve it. 
2. The literature review is sufficient. Most previous works have been discussed and compared.
3. This framework is well-motivated. To solve the curse of dimensionality in state space, using the properties of permutation invariance and equivariance in MARL might be a natural way. Though previous works have discussed these properties, this work has an advantage over others that it only needs to modify the input and output layer.

**Weaknesses**

1. The proposed method is not sound. (1) As discussed in Sec.2, Deep Set & GNN have limited representation capacity, due to the use of shared embedding. However, the proposed framework also uses a shared embedding $h_i$. In Sec.4.3, the hyper net $\text{hp}_{\text{in}}$ and module B are all shared. (2) In Sec.4.2, the observation of an entity chooses a weight matrix via probability. There is no guarantee that the entity can always choose the same weight matrix because the probability will change when this entity is in a different state. (3) HyperNet is also shared. What is the difference between $h(o_i,\theta_h)^\top\cdot o_i$ and $f(o_i,\theta_f)$?
2. The empirical evaluation results can be improved. This work uses a complex method to match the observation and action belonging to the same entity. What if we just sort the entities according to distance from the focal agent? This might be a strong baseline. 
3. The results are not significant enough compared with baselines considering PI and PE properties, such as UPDeT-QMIX, in Figure 6. 
4. This paper does not provide any discussions of limitations. (1) Is there any observation space or action space that cannot be factored? (2) Do all actions in MARL belong to PI or PE? What if there is an action that can tie *two* enemies together with a rope? The entity-correlated action space might not be a general assumption. 
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The proposed framework provides a novel and pluggable way to exploit the properties of PI and PE to reduce the dimensionality of state space in MARL.
summary_of_the_review: This paper is interesting and the proposed method might be an easy way to be utilized to any other methods in the considered settings. However, the soundness and the empirical evaluations should be further verified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the curse of the dimensionality problem in multiagent reinforcement learning (MARL), where the state-action space grows exponentially as the number of agents increases. To address this challenge, this paper proposes two novel implementations that exploit the permutation invariance (PI) and permutation equivariance (PE) properties. Specifically, the authors introduce the dynamic permutation network (DPN) and hyper policy network (HPN) that can guarantee PI and PE properties. Evaluations in SMAC, Google Research Football, and MPD show that the proposed frameworks outperform competitive baselines. 
strength_and_weaknesses: **Strengths:**
1. The proposed frameworks can be applied to existing MARL methods by following the minimal modification principle and modifying the input and output layers only.
2. Experiments are performed using the benchmark domains and show state-of-the-art performance compared to competitive baselines.
3. The paper is generally well-written and explained clearly. 

**Weakness:**
This paper is built on three assumptions: 1) the observation and action space are factorized, 2) a different entity order in observation does not affect information (i.e., PI property), and 3) the agent has prior knowledge about the structures of its observation and action space (e.g., in Figure 1, the agent knows which information corresponds to ally and opponent agents and which actions correspond to PI and PE actions). While I agree with the authors that assumptions #1 and #2 generally hold in many MARL settings, assumption #3 can be a strong assumption and can limit the applicability of the proposed framework. For example, the deep set method exploits PI by requiring assumptions #1 and #2, which still hold in model-free MARL. However, to additionally exploit PE, this paper requires assumption #3, which is generally not available for model-free settings. If assumption #3 is strong, it would be beneficial to discuss which potential methods can be used to address assumption #3. 

**Questions:**
1. The related work section states that the representation capacity of the deep set and GNNs is limited due to the use of the shared embedding layer. Do these methods have to use the shared embedding layer? If not, would it be easy to modify these baselines not to use the shared embedding layer and thus improve the representation capacity of these methods?
2. The problem statement is based on Dec-POMDP, the standard framework for collaborative settings. However, this paper uses competitive setting examples, such as SMAC, so I wonder why the problem statement is based on collaborative settings.
3. Regarding the PI Input Layer A in Section 4.2, the weight selection network is *learned* to select the corresponding weight matrix. Which objective is used to train the weight selection network? Also, because this network is learned, which can be noisy at the beginning of training, I would like to clarify how PI is guaranteed. 
clarity,_quality,_novelty_and_reproducibility: **Clarity:** The paper is generally well-written and conveys the main insights well.  
**Quality:** The amount of quality depends on how strong assumption #3 is.  
**Novelty:** While PI and PE properties are separately studied in previous works, this work is the first paper that exploits the two properties simultaneously and explicitly.   
**Reproducibility:** The source code is provided in the supplementary material to reproduce the results.  
summary_of_the_review: I would like to initially vote for 6 (marginally above the acceptance threshold). After the authors' response to my concern and questions, I will make a final decision on the recommendation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses permutation invariance and equivariance in Dec-POMDPs. In a nutshell, as the number of agents increases there is a large number of identical states due to the arbitrary ordering of the agents. 
The authors suggest two methods, Dynamic Permutation Network (DPN) and a Hyper Policy Network (HPN), to address the issue and show convincing performance improvements.

Strength: 
Solid experimental evaluation, including new results that were added during the reviewer period. 

Weakness: 
 The method is somewhat of a straightforward application of results known in other areas of machine learning, in particular from the representation learning community. 


justification_for_why_not_higher_score: The method is somewhat of a straightforward application of results known in other areas of machine learning, in particular from the representation learning community. 
justification_for_why_not_lower_score: Solid experimental evaluation, including new results that were added during the reviewer period.
Decision: Accept: poster