Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper proposes to incorporate a masked attention mechanism in QMIX for value function factorization to disentangle value predictions from irrelevant agents/entities. The masking is based on a random sampling from the whole set of agents to from random subsets, based on which it can compute within-group and without-group Q-functions. The method is able to handle varying types and number of agents. The paper conducts experiments on a simple game to understand the effect, and then test on 3 SMAC games, which shows the effectiveness of the proposed REFIL method.

Strong points:
- The paper is well-written and clear.
- The paper studies an important topic in MARL, i.e., how to deal with varying types and number of agents, and propose a simple yet effective approach, which incorporates attention mechanism in QMIX with random masking.
- Experiments on several games illustrate the effectiveness of the method, with proper ablation study to understand the importance of each component (attention, mixing network, random masking).

Concerns:
The main focus of the paper is to “disentangle value predictions from irrelevant entities”. However, not only does REFIL relies on Q_I (within-group Q-function), but also it relies on Q_O (without group Q-function). If the agent successfully learn this neglect of irrelevant entities, focusing on Q_I would be enough, without triggering additional computation of unnecessary Q_O. Could authors better explain this? In addition, consider the breakaway example, as the attacker has only to focus on the goal keeper, is the random sampling scheme from all agents effective compared with counterparts that only need to focus on the goalkeeper? Could authors conduct additional experiments on football to better support the claim?
review_title: Interesting problem, simple but effective method

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method for randomized factorization of multi-agents for efficient learning. The idea is inspired by the presence of irrelevant entities present in an agent's observational view and how removing those could aid the learning process.
Agents are randomly divided into groups so that an agent can separately measure the influence of entities present in the same group and the entities present in the other groups. Since the groups are randomized, this helps the agent to create groups of variable size based on its utility prediction of in-group and out-of-group entities.

Although the paper only uses two groups for derivation and experiments, it claims that the same method can be applied to more than two groups but is yet to be demonstrated.


The paper is easy to read and the figures are excellent and self-explanatory. The environments chosen are also indicative of the importance of each component. The authors also found that randomized factoring performs better empirically than using domain knowledge while the state-of-the-art lies in the combination of the two. Further, the idea of training variable-sized hypernetwork is quite fascinating and fits well in the overall framework.

Some questions:
Figure 3b does not show results till convergence, please put the entire plots.

I think it would be better if the SMAC setup is explained in more detail. I couldn't understand how the tagging is done (in Appendix). Does it deterministically tie an action to an enemy?

QTRAN has been shown to perform better than QMIX in competitive domains. I would encourage the authors to compare the results with this baseline too.
review_title: Novel approach to factorize entities in a multi-agent env with supportive empirical evidence

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes an observation factorization method to avoid the influence of the irrelevant part on value estimation. Specifically, they design an entity-wise attention network with a masking procedure. This network is used to filter the irrelevant part of the original observation of each agent. Then the output is used to estimate the individual q-value, as well as input to the mixing network to generate the Q_tot. Two kinds of Q_tot are trained together by combing two loss functions linearly with a hyper-parameter. Experimental results show REFIL combined with QMIX surpasses vanilla QMIX and VDN in several SMAC scenarios.

This paper is related to the topics of ICLR. However, I think the related work is not sufficient to cover the background. More detail comments can be found below. 

*****Some specific comments:*****

It is not clear that what is the initialization of two masks, and how to update the masks. 

The authors mentioned there are two groups of entities. However, the entity type is also unclear. I guess SMAC only contains two entity types: alive agents and died agents? How to represent an entity inactive?

One question is why just consider two kinds of groups, what would happen if there exist more than two groups for all entities. In SMAC or soccer, it does contain more than two common patterns. Furthermore, it seems that the masking procedure is hard to extend to the situation with a larger number of groups.

Actually, I think REFIL is similar to ROMA [1] and ASN [2] in different ways. First, REFIL considers two kinds of groups corresponding to a simple version of ROMA which has two roles. Second, REFIL does the same thing as ASN that learns the value estimation by considering a more useful part of the observation. ASN directly divide the observation based on the action semantics, while REFIL tries to learn a suitable observation factorization through entity-wise attention with masking. However, these two very relevant works are not discussed and compared in this paper.

Some suggestions,

I think current experiments could not well support motivation. If authors show some examples in SMAC that what kinds of common patterns agents learn would be better to support this idea.

Since REFIL can be integrated into current MARL algorithms, it is better to consider more recent published MARL methods as baselines, such as QTRAN, QATTEN, QPLEX.

[1] Roma: Multi-agent reinforcement learning with emergent roles. ICML. 2020.

[2] Action Semantics Network: Considering the Effects of Actions in Multiagent Systems. ICLR. 2020.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a randomized entity-based attentional mechanism to regularize the observation space for efficient multi-agent reinforcement learning. Specifically, the authors expect that their method can help agents focus on entities that are relevant to their decision-making process. The aim of the paper is well-positioned in multi-agent settings and are expected to help improve performance by exploiting the loosely coupled structure of multi-agent tasks (although the authors do not explicitly model the decision dependency among agents). However, I have some doubts about whether the proposed method can address the target of the paper.

Intuitively, in multi-agent settings, the number of entities is at least the number of agents, say O(n). O(n) is a quite optimistic estimation because we even do not consider other entities. If the authors want to find an optimal bi-partition over the entity space for each agent, the search space is at least O(2^n), which grows exponentially with the number of agents. To design an efficient search algorithm over such a large space needs to take advantage of some well-designed inductive bias or heuristics. The authors use a random strategy here, which, in my opinion, is not sufficient to guarantee a satisfactory solution. There is no denying that randomization can give a good solution in some cases, but this can not be held as a general rule. Even if the bi-partition structure is trained end-to-end, I also suspect that learning such a structure is not easier than learning from scratch.

Additionally, I think the authors largely ignore the contribution of a related work [Agarwal et al., 2020]. Although they cite this paper in Sec. 5, unlike what is stated in this paper, the main contribution of [Agarwal et al., 2020] is a GNN-based attentional mechanism over entity spaces. They propose to let agents learn to attend to different entities under different observations. In this way, the target of [Agarwal et al., 2020] and this paper largely overlap. I was expecting that the authors provide a thorough comparison with [Agarwal et al., 2020] in their experiments. If the authors can demonstrate that their method can outperform [Agarwal et al., 2020], I will consider improve my rating.


[Agarwal et al., 2020] Agarwal, A., Kumar, S., Sycara, K. and Lewis, M., 2020, May. Learning Transferable Cooperative Behavior in Multi-Agent Teams. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (pp. 1741-1743).
review_title: A more serious discussion about [Agarwal et al., 2020] is expected.

Meta-review: This paper proposes an attention based technique to focus on relevant entities in multi-agent reinforcement learning.  While the effectiveness of the proposed method is demonstrated on some tasks, there remain major concerns including the following:
1. It is not sufficiently convincing that the proposed method performs well in more complex domains
2. Novelty over Agarwal et al. and MAAC is rather minor
Decision: Reject