Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The motivation for this paper is straightforward. The author believes that the semantic information of actions should be explicitly considered in the multi-agent reinforcement learning problem, and the information needed to make different semantic actions should be different. To this end, the author divides the action semantics into two categories, one is actions that only affect the environment and itself, and the other is actions that affects the other agents. Making a decision corresponding to the former requires relying on all local observations of the agent, but making a decision corresponding to the latter requires only local observations related to the affected agent.

To this end, the author proposes a novel network structure ASN that can achieve the above objectives. Because it is a modification of the network structure, ASN can be combined with any type of multi-agent reinforcement learning algorithm to improve its convergence speed and final performance. The author conducted a rich experiment and verified the validity and superiority of the ASN structure from various angles.

But for this paper, I have doubts in the following aspects:
1. In the experimental part, the author compares the ASN-based algorithm with QMIX and VDN in the StarCraft II environment, but the performance of the benchmark algorithm on the 8m and 2s3z maps is significantly lower than that of the original paper. Because the author did not use the RNN network when implementing these algorithms? If so, why not implement it with the RNN network? Can ASN also be combined with RNN?
2. The basic idea of ASN is similar to the attention mechanism, and the experimental part also shows that the algorithm with attention module also achieved good results. I think the author should also compare with the MAAC algorithm (Actor-Attention-Critic for Multi-Agent Reinforcement Learning, ICML2019). There are two reasons for this: First, the MAAC algorithm is a SOTA MARL method based on the attention mechanism; secondly it belongs to the actorcritic algorithm. In the comparison part of the JAL algorithm, the author only considered the value-based methods VDN, QMIX, etc., without considering the policy gradient method or the actor-critic method.
3. In the experimental part, the author also compares the proportion of ASN-based methods and benchmark algorithms that perform non-valid actions during training, such as to attack an agent outside the scope. However, in the network structure of ASN, if an agent is outside the field of view, it will not output the Q value or probability of the action corresponding to that agent. Does this mean that ASN is impossible to choose non-valid actions? So is the corresponding experiment meaningless?
4. If an action can affect multiple other agents at the same time, can the ASN network handle it?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper focused on the multi-agent systems, and proposed a new network architecture (aka ASN). In the new architecture, the action set is  manually split into two subsets, each of which contains the actions that affects other agents or not.  Besides the net architecture, the authors also discussed several important issues including “Multi-action ASN” and  parameter-sharing. In the experiments, the authors evaluate the proposed nets in Starcraft and Neural MMO, and shows superior performance compared with other methods. 

Comment:
This paper is relatively well-written and clearly introduces the basic idea of the work. The authors focuses on an important issue for MAS, i.e. how to reduce model complexity when increasing the number of agents. In their work, the authors would like to decompose the action set into two smaller sub-sets based on the so called action semantics, such that the designed nets have simpler and more straightforward (efficient) structure. It is interesting and  somehow convincing. One question on this point how to split the action set properly? My concern is from more general cases, where boundary  to classify the action set would be not so clear. In this scenario, how to obtain an proper split. And what would happen if we choose the wrong subset? At the beginning, I though the split would be learned by some sub-nets, but after reading I found it seems to manually set it. This is the motivation why I concern this problem. 

Another point I concern is about the parameter-sharing (PS) issue in Page 5. I understand the authors want to impose the PS trick to reduce the training complexity for the model.  I also agree with this, but there is several points, which confuse me somewhat. (1) I’m still confused the detail of PS. Does PS means the agents share the exactly same weights, same structure, some correlation or else? (2) I know the PS trick is popularly used in multi-task learning, in which the designers have to face the question “HOW MUCH common information should the individual networks share”. It means that PS trick might results in challenging model selection problem in practice. Hence I am curious how such issue is handled in the proposed model? I know it might not be the main contribution of this paper, but I think it is important to discuss, which might be missed in the paper. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a neural network architecture that provides an agent-agent based embeddings that are used for actions that directly affect specific agent. Proposed architectural choice exploits (implicitly assumed) independence of some actions wrt. observations of agents that it is not directly affecting. Authors perform experiments in minigame of SC2 (controlling small armies) and in NeuralMMO (analogous setup).

It is worth stressing, that apart from conducting typical analysis, authors also put forward more qualitative hypotheses, that are then verified by custom experiments. I highly value this way of conducting research, as it provides way more insights into the method than just pure return plots, often encountered in similar papers.

Major comments
- Please be more explicit in the abstract, when talking about the experiments conducted. Claiming "Experimental results on StarCraft II and Neural MMO show ASN significantly improves the performance of state-of-the-art DRL approaches compared with several network architectures." could be misinterpreted as providing a solution that can play SC2 better than state-of-the-art methods, while the actual result is to be able to micromanage small fights in this environment well, and thus it would be fairer to say "on StarCraft II Minigame" or more explicitely "StarCraft II small scale fight simulation" or anything else in this spirit. These are great research domains, there is no need to overclaim the results.
- A proposed architecture assumes (to some extent) an independence of effects of action-agent pairs (action aimed at player i is conditionally independent from observation of agent j). There is still a way to represent it, but only by bypassing the proposed architecture, and expressing this behaviour directly in e^i. While this sort of decomposition makes sense for various heavily localised problems (such as micro-management of fights in games like SC2 or NeuroMMO) it looks like a potential limitation, and assumption, that is not articulated in the text. Have authors performed any analysis looking into this phenomenon?
- It is unclear, how ASN compares or relates to the architecture used in OpenAI Five network (https://d4mucfpksywv.cloudfront.net/research-covers/openai-five/network-architecture.pdf) which arguably should be used as a baseline, given that it follows a similar idea of producing per-agent embedding, that influences part of action space, that is directly related to influencing other agents (targeting opponents). Despite not being peer reviewed, it is a well known, and clearly described method, which should at the very least be acknowledged as prior work.
- Some equations are hard to parse, e.g.:
-- page 5, sum_i=1^b [(y_t^tot - Q_tot(s,a; theta))^2]  <- "i" is never used in the equation (I guess it changed meaning, and is no longer agent identity, but rather batch index?)
-- page 4, what does expectation over "t" mean? In the introduced notation, expectation is taken wrt. control policy pi_i, interpreted as a joint probability distribution over agent actions and environment state transition. Can this be unified?

Minor comments
- "When one of the arm units dies" -> "When one of the army units dies"
- details of ASN-PPO could be safely moved to the Appendix, as it is a direct substitution regular policy updates with (2), rather than a separate contribution that requires in-depth description (and given that paper is quite long, it could use a bit of message "sharpening")

I am happy to revisit (and increase) the rating assigned, once authors address the comments above, especially the relation to OpenAI Five architecture.
review_title: Official Blind Review #2

Meta-review: The authors address the challenge of sample-efficient learning in multi-agent systems. They propose a model that distinguishes actions in terms of their semantics, specifically in terms of whether they influence the acting agent and environment or whether they influence other agents. This additional structure is shown to substantially benefit learning speed when composed with a range of state of the art multi-agent RL algorithms. During the rebuttal, technical questions were well addressed and the overall quality of the paper improved. The paper provides interesting novel insights on how the proposed structure improves learning.
Decision: Accept (Poster)