Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the problem of coordination in multi agent reinforcement learning, particularly in the decentralized training / partial observability setup. The paper proposes to derive a self-supervised reward from alignment. Agents are encouraged to take actions that match other agents' expectations when they cooperate and they are encouraged to do the opposite with competing agents.

For each agent the reward is computed using a dynamics model. This model is trained by predicting the next observation given the current action and observation. This model is then used to derive a reward that estimates the neighbors expectations of the agent's next state. The reward is decentralized and does not require the centralized training / decentralized execution which is common in the multi agent reinforcement learning literature.

The proposed reward is evaluated across many problems using the multi agent particules environment and Google Research football. The authors compare their algorithm with several existing baselines and the alignment based reward is shown to perform better than existing techniques for cooperative and competitive problems. 

strengths_and_weaknesses: I found this paper to well be written and easy to follow. The authors made an effort to provide many helpful illustrative examples of the behaviour of the proposed technique. The method is simple and appears easy to implement.

The empirical evaluation of the paper is solid, it includes cooperative and competitive problems, with decentralized and centralized training and the proposed method is compared with several existing baselines. I found section 5.7 to be useful to understand how alignment reward helps and what are the limitations, particularly that the overall performance of the system decreases when the accuracy of the dynamics model decreases or that the method is less is competitive with heterogenous agents are used.
questions: L282 "When the number of agents increases, alignment scales well in all multi-agent particle tasks except for heterogenous navigation."
Do you expect this statement to remain true when the number of agents increases to hundreds or even thousands? In that case I would imagine that the summation L159 would make the alignment reward pretty noisy and coordination between agents would be more difficult.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed an intrinsic reward that learns each agent to align with the predictions of neighbors in a fully decentralized way for improving multi-agent coordination. They compared the proposed method with the intrinsic reward-based baselines, e.g. curiosity-based approach in several environments including Google Research football, and provides some investigations to show the performance improvement of the proposed method.
strengths_and_weaknesses: Strengths 

-	This paper is well written and easy to follow. 
-	This paper deals with enhancing predictability of other agents in a fully decentralized way, which is a significant problem in MARL.
-	This paper provides some experiments to show why the proposed method works.

Weaknesses

-	The novelty of this paper is not enough for accepting the venue of NeurIPS.
-	There are some concerns of the proposed method (Please see “Questions”)
-	The lack of experiments on the popular benchmarks such as SMAC and the sparse environments.

questions: 1. The proposed intrinsic reward is based on how much the neighbors predict the agent well. For the fully decentralized training, the authors replace the neighbors’ dynamic models with their own dynamic models. Eventually, the proposed intrinsic reward is the opposite of the novelty-based intrinsic reward approach, and thus the multiple agents will be trained to visit the familiar states that each agent can predict well. Although the approximation of the neighbors’ dynamic models with a proxy is empirically good, it is not exactly the same as the neighbors’ models. Each neighbor may predict the agent differently. Can we say that the proposed method enhances the alignment? the performance improvement can come from other reasons.

2. Since the proposed intrinsic reward is the opposite of the novelty-based intrinsic reward, can the proposed intrinsic reward degrade the exploration?  

3. In general, the curiosity-based approaches are not good at the dense-reward environment. For a fair comparison, the authors should provide experiments on both dense/sparse environments. 
4. Furthermore, it would be great if the authors provide the performances as the learning processes.
5. The authors should provide the experiments on the SMAC environment. Also, the intrinsic reward-based baselines such as [1] are missing. The authors should compare the proposed method with [1]. 
6. In Section 4.2, the authors restricted the future prediction from j’s point of view by using the portion of j’s observation i can see. The reviewer thinks that for the fully decentralized training, A should be unaware of how much B can observe. Is it okay that Agent i knows o_{i \and j} in the fully decentralized training?

[1] Zheng, Lulu, et al. "Episodic multi-agent reinforcement learning with curiosity-driven exploration." Advances in Neural Information Processing Systems 34 (2021): 3757-3769. 
limitations: As mentioned in the “Questions”, the author should provide the experiments on the SMAC environment and compare the proposed method with the state-of-the-art baseline of the intrinsic reward-based MARL algorithm [1].
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This paper proposes to tackle the problem of multi-agent learning under decentralised training and partial observability through using a novel self-supervised intrinsic reward that encourages agents to behave predictably to their neighbouring teammates and unpredictably to their adversaries.
- A dynamics model (trained on the agent’s own observations as a proxy given training is decentralised), is used to predict future states from other agents’ viewpoints under partial observability. The negative error in this prediction is then used as an intrinsic reward to supplement training in environments with sparse extrinsic rewards.
- The effectiveness of this approach is compared to curiosity driven intrinsic rewards on 6 cooperative and competitive multi-agent tasks. Further experiments are conducted in regimes of full observability, centralised training, and zero shot coordination.
strengths_and_weaknesses: ## 

- *Originality:*
    - The authors frame their work well in the context of the literature on intrinsic motivation in reinforcement learning, and motivate their approach from the natural world. I appreciate that they explicitly state the differences of their approach to closely related work, e.g. Ndousse et al 2021.
    - Some references need to be corrected:
        - Line 45: “*Only a few attempts explore other forms of multi-agent intrinsic rewards”.*   A more specific, referenced statement would be better here.
        - Line 33: To my knowledge, there is no reference to decentralised training and partial observability in Lowe 2017. MADDPG is a centralised training, decentralised execution algorithm. The authors should reference more appropriate work for that statement on task-specific reward shaping to motivate their work.
- *Quality*:
    - Two standard coordination and competition benchmark environments are used to evaluate their framework. The same architectures and optimisation algorithms are chosen to provide a fair comparison against other intrinsic rewards. I particularly liked the symmetry breaking experiments to demonstrate learned coordination.
    - Table 1 shows promising results in small scale decentralised, partially observable settings. However results are not consistently *significantly* better than the baselines across tasks and settings. There are claims of superior performance in some tasks in the text but the the baselines have similar performance to the proposed method within standard error. For example, Figure 3 is selective to show results against SPARSE while CURIO performs similarly to ALIGN within error. Table 1 3v1 w/ keeper and Table 2 keep-away - CURIO baseline has similar performance within error. Symmetry-breaking performance is also not significant when scaled (appendix). In general the authors should be careful not to overstate claims unless there is statistical significance to back it up.
    - To improve the clarity of results I would suggest incorporating some learning dynamics curves. Only converged test rewards and task-specific converged metrics are currently given. Does coordination help with sample efficiency of training?
    - Comparison to Ndousse et al 2021 would also be useful to see whether this reward framing has a benefit over their proposed approach.
    - On line 161 there is a claim that the proxy dynamics model is validated in heterogeneous tasks but in the experiments it is shown that this is not the case in the experiments.
- *Clarity*:
    - The paper is well written in many parts. Paragraph headings and subsections are used effectively and the paper on the whole reads quite intuitively. Figure 1 is clear and informative to the reader.
    - To improve clarity:
        - the authors mention a gaussian noise experiment in the main text and refers to appendix but I cannot find this section in the results.
        - Figure 1: I would suggest increasing the font width for readability (particularly the equation)
        - The tables and figures are quite far away from their relevant sections in the text which hinders readability.
        - Line 293-294 : “analyse this in future experiments” - it is unclear if this means later in the paper or in future work.
    - A careful proofread is also required. Some issues I spotted:
        - Line 256: “a” not “an” goal state
        - Table 2 caption: misspelled “trained”
        - Iqbal & Sha 2019a and 2019b seem to refer to the same paper.
        - Line 160 misspelled “empirically”
        - Line 178: misspelled “team”
        - State and action space paragraph starting on Line 194 has a number of grammatical errors.
- *Significance:*
    - The paper addresses the problem of decentralised training which is less explored in the literature compared to centralised training but important for the scalability of MARL.
    - The alignment idea to use predictability of actions as an intrinsic reward is novel to my knowledge and well motivated. However, as the authors note in the related work section, there is a body of similar work that utilise model-based predictability in different ways e.g. as an auxiliary loss. It is not clear to me yet why using this inductive bias as a reward instead is significant, particularly without an experimental comparison.
questions: 1. Did you compare this intrinsic reward based approach to other (non intrinsic reward based) approaches to the sparse learning problem (e.g. Ndousse et al 2021)?
    - Some discussion around why the framing as an auxiliary reward instead of an auxiliary loss would be helpful.
2. Under what conditions do we expect alignment/misalignment to work well? A discussion around why we see that ALIGN self, team and adv have varying quality on different tasks would be helpful. 
    - For example, why does modelling from the other agent’s point of view, instead of directly predicting your own future state (which incorporates the other agents action) more helpful? When and why is ALIGN_team better than ALIGN_self? It seems that the team set up is just a partially observed view of the self set up, because the agent’s own dynamics model is used.
    - In the adversarial case, it seems that the agent will learn to be unpredictable to itself (as it uses its own dynamics model as a proxy) so it is unclear to me why this would help. Conversely, ALIGN_team and ALIGN_self seem to work better than ALIGN_adv on some competitive tasks indicating that it is sometimes better to be predictable to an adversary which seems counterintuitive.
    - In the first paragraph of Section 4 there is discussion about gaining information outside of the agent’s receptive field through encountering surprising actions from other agents. How does the alignment reward help convey this information? Does this help at execution time?
3. In the centralised regime experiments, did you use the actual dynamics model learnt by each agent to calculate the intrinsic reward or did you still use the individual agents dynamics model as a proxy as in the decentralised case?
4. Do you have any suggestions for how to improve performance using alignment for heterogenous agents?
5. Did you evaluate zero shot coordination with CURIO models as well?
limitations: - The authors state limitations of their approach, namely:
    - the small action/state spaces of the environments considered.
    - difficulty generalising when there are heterogenous agents due to agents learning only one dynamics model (of themselves). They argue that this enables more scalability, but demonstrate that this leads to decreased performance.
    - The method is reliant on the ability to learn a good dynamics model which may be more difficult in noisy, real-world environments.
- I have addressed the other limitations in the strengths and weaknesses section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an auxiliary training reward called ALIGNment. Agents are rewarded when either they (ALIGNself) or their teammates (ALIGNteam) can predict their actions, or when their adversaries (ALIGNadv) cannot predict their actions. The authors use experimental evidence to support the intuition that alignment helps agents coordinate which subtasks to complete and helps agents coordinate zero-shot with new partners. Compared with 3 prior approaches in 6 Google Research football environments and 5 multi-agent particle environments, at least one ALIGN variant achieves state-of-the-art results in every task except for the heterogeneous navigation task.
strengths_and_weaknesses: ## Strengths
Overall, I found the method easy to understand. Algorithm 1 makes the method clear.

The experimental evaluation is thorough, comparing against 3 prior approaches and 11 total tasks. At least one ALIGN variant performs best across all but one task.

The authors include additional experiments to help study why ALIGNment improves performance.

As this is an intuitive approach that is explained clearly and supported by thorough experimental evaluation, I recommend acceptance.

## Weaknesses
The word “alignment” already has a meaning in artificial intelligence. Specifically, it is used to mean how well an artificial intelligence is aligned to human preferences. See, for example, “The Alignment Problem” by Brian Christian. I would recommend the authors use a different word for their method, such as “Predictability.” This renaming would respect the existing use of the word “alignment” as well as be more specific about what the auxiliary loss is rewarding.

I didn’t find Figure 1 very helpful for understanding the method. The idea behind alignment – a predictive loss – is very simple, yet Figure 1 is quite complex. I’d recommend trying to redesign Figure 1 to make it as simple as possible.

The overall results are presented in two dense tables, Table 1 and Table 2. It would be helpful if the authors could present aggregated results in a single figure. If possible, I would recommend normalizing results using the performance of an expert policy and a random policy as is done in the D4RL benchmark [1], i.e., computing (performance - random) / (expert - random). This gives a common scale over which the authors can aggregate.

Only the maximum performance is bolded in the results tables. I recommend bolding all results within 5% or 10% of maximum performance, to account for noise, as is done in other work such as [2].

[1] D4RL: Datasets for Deep Data-Driven Reinforcement Learning

[2] Offline Reinforcement Learning with Implicit Q-Learning
questions: Did you train the Sparse, Curio-self, and Curio-team methods yourself? Or did you take numbers from prior work? It seems like you trained them yourself, but it would be helpful to make this more clear.  (I think it’s stronger if you can report numbers directly from prior work where applicable, because the authors of prior work have an incentive to tune their algorithms as well as possible.)

If you trained prior methods yourself: Did you spend equal effort tuning the prior approaches as you did tuning your own method, and can you quantify this in the paper (eg by having tested the same number of hyperparameter settings)?
limitations: The authors note that their method, ALIGN, performs worse than prior methods in Heterogenous navigation. They hypothesize that this is because the differing size and speed of agents in Heterogenous navigation makes it difficult for agents to predict each other. I appreciate that the authors analyze this limitation.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies a new self-supervised MARL intrinsic reward, called Alignment, to facilitate learning in a decentralized training paradigm under partial observability. Inspired by the self-organization principle in Zoology, the intrinsic reward of Alignment aims to guide agents to match their teammates' expectations in order to emerge cooperative strategies. The experiments are carried out on the multi-agent particle and Google Research football, and the results show that Alignment outperforms sparse and curiosity-based intrinsic rewards.

strengths_and_weaknesses: The reviewer will list the main strengths and weaknesses as follows:

STRENGTHS

1) This paper introduces an interesting intrinsic reward, called Alignment, in decentralized training under partial observability. This idea is inspired by Zoology and brings a new heuristic to emerge coordination during decentralized training.

2) This paper conducts several experiments on popular MARL benchmarks to show that the Alignment intrinsic reward is a more useful method than curiosity-driven exploration.

WEAKNESSES

1) The major concern is that the method of Alignment (see Section 4) theoretically conflicts with the decentralized training paradigm. In decentralized training, agents cannot share their expectations. Thus, each agent theoretically cannot match their teammates' expectations because they do not know them. For example, the dynamics model of agent $i$ in line 150 is trained on the $o_i$ and $a_i$ but the alignment intrinsic reward $r_{in}$ defined in line 155 apply the dynamics model of agent $j$ on the $o_i$ and $a_i$. Note that the dynamics model of agent $j$ is trained on the set of $o_j, a_j$ rather than $o_i,a_i$. $r_{in}$ defined in line 155 may induce huge penalty due to this distributional shift from $o_j,a_j$ to $o_i,a_i$.

2) When the agent aims to match their teammates' expectations, what if their teammates do not reach a consensus? For example, in the definition of $r_{in}$ on line 155, if agent $i$'s neighbors have completely different expectations, which expectation does agent $i$ match?

3) Sample efficiency is an important measure for MARL evaluation. Could you provide the learning curve during training to better visualize the sample efficiency in comparison? Tables may not be a natural way to demonstrate the advantage of exploration methods.
questions: The following questions correspond to three points listed in the WEAKNESSES.

1) How do the agents share their expectations with teammates during decentralized training?

2) When the teammates have entirely different expectations for agent $i$, how does agent $i$ match their expectations?

3) Why use tables instead of learning curves to visualize learning performance?
limitations: The main concerns are listed in the WEAKNESSES. The authors should justify the soundness of the proposed method in Section 4.2.
review_title: null

Meta-review: metareview: This paper introduces a novel method for decentralised training in both cooperative and competitive environments. 
The main insight is that agents should be predictable to their team mates but unpredictable to adversaries.
Crucially, each agent relies on a local model to optimise this objective, making it compatible with decentralised training.

There were a few concerns from reviewers around both how the method applies to the decentralised training regime and regarding the naming. The authors managed to address the concerns appropriately, leading the actively engaged reviewers to increase their scores substantially. Reviewer TEM3 kept their rating at a borderline reject even though the concerns were addressed by the rebuttal. As such I recommend to discard this review. Reviewer whMR stated that they would increase their score but to the best of my knowledge didn't do so. Again, their review should be seen in this context. 

confidence: Certain
Decision: Accept