Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper addresses the issue of robustness in cooperative multi-agent RL setups, where the inclusion at test time of an agent that makes error or is even adversarial can drastically decrease performance. The main idea is to compute a correlated equilibrium, by allowing all agents policies to depend on a common signal. To encourage the actions of the agents to correlate, they add a mutual information loss (i.e. a retrodiction that encourages the global latent to be predictable given the action taken). 

High level review:
The paper introduces a natural solution to the studied problem. However, it suffers from a number of flaws that precludes acceptance at ICLR in my mind.

- The proposed method is extremely similar to previous work (Chen et. al and Kim et. al). In particular Chen et al. also suggest something essentially equivalent to the Infomax loss (see equation 5). Their particular definition of the distribution is solely needed for the theoretical proof that correlated policies can be obtained by having a global latent variable (theorem 2.2 in their paper, which is essentially equivalent to proposition 3 in this paper; both theorems are fairly standard use of latent variables to represent correlated distributions). I fail to appreciate the genuine contribution of this paper over Chen et. al. Note that neither approaches (Chen et. al, Kim et. al) are used as baselines in this paper, while they clearly should be. 

- The writing is poor and can often be very confusing.

- The theoretical section does not connect well to the rest of the paper ; section 3 is almost entirely disconnected from what the authors suggest doing (equation 4 is effectively not solved at all, so why introduce this hierarchy of relaxation if the proposed method only poorly relates to it? In the end, the authors train instead an agent in a regular MARL setup with a global variable, and test it in the robustness setting; this is a heuristic that entirely replaces the maximin approach of equation 4 as far as I can understand). Is the counterexample in section 4 novel, or folklore? (It feels very familiar). I would not couch it in terms of a proposition (which is oddly informal - 'can be much larger' - for a theoretical statement), and instead study it more carefully as an intuitive example for why correlation is needed.  A team could play in a decentralized way and obtain very good reward by using the (1,1,1,1,1) action with probability one (which does not require correlation). The issue is that this strategy is not robust to adversaries. This should be explained in more detail.

Reviewer familiarity with the work: I am moderately familiar with the MARL literature (MADDPG, COMA; I worked on on MARL paper), and with game theory (correlated equilibrium). The precise setting of the paper I was not familiar with (robustness with respect to switching from cooperative to mix-cooperative; QMIX algorithm). I was not aware of the work of Kim et al.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Overview

This paper considers robustness in the context of multi-agent setting with decentralized execution, in the scenario where one agent may behave sub-optimally (either by being adversarial or simply taking random actions) on a fraction of the time-steps.

The solution relies on attempting to find a correlated equilibrium by providing agents with a common source of randomness.

The paper then exposes some experimental results on Starcraft. The experiments seem sound and prove the point. However, some details seem to be missing, potentially raising concerns for reproducibility, and the exposition could potentially be improved for better clarity.
Despite these short-comings, which should be easily addressable, the results are convincing and the simplicity of the approach warrants its acceptance.

## Method

The paper starts by theoretically justifying the need of finding a correlated equilibrium, and thus do a good job at motivating the approach.

The method relies on augmenting the agents by providing access to a shared source of entropy, in the form of a shared low dimensional random variable $z$.

To encourage the network to rely on this $z$, the authors add a variational-lower bound type of loss that experimentally helps the performance.

## Experiments

The authors carry out experiments on some Starcraft environments of the SMAC suite.
They build on an established algorithm for these environments, namely QMIX, and provide a reasonable "vanilla" adversarial training scheme as baseline.

Globally, the full method using the variational lower bound loss seems to be performing consistently better than both the adversarial training baseline and the version of the training with only access to z but no variational lower bound loss.

The reporting of the results could probably be improved:
* Figure 2 and 3 use different vertical axis range, making direct comparison difficult
* Training curves are indeed interesting to give an intuition of the general behavior, but I think it would be valuable to provide in the main text an aggregated table summarizing the results in table 1 and table 2 from the appendix, with proper confidence intervals (something like an average performance over maps and agents). This would better convince the reader that the better performance of QMIX-GM is a general trend in all the possible settings.


The paper specifies that the method uses a 3-dimensional uniform distribution in $[0,1]^3$ for $z$, however the design decisions behind such a choice are not discussed or experimentally backed. In particular, it is not clear if the dimensionality of this variable plays any role in the final performance, and if/how practitioners may have to tune this knob when applying the method to their own problem.

Finally, some details are lacking in the architecture used:
* The text says that the latent code $z$ is added as input to the Q network, but doesn't specify the architectural changes that are made to made this possible.
* As for the variational approximation, the text reads "[we] apply a neural network to output its mean and variance". However, not details are provided on this network, and in particular whether it shares parameters with the aforementioned Q network

review_title: Clear exposition, sound experiments, some details missing

final_score: rating: 3: Clear rejection
review_content: Summary

Robustness in the multi-agent setting is a nuanced concept, as (a subset) of agents can act adversarially, while the non-adversarial agents can be trained to be more robust. The authors propose to solve a max-min problem, in which some agents are optimizing their rewards taking into account some agents maybe acting suboptimally/adversarially. 

The authors implement learning using QMIX and evaluate on SMAC. They train a model p(z | a, o) for a global latent variable (using a variational lower bound on mutual info between z and a) that each robust agent conditions its policy on. 

The authors claim this method can yield significant performance improvements over vanilla adversarial training.


Strengths

- Robustness is not well explored in the multi-agent setting, and using a correlated equilibrium seems like an interesting way to model coordinated robust training for (a subset) of MARL agents.

Weaknesses

Throughout, the writing is not very clear. 

- The notation in (3) and (4) is sloppy: which indices are "team" and which ones are "mis"? 
- Sect 4.1 talks about differences between correlated eq and decentralized equilibria, but the authors never formally define what the difference is between "cooperative agents" (which they claim emerges under centralized training, decentralzied execution), and a correlated equilibrium (in which some agents cooperate, presumably, but others don't). I assume the authors are getting at the normal definition of correlated eq: there is some (global) random variable that the policies are conditioned on, but the text is very unclear about this.
- The propositions 1 - 2 seem out of place without a formal defn of correlated eq (see above).
- The text around prop 3 never mentions that agents presumably now learn a policy pi(a | s, z). This should be clarified.
- Eq 7 is missing indices i, or should clarify the "a" refers to a joint action, etc. 

Conceptual issue:

- Adversarial agents are defined as taking actions that minimize their *own* Q-value (i.e., agent i executes argmin_i Q_i). But shouldn't the worst mistake be the action that minimizes other agents' value? It seems this definition of "adversarial" agent is rather weak, and counter to what is defined in Eq (1 - 3), where adversarial agents want to minimize other agents' value as well? This is where the notation again causes confusion.

The experimental results are too thin to conclude the authors' method is indeed effective. 

- What is the significance (if any) of having different agents (6, 4, etc) as the "adversarial" ones in Figures 1, 3? 
- The authors do not explain what the structure of SMAC is (the experiment environment). 
- The authors should visualize the distribution of actions executed by the robust/adversarial agents to give some intuition of the qualitative differences. 

Questions

- What is a "flat maximum" (above Eq 3)?

review_title: Interesting problem, but vague writing and insufficient experimental proof

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers a case in the cooperative multiagent reinforcement learning where one single agent can behave adversarially. It claims that the performance of the whole MARL system  would deteriorate significantly  and demonstrate this phenomenon both theoretically and empirically. Then the author proposes a solution, i.e. , solving the correlated equilibrium in the game.  To seek  the correlated equilibrium, the author introduces a global random variable z and adds a mutual information regularization term.  Indeed this idea is easy to follow since it directly follows the definition of the correlated equilibrium. At last, it tests the algorithm in the SMAC environment and compares it with QMIX algorithm.




Question 1: How do you get the global random variable z in Q(o_i, z,a_i) in the execution. Is it sampled from the posterior distribution q(z|o,a)? 

Question 2: my main concern is about the novelty of this paper. Can you discuss the difference between your method and MAVEN[1], where the author also introduces the latent variable z and uses the mutual information on z and observed trajectory.  The idea seems to be almost the same especially when it is applied on QMIX.

Question 3: The author should include more baselines. Here I just see the result from QMIX. The author should test this idea on other classical methods, e.g., COMA and many others.

[1] MAVEN: Multi-Agent Variational Exploration

review_title: Comments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes a new method for making cooperative MARL algorithms more robust to teammate mistakes. The approach learns correlated equilibria that depend on a global random variable. The setting is discussed with some theoretical results and experiments are provided to compare versions of their approach in Starcraft (SMAC). 

Safe and robust MARL is an important topic. There may be settings in which agents make mistakes or can be in dangerous situations. Making agent policies robust to such circumstances will be necessary in real-world scenarios. Also, the idea of using mutual information to improve agent performance is promising. Sometimes, such information is available (or can be available cheaply) so why not take advantage of it?

There are some questions about the exact robust MARL setting that is approached in this paper. In particular, the paper assumes the setting where only one agent makes a mistake at each timestep and that mistake-making agent is adversarial (i.e., chooses the worst possible action for the team). The specific setting in Eq 5 has the agent choose the minimizing action with a small probability and the maximizing action with the remaining probability. When is such a setting realistic? If all the agents are assumed to be trained centrally with all the agent policies known, why would an agent become adversarial in this way? The setting should be motivated to better understand how realistic and general it is. 

The theoretical results are extensions of the normal form case to the stochastic game case. The theory appears novel and as a result useful, but not a major contribution. 

The algorithm is somewhat straightforward, but there are novel aspects and the idea is promising. As the paper points out, other approaches have used global random variables, but not in this robust context and not with the form of mutual information that is added to improve the use of the global signal. 

The experimental results show the method works well, but they are not sufficiently comprehensive. For example, how does it perform if *no* agents make mistakes? This appears to be shown in Figure 2, but the results aren't clear. It seems like sometimes the proposed methods do better than the normal policy. This shouldn't be possible in the fully observable case, but may be due to the fact that SMAC is partially observable. Using policies that are dependent on a global random variable can improve performance in partially observable settings. This is known (e.g., see the paper below), but never discussed in the paper. SMAC domains are partially observable so it isn't clear how much that is a factor. Because the paper and theory are described from a fully observable perspective, the paper should also have results for the fully observable case. The SMAC results are useful, but the proposed method should also be discussed in terms of partial observability. 

Bernstein, Daniel S., et al. "Policy iteration for decentralized control of Markov decision processes." Journal of Artificial Intelligence Research 34 (2009): 89-132.

The writing is generally understandable, but it should be polished. There are some issues that are not clear such as the fact that a policy in the partially observable case (defined in 3.1) could do better by conditioning on the history. 
review_title: Blind review

Meta-review: The paper tackles the interesting area of cooperative multi-agent learning and presents a promising method to make MAL robust to mistakes of teammates, while learning correlated equilibria. Reviewers find the presented setting and theoretical contributions limited and the experiments not extensive enough; also some technical details about the architecture are lacking, and the notation and writing can be substantially improved upon. As such the paper does not seem ready for publication at this stage.
Decision: Reject