Robust Multi-Agent Reinforcement Learning Driven by Correlated Equilibrium
Authors: Yizheng Hu
Kun Shao
Dong Li
Jianye HAO
Wulong Liu
Yaodong Yang
Jun Wang
Zhanxing Zhu
Venue:  (2021)
Keywords: Robust
Multi-agent
Reinforcement Learning
Correlated Equilibrium

Abstract: In this paper we deal with robust cooperative multi-agent reinforcement learning (CMARL). While CMARL has many potential applications, only a trained policy that is robust enough can be confidently deployed in real world. Existing works on robust MARL mainly apply vanilla adversarial training in centralized training and decentralized execution paradigm. We, however, find that if a CMARL environment contains an adversarial agent, the performance of decentralized equilibrium might perform significantly poor for achieving such adversarial robustness. To tackle this issue, we suggest that when execution the non-adversarial agents must jointly make the decision to improve the robustness, therefore solving correlated equilibrium instead. We theoretically demonstrate the superiority of correlated equilibrium over the decentralized one in adversarial MARL settings. Therefore, to achieve robust CMARL, we introduce novel strategies to encourage agents to learn correlated equilibrium while maximally preserving the convenience of the decentralized execution. The global variables with mutual information are proposed to help agents learn robust policies with MARL algorithms. The experimental results show that our method can dramatically boost performance on the SMAC environments.

Under review as a conference paper at ICLR 2021
ROBUST MULTI-AGENT REINFORCEMENT LEARNING DRIVEN BY CORRELATED EQUILIBRIUM
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we deal with robust cooperative multi-agent reinforcement learning (CMARL). While CMARL has many potential applications, only trained policy that robust enough can be confidently deployed in real world. Existing works on robust MARL mainly apply vanilla adversarial training in centralized training and decentralized execution paradigm. We, however, find that a CMARL environ- ment contains adversarial agent, the performance of decentralized equilibrium might perform significantly poor for achieving such adversarial robustness. To tackle this issue, suggest that when execution the non-adversarial agents must jointly make the decision to improve the robustness, therefore solving correlated equilibrium instead. We theoretically demonstrate the superiority of correlated equilibrium over the decentralized one in adversarial MARL settings. Therefore, achieve robust CMARL, we introduce novel strategies encourage agents learn correlated equilibrium while maximally preserving the convenience of the decentralized execution. The global variables with mutual information are pro- posed to help agents learn robust policies with MARL algorithms. The experi- mental results show that our method can dramatically boost performance on the SMAC environments.
INTRODUCTION
Recently, reinforcement learning (RL) has achieved remarkable success in many practical sequential decision problems, such Go (Silver et al., (017). chess (Silver et al. 2018), real-time strategy games (Vinyals et al., 2019), etc. In real-world, many sequential decision problems involve more than one decision maker (i.e. multi-agent), such as auto-driving, traffic light control and network routing. Cooperative multi-agent reinforcement learning (CMARL) is a key framework to solve these practical problems. Existing MARL methods for cooperative environments include policy- based methods, e.g. MADDPG (Lowe et al. (017). COMA (Foerster et al. 2017). and aalue-based methods, e.g. VDN (Sunehag al. a 2018). QMIX (Rashid et 2011)) QTRAN (Son al.) 2019). However, before we actually apply CMARL'S policy into real world applications, a question must be asked: these learned policies safe or robust to be deployed? What will happen if some agents made mistakes or behaved adversarially against other agents? It will be most likely that the entire team might fail achieve their goal or perform extremely poorly. (Lin et al. (2020) demonstrates the unrobustness in CMARL environment, where a learnt adversarial of one agent can hugely decrease the team's performance.
Therefore, in practice, we expect to have multi-agen team policy in a fully cooperative environ- ment that is robust when some agent(s) make some mistakes and even behave adversarially. To the best of knowledge, very few existing works on this issue mainly use vanilla adversarial training strategy. Klima et al., (2018) considered a two-agent cooperative case, in order to make the policy robust, agents become competitive with a certain probability during training. Li et al. (2019) pro- vided a robust MADDPG approach called M3DDPG, where each agent optimizes its policy based on other agents' influenced sub-optimal actions.
Most -th MARL algorithms utilize centralized training and decentralized execution (CTDE) routine, since this setting is common in real world cases. The robust MARL method M3DDPG also followed the CTDE setting. However, existing works on team mini-max normal form or extensive form games show that if the environment contains adversarial agent, then the

---

Under review as a conference paper at ICLR 2021
decentralized equilibrium from CTDE routine can be significantly worse than the correlated equilib- rium. We furthermore extend this finding into stochastic team mini-max games. Inspired by this im- portant observation, if we urge agents to learn correlated equilibrium (i.e. the -overarrralliall agents jointly make the decision when execution), then we may achieve better performance than those following CTDE in robust MARL setting.
In this work, we achieve the robust MARL via solving correlated equilibrium motivated by the latent variable model, where the introduction of a latent variable across all agents could help agents jointly make their decisions. Our contributions can be summarized as follows.
We demonstrate that in stochastic team mini-max games, the decentralized equilibrium can be arbitrarily worse than correlated one, and the gap can be significantly larger than in normal extensive form games.
With this result, we point out that learning correlated equilibrium indeed necessary in robust MARL.
We propose a simple strategy to urge agents to learn correlated equilibrium, and show that this method can yield significant performance improvements over vanilla adversarial training.
2 RELATED WORKS
Robust RL The robustness in RL involves the perturbations occurring in different cases, such as state observation, environment, action or policy, opponent's policy, etc. 1) For the robust- ness to state or observation perturbation, most works focused on adversarial attacks of image state/observation. Pattanaik al.) (2018) used gradient-based attack on image state, and vanilla adversarial training was adopted to obtain robust policy; Fischer et al. al.] (2019) first trained a normal policy, and distilled it on adversarial states achieve robustness; Ferdowsi al. (2018) applied adversarial training to autonomous driving tasks that interfered agent's 's input sensors based on en- vironment, and then conducted adversarial training: 2) For the robustness to environment, robust Markov decision process (MDP) could be used to formulate this problem. Many works (e.g. WWese- mann al. 2013); ; Lim al. 2013) have studied this model and provided both theoretical analysis and algorithmic design. In deep RL scenario, Rajeswaran et al. (2016) used Monte Carlo approach train agent, while Abdullah AbbullA et al. (2019): Hou et al.] (2020) adopted adversarial training to obtain robust agent to all environments within Wasserstein ball. Mankowitz et al. (2019) conducted adversarial training
MPO algorithm to optimize the performance the worst performance envi- ronment. 3) To be against the perturbation of action policy, Tessler et al. (2019); Gu et al. (2018); Vinitsky et al. (2020) considered the case that agent's action may be influenced by another action, and conducted adversarial training. 4) For the robustness to opponent, Pinto al. (2017); Ma et al. (018 focused on the case that agent's reward may be influenced by another agent, and adversarial training was implemented to solve the two-agent game to obtain robust agent.
Correlated Equilibrium Correlated equilibrium is a more general equilibrium in game theory compared Nash equilibrium. In cooperative task, if the team agents jointly make decisions together, then the optimal team policy is correlated equilibrium. Correlated equilibrium widely studied in game theory (e.g. Hart & Mas-Colell (2001 (2000); NNyman (1997)) In team mini-max game, solving the team's ' correlated equilibrium in normal form game is straightforward (just treat the team single agent); Celli & Gattil (2017): Zhang & An (2020; Farina al. (0018) proposed various algorithms to solve correlated equilibrium in extensive formal games. In deep RL scenario, Celli et al. (2019) applied vanilla hidden variable model solve correlated equilibrium in simple repetitive environments, while information loss with hidden variable model was used in Chen et al. (2099) to solve correlated equilibrium normal multi-agent environment.
ROBUST MARL
3.1 BACKGROUND
A typical cooperative MARL problem can be formulated as a stochastic Markov game (S, (Ai) P), where S denotes the state space, (A:) is i-th agent's action space. The en-
2

---

Under review as a conference paper at ICLR 2021
vironment starts at state Sy based some initial distribution po- At each time step t. agents select joint action a, E 1- based on some policy Tteam (ar [st) and receive reward r(s,, The en- vironment will transfer to new state Se+1 P( a,). The goal is to maximize agents' expected accumulated reward: maxxicam "
E r(8,, ax)].
Most otfte-of-the-art MARL algorithms utilize CTDE routine, that is, each agent selects its own tion independently. If the environment is fully observable, then at each timestep t, each agent selects an action 0ür E Ad based some policy Ty (axx (se,, and if the environment is partially observable, then at each timestep I, each agent will receive an observation Oit based on 8t, and select action based on policy ny =a(a)l)0))) And the goal becomes max
Ba
S [ r(st, a1m,,)].
3.2 ROBUST MARL AND VANILLA ADVERSARIAL TRAINING
The motivation of our work is to obtain a policy that is robust when one agent makes some mistakes. In normal MARL algorithm, the team guarantee achieve high reward only when all agents accurately execute their optimal strategies. However, this may not always be true in real world scenarios. Real world agents may occasionally make mistakes (e.g. machine malfunctioning). To achieve the robustness to this kind of mistakes, we propose to solve the worst case mini-max prob- lem:
Er(si, r(si, ((s1,a1mm)) aim.))
Fixed or V i, max min Exg TT Biim Ki.mia
s.t. D Di S
where {-i} means all except mimis means the mistaken policy that the mistaken agent actually perform. D is some kind of distance measure, since we cannot expect that the team policy can still be robust when one agent makes very big mistakes.
Unfortunately, this mini-max problem is hard to solve, since it contains two MDPs nested with each other. On the other side, the common case real world that, agents make mistakes randomly, i.e. these mistakes are most likely not related to the team's 's goal and other agents' policies. Also, agents typically make mistakes only occasionally, since agents that always/frrquently makes mistakes are not allowed to be deployed in practice. So following these considerations, consider a simpler case and instantiate the robust cooperative MARL into QMIX algorithm.
We consider weaker worst case mini-max problem. Since we assume agent only make mistakes occasionally, we consider the case that the mistaken agent executes its "worst" action in a certain probability Also we assume that the mistakes most likely not related to the team's goal and other agents' policies. In QMIX, an agent doesn't 't consider other agents' policies, then its worst action is the one that minimizes its own Q, function (since in QMIX &Qua to lower Qtoe). In summary, we let the mistaken agent perform
aQ. 2 0, lower Q; will lead
farmaxx, Qi(s, a) larg Qi(s, a)
with prob. 1-E with prob.
Gi,mis
and apply vanilla adversarial training obtain a robust policy. The detailed algorithm framework 2
described Appendix A.I The performance vanilla adversarial training will be considered as baseline.
ROBUST MARL WITH CORRELATED EQUILIBRIUM
4.1 ROBUST MARL REQUIRES CORRELATED EQUILIBRIUM
In this part, we will show that with naive adversarial training in a centralized training and decentral- ized execution fashion, the learned policy might be sub-optimal in adversarial settings, thus requiring a correlated equilibrium.
In typical MARL settings, if the environment is fully cooperative, then the algorithms with central- ized training and decentralized execution (e.g. QMIXMADDP/CCOMA etc) can achieve state- of-the-art performance in certain environments. This indicates that at least for these environments,
3

---

Under review as a conference paper at ICLR 2021
correlation in execution is not necessary. Furthermore, Lauer & Riedmiller (2000) proved that de- centralized exccuted policy can achieve optimal performance for fully observable and fully cooper- ative RL.
However, in the robust MARL setting, some agent(s) become adversarial, indicating that the envi- ronment is not fully cooperative now. So the question is: whether the correlation in execution is necessary in adversarial scenario?
For the settings of Eq. I the problems actually become team mini-max game. The works team mini-max normal form or extensive form games (von Stengel & Koller 1997, [Bsslico et al. 2016 Celli & Gatti 2017), pointed out that the decentralized equilibrium be significantly worse than correlated equilibrium at least for some games, In normal form team mini-max max game, Basilico et al. (2016) proved that the gap between correlated and decentralized equilibrium is most O(m" (mm"")), 2), where n is the number of agents (includes the adversarial) and m is each agent's 's number of action. And the bound tight.
During this section, we simply define the "correlated equilibrium" as the equilibrium of optimal "correlated policy", that is, the team learns the optimal policy Tieam(alas together, and "decentral- ized equilibrium" as the equilibrium of optimal "decentralized policy" learned by CTDE algorithm.
We denote Eeor as the team's is expected reward under their optimal correlated policy, and Edec as that in their optimal decentralized policy. In MARL, agents play stochastic game. Since a re- peated normal form game is special case of stochastic games, so stochastic games. Moreover, we find that this gap can be even larger in stochastic games than that normal form games, because stochastic game sequential game, and agents' previous actions can influence the future state and therefore affect the future reward.
Ecer can be at least m=-2 in
Bate
Proposition 1. There exists a stochastic game that Erme
> m
Edec
Proof. Consider this example: is adversary. A, {1,... "....,m) .]} i 1..-n. Discount factor < 1. The team's reward a1 -1....a, an
(S1, initial state S1. Agents ... team,
function r(S,, a) 0, Va. r(S2, a) otherwise
Deterministic state transition function
SS -... an T((2,,, a) = S2
Game ends
T(S1, a) = SS otherwise
otherwise
In each state, the team's optimal correlated policy is to perform correlated action (1.-... .).... {m.... . with equal probability m Because the team perform any of these actions with probability less than 1. the adversary will perform that action, which will reduce team's reward.
m
However, if the team plays in a decentralized way, then each agent's optimal policy is to perform each actions with equal probability
We can prove that in this example, Edec Appendix A.2
Exer.
m 2n 10m-4(1- r) The detailed derivation can be found in
In fact, the gap between correlated and decentralized equilibrium in stochastic team mini-max game can be arbitrarily larger than normal form game, elaborated in the following proposition. The de- tailed derivation can be found in Appendix A.2
Proposition 2. V fixed E., there exists a stochastic game which
Bror Eaeec
(m-
Therefore, in robust MARL settings, CTDE algorithms may no longer achieve optimal performance. To achieve better robustness in adversarial settings, we need design some methods that can urge agents to learn correlated equilibrium.
4.2 LEARNING CORRELATED EQUILIBRIUM
In this work, we propose a novel approach to learn correlated equilibrium for robust MARL agents: using global random variable with mutual information.
4

---

Under review as a conference paper at ICLR 2021
This method is inspired by the idea of InfoGAN (Chen al. 2016). The idea is to add global random variable 2 as am extra input of the Q network, i.c. changing Qi(og, Qa) to Qi(Ou, 2,, ag)) Although the global random variable itself is meaningless, agents may learn to perform correlated equilibrium based on its value. The intuition is that, taking the example in Proposition [ we add a global random variable sampled from [0, 1] uniformly each timestep, then each agent might learn perform action when 2 E [0. ),,, perform action when Therefore, the overall policy is the optimal correlated policy of the team.
m ) and so
The following proposition is straight forward using the definition of correlated equilibrium. We also give a simple derivation in Appendix A.2
Proposition 3. For a fully observable, finite discrete action MARL environment, if all agents receive global continuous random variable then there exists deterministic policy pa 2) 8 x R + A e {I,...,, ,n} equivalent to the team's 's optimal correlated (stochastic) policy π (a1, ' anlss)
We now summarize two advantages of the proposed global random variable method for learning correlation equilibrium in robust MARL settings.
We can allow agents to perform correlated equilibrium while maximally preserving the property of CTDE. Since the global latent variable is independent of states, the only ele- ments that agents need to share is random number generator and a random seed. This can be done before the game starts, and agents can still perform "decentralized" policy during the game.
In MDP or fully observable fully cooperative correlated multi-agent MDP, one can prove that there must exist an optimal deterministic policy (Puterman) 2014), and therefore de- terministic policy algorithm can learn optimal policy. However, this property will not hold if the the multi-agent MDP is not fully cooperative, the optimal correlated policy in the example in Proposition rithms from learning optimal correlated policy directly. But when agents shares global random variable, deterministic policy approaches will be possible to learn a policy that is equivalent to the optimal correlated policy.
[ must be stochastic. This will prevent deterministic policy algo-
To avoid the ignorance of the global random variable's information for each agent, we propose to maximize the mutual information between the random variable and the agent's action I(2;; aa), and therefore agents must use the information from z. Also, to avoid solving the posterior P(2 directly, the variational lower bound derived as an approximate objective, like InfoGAN and Barber & Agakov 2003) (The subscript i,it is omitted):
[2, 2 Ez,a log [(2/a)
= Ez,a [loaa(= (x)).
(3)
p(=)
where q(2/a) is the variational approximation of P(zla).
Unlike InfoGAN using latent variables to model the information that is irrelevant of generator source (width, italic), reinforcement learning with latent variable, we prefer applying the latent variables model the correlated policy that is related to current observations. Thus, instead of simply utilizing ((z;; aa),, it's better to use the conditional mutual information [[2;; ao))).).
A similar variational lower bound can also be derived approximate this conditional mutual infor- mation (The subscript i,il is omitted):
log p(z a. o)
= Ez log q(2(a,0) p(z(o) + E., (aa.DDKk(((aaaao)llltett
[[z, alo) = Ez,a,o
p=200)
Q(=10,0)
2 Ez,a,
log p(210)
Eza. log ((za(z)), + (()) (Since p(z1o) = =p(=))
Since the entropy term (z) is constant, thus ignored, we can define C, Bo..a... [log (z ag o,)], and use Liot LRL+ +y as the overall loss function. In experiments, we follow InfoGAN's idea, configure variational approximation g(=l-) Gaussian
5

---

Under review as a conference paper at ICLR 2021
distribution, and apply neural network to output its mean and variance. Algorithm describes the overall training procedure.
Algorithm 1: Overall training procedure
Initialize replay buffer D for each epoch do
for sampling loop do
Obtain current 9t or O,
Sample 2N using some distribution.
Rollout action a, (using e-greedy or other method, with Qi(st, 24, ase) or
Qi(ou, 2t, Q.z)).
Select and rollout the mistaken action air,mis using Eq. 2 Let
At.mis (0iz... ... (0: 1,t, mit, 0i-1,t
Perform action ai,mis and get rt, 81+-1 or Oc+1-
Store transition (st, 2t, ar.mia: Tt, St+1) or (St, Or, 2 armis Tt,, S141, Oc+1) D.
end
for training loop do Sample minibatch M from replay buffer 2) Compute QMIX loss LTD-
Compute overall loss Liot LTD + Cr with L, described above.
Perform update step of QMIX with loss Lrot-
end
end
al. (2020) involve the idea of global random variable and mutual information. However, both of these works aim improve coordination in standard MARL tasks, while we focus on solving robust CMARL. To the best of our knowledge, we are the first demonstrate the importance of correlation this kind of robust MARL setting. So as a first-step work, we jujust apply the most straightforward method to demonstrate this important finding:
Comparison with existing works. Both Chen et al. (019) and Kim et al.
In our work, we formulate the global random variable as certain common knowledge be shared between agents, which is straightforward, and do not need to specify its distribution as Chen al. (2019) (p=(2) in Theorem 2.2 )
Kim al. (2020) use the mutual information between each pair agent's policy st)) encourage agent to coordinate. This is much complicate method. As a first-step work, we suggest to employ conditional mutual information T(2; als) (or T(2:alo) to encourage agents
[ St)
information from 2, and allow agents to learn the correlated method themselves. Chen et al. (2019) proposed similar mu- tual information settings, but since they formulated (z()), then the mutual information I (2; (2; s)) over joint distribution used.
Whether the more complicated correlation method can achieve even better performance in robust CMARL settings is still an open question.
EXPERIMENTS
We test method with QMIX algorithm in SMAC (Samvelyan al.) 2019) environment. We follow the experiment settings in QMIX (Rashid et al. 2018). The only difference is we train and evaluate with agents' mistaken action in Eq. 2 We test two robust settings like Eq. (I) including: 1) only one fixed agent will make mistakes; 2) all agents can randomly make mistakes, but in each timestep at most agent make mistakes.
Environment and Network Architecture We use the SMAC environment involving decen- tralised micromanagement scenarios, where each unit of the game controlled by an individual RI. agent. It is partially observable environment where each agent's observation is within cireu-

---

Under review as a conference paper at ICLR 2021
lar area around it. The action space is discrete. For further details of the environment, please refer to (Samvelyan et al. 2019).
In the training, we use 3-dimensional independent uniform distribution (O, 1) as global variable, and set MI loss coefficient A = 0.1. These are just hyperparameters, and we choose them based on performance.
We uses the same experiment settings of QMIX paper, including the architecture of the Q net- work, all the hyper- parameerre settings and the performance evaluation method (evaluate every 10K timesteps with 32 games).
For the glohal variable part, to add latent variable into the model, just extend the agent's obser- vation space. For the mutual information part (g(=(a,0), we uses an independent fully connected network with 2 32-unit-layers and ReLU activation function. The network takes agent's observation (without latent variable) and agent's 's action as input, and output a 6-dimensional vector with first 3 dimensional as mean and last dimensional as variance. During training. we gather all agents' observation and action data together and feed them into this network together.
Map and Agent Selection To evaluate our method, we select the maps and agents such that the original QMIX can achieve good performance in non-robust settings, but its policy is not robust with the selected agents.
By examining several maps about QMIX's performance and robustness, we choose four maps to evaluate our method: 8m, 2s3z, 3m, 3s5z. The detailed information can be found in Appendix [A.3 In all the evaluations, we choose E = 0.3 for 3m, and E = 0.5 for other 3 maps.
We evaluate the "random agent" case in all except 3s5z maps. Because in 3s5z map, normal policy will still obtain 68% of winning rate when a random agent has 50% probability to select its worst action. For the "fixed agent" case, since many agents in these maps are homogeneous, we only select some representative agents for evaluation. The selection criteria is mainly based on the robustness of the normal QMIX policy, and the detailed information can be found in Appendix A.3] We evaluate a total of 9 different agent settings.
Adversarial Testing Results In this part we compare 4 different settings: normal policy (train without adversarial but test with adversarial), vanilla adversarial training (baseline), adversarial training with global variable but without mutual information loss, adversarial training with global variable and mutual information loss. In the whole section, we use the following abbreviations to indicate these four settings: NP, VA, GV, GM.
We run each experiment times like QMIX does, and report the 25% and 75% as the upper and lower error bar. The mean winning rate during training is shown in Figure D We also plot the mean winning rate of the last few steps (in dotted line), and the performance of NP is plotted as black dotted line. The plots show that by adding global variable and mutual information loss, the performance of GM in most settings is better than VA and GV in average. The performance of GV is sometimes but not always better than VA. The reason might be that, without mutual information loss, agents will become easier to ignore global variable's information. Additional experimental results can be found in Appendix A.4
In adversarial settings, the testing variance is larger than normal settings. So in order to evaluate the performance more accurately, we select the high checkpoint of each run within the error bar, and test it with 1000 episodes. Also, we test the model with different percentages of adversarial rate. The results are depicted in Figure 2 and further show that training with global variable and mutual information can lead to a better adversarial testing performance. The raw data can be found in Appendix(A.5. [A.5 in T
Additional Baselines of Partially Observable Environments In Section 4 we use some exam- ples to demonstrate the importance of correlation in robust CMARL settings for fully observable environments, where correlation is not important in normal CMARL settings but plays an key role in robust CMARL settings.
However, since SMAC is a partially observable environment, and some existing works (c.g. Bern- stein et al. 2009)) pointed out that providing global random variable to all agents can actually
7

---

Under review as a conference paper at ICLR 2021
improve the performance in partially observable environments, So in order to demonstrate that cor- relation is more important in robust CMARL settings, we build another baseline the following.
We train policy in normal settings with global variable and mutual information loss (we abbreviation NG to indicate it), and then evaluate in adversarial settings. We compare the average performance improvement of NG/NP and GM/VA. The results be found in Table [ here and Table in Appendix [A.4
The results shows that, in most maps and agents, the performance improvement of NG/NP is less than GM/VA. This indicates that "correlation is more important in robust settings" to a certain extent: in the same evaluation settings, the improvement of normal settings (normal training) is less than robust settings (adversarial training).
2m Random
96
0.5
61
0.5%
152
322
2:00
200
402
15
100
200
500
Figure 1: Test winning rate v.s. training steps for different maps and adversarial agents (NP: nor- mal policy; VA: vanilla adversarial training; GV: adversarial training with global variable; GM: adversarial training with global variable and mutual information loss)
Ageni
Agont
Agent
Agont.
Afrersaaa
Figure 2: Adversarial testing results with different adversarial rates.
Table 1: Comparison of NG/NP and GM/VA: Mean test winning rate of different settings (NP: normal policy; NG: normal training with global variable and mutual information loss; VA: vanilla adversarial training: GM: adversarial training with global variable and mutual information loss)
Map and Agent
8m Agent 6
2x3z Agent 4
3m Random Agent
3s5z Agent 3
Adv Eval Rate
0.5
0.5
0.3
0.5
NP
14.4
5.7
32.8
9.0
NG
25.1
8.0
40.7
15.8
Gap
10.7
2.3
7.9
6.8
VA
50.2
53.9
44.4
45.7
GM
63.6
72.6
51.0
73.5
Gap
13.4
18.7
6.6
27.8
Apart from testing the adversarial agents, we also test scenarios that agent makes non-adversarial mistakes, i.c. agent plays random action with certain probability:
Random Testing Results
arg max, Q(s, a)
with prob. 1 We use the same above.
amii
Random choice from A
with prob. s
8

---

Under review as a conference paper at ICLR 2021
Agent
2532 Agent
Famdom Agent
3552 Agent
300
330
100
100
W1 1863
1000
kandinn AMP
Rarda' Fatte
Kar Kate
Fandon FM
Figure 3: Random testing results with different random rates.
The results are depicted in Figure 3 which show that training with global variable and mutual information can lead to a slightly better random testing performance, The raw data can be found in Appendix A.5 Table
CONCLUSION AND DISCUSSIONS
In this work, we focus robust CMARL when one agent in the team makes mistakes or even behave adversarially. We found that in team mini-max stochastic games, the performance of decentralized equilibrium be significantly worse than correlated equilibrium. Thus in order to achieve better robust CMARL performance, we propose a method which uses global variables with mutual infor- mation help agents learn correlated equilibrium. The experimental results show that this method can achieve better performance compared to vanilla adversarial training.
Robust CMARL is important research direction, as it will directly determine whether we can safely deploy CMARL into practical problems. To the best of our knowledge, this research direction is still in the early stage. In this work, we just provide simple method solve the robust CMARL problem. Several future directions can be explored:
The experiments of this work show that in some settings, the robust policy will slightly decrease the non-robust performance. So one possible future work is to balance the perfor- mance and robustness.
We only consider a weak worst mini-max problem. Whether it is possible to solve the real adversarial case (i.e. Eq. ( remains a future work.
Besides, we only focus on the robustness of one agent. In reality, perhaps all agents can have certain probability to make mistakes. Therefore, another possible future work is to consider this case.
Also, mentioned in the end of Section 4, we just the most straightforward correlation method to demonstrate the importance of correlation robust CMARL. Whether more complicated correlation methods can achieve even better performance in robust CMARL settings is still an open problem.
REFERENCES
Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovie, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. arXiv preprint arXiv:1907.13190 2019.
David Barber and Felix V Agakov. The im algorithm: variational approach information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Nicola Basilico, Andrea Celli, Giuseppe De Nittis, and Nicola Gatti. Team-maxmin equilibrium: efficiency bounds and algorithms. arXiv preprint arXiv:1611.066
2016.
Daniel S Bernstein, Christopher Amato, Eric A Hansen, and Shlomo Zilberstein. Policy iteration for decentralized control of markov decision processes. Journal of Artificial Intelligence Research, 34:89-132, 2009.

---

Under review as a conference paper at ICLR 2021
Andrea Celli and Nicola Gatti. Computational results for extensive-form adversarial team games. arXiv preprint 171.:9171..0630,, 2017.
Andrea Celli, Marco Ciccone, Raffaele Bongo, and Nicola Gatti. Coordination in adver- sarial sequential team games via multi-agent deep reinforcement learning. arXiv preprint arXiv:1912.07712 2019.
Liheng Chen, Hongyi Guo, Haifeng Zhang, Fei Fang, Yaoming Zhu, Ming Zhou, Weinan Zhang, Qing Wang, and Yong Yu. Signal instructed coordination in team competition. arXiv preprini arXiv:1909.04224 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016.
Gabriele Farina, Andrea Celli, Nicola Gatti, and Tuomas Sandholm. Ex ante coordination and collusion in zero-sum multi-player extensive-form games. Advances Neural Information Processing Systems, pp. 9638-66880 16-99644,22 2018.
Aidin Ferdowsi, Ursula Challita, Walid Saad, and Narayan Mandayam. Robust deep reinforce- ment learning for security and safety in autonomous vehicle systems. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 307-312. IEEE, 2018.
Mare Fischer, Matthew Mirman, and Martin Vechev. Online robustness training for deep reinforce- ment learning. arXiv preprint 191.v01911.00887, 2019.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. arXiv preprint arXiv:/705.089200 2017.
Zhaoyuan Gu, Zhenzhong Jia, and Howie Choset. Adversary a3e for robust reinforcement learning, 2018.
Sergiu Hart and Andreu Mas-Colell. A simple adaptive procedure leading to correlated equilibrium. Econometrica, 68(5):1 2000.
Sergiu Hart and Andreu Mas Mal-Colell. A reinforcement procedure leading correlated equilibrium. In Economics Essays, pp. 181-200. Springer, 2001.
Linfang Hou, Liang Pang, Xin Hong, Yanyan Lan, Zhiming Ma, and Dawei Yin. Robust rein- forcement learning with wasserstein constraint, 2020. URL httpt://///ppeeeeeeeee. net/ [oormmid=Hkee TEYDr.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. A maximum mutual informa- tion framework for multi-agent agget reinforcement learning. arXiv preprint arXiv:2006.0273: 2020.
Richard Klima, Daan Bloembergen, Michael Kaisers, and Karl Tuyls. Learning robust policies when losing control. In Adaptive and Learning Agents workshop AAMAS, 2018.
Martin Lauer and Martin Riedmiller. An algorithm for distributed reinforcement learning in coop- erative multi-agent systems. In Proceedings of the Seventeenth International Conference on Machine Learning. Citeseer, 2000.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4213-4220, 2019.
Shiau Hong Lim, Huan Xu, and Shie Mannor. Reinforcement learning in robust markov decision processes. In Advances in Neural Information Processing Systems, pp. 701-709, 2013.
Jieyu Lin, Kristina Dzeparoska, Sai Qian Zhang, Alberto Lcon-Garcia and Nicolas Papernot. On the robustness cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2003.03722, 2020.
10

---

Under review as a conference paper at ICLR 2021
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compeitiy environments. In Advances in neural infor- mation processing systems, pp. 6379-6390, 2017.
Xiaobai Ma, Katherine Driggs- CaagseCa and Mykel J Kochenderfer. Improved robustness and safety for autonomous vehicle control with adversarial reinforcement learning. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1665-1671. IEEE, 2018.
Daniel J Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki. Jost Tobias Springenberg. Tim- othy Mann, Todd Hester, and Martin Riedmiller. Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv: (00i:v:/000005///6 2019.
Abraham Neyman. Correlated equilibrium and potential games. International Journal of Game Theory, 26(2):223-23 227, 1997.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgen Systems, pp. 2040-2002. International Foun- dation for Autonomous Agents and Multiagent Systems, 2018.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In Proceedings the 34th International Conference on Machine Learning-Volle -Volume the
70, pp. 2817-2826. JMLR. org, 2017.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Aravind Rajeswaran, Sarvjeet Ghotra, Balaraman Ravindran, and Sergey Levine. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint 1rXiv:1610.0128. 2016.
Tabish Rashid. Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: / 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902::/900.04400 040444 2019.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self- elay. Science, 362(6419): 1140 1144, 2018.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learn- ing to factorize with transformation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv: 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinícius Flores Zam- baldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based team reward. In AAMAS, pp. 2085-2087, 2018.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and appli- cations in continuous control. arXiv preprint 1r9Xiv1901.000 09144, 2019.
Eugene Vinitsky, Yuging Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825 2020.
11

---

Under review as a conference paper at ICLR 2021
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782):350-33) 2019.
Bernhard von Stengel and Daphne Koller. Team- maxminn equilibria. Games and Economic Behavior, 21(1-2):309322 -21, 1997.
Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-183, 2013.
Youzhi Zhang and Bo An. Computing team-maxmin equilibria in zero-sum multiplayer extensive- form games. In AAAI, pp. 2318-2325, 2020.
A APPENDIX
A.1 VANILLA ADVERSARIAL TRAINING FRAMEWORK
Algorithm 2: Vanilla adversarial training framework
Initialize replay buffer D for each epoch do
for sampling loop do
Obtain current 9t or and rollout action a, (using e-greedy or other method).
Select and rollout the mistaken action aat,mis using Eq. 2 Let
At,mis (A1: ... 1.-1,f Oit,miss 0i-1,t" ..
Perform action miiii and get Tr, S141 or Or+1-
Store transition (st, ar, Tt, 8141) or (St, Oz, ar,mis Tt, 86+1, Or+1) to D.
end
for training loop do
Sample a minibatch MM from replay buffer D.
Compute QMIX loss CTD.
Perform update step of QMIX with loss Ltp-
end end
A.2 DERIVATION OF PROPOSITIONS
Detailed derivation of Proposition 1 In correlated equilibrium, every step in S, has + probability to transfer to S2, and every step in S₂ has m probability to get reward. If current state S2, then the expected reward is:
1
1 +
1 (I-
(1 (1-2) ((..))
1
7
?
1+
m
m
m
m
m
1
1+
((
_
7)
(11- m ))>)
7) +
2
m
m
1 def k
((1 1
?) m
m
So the overall expected reward of the correlated equilibrium is:
(1-1)
1
L
1
(1- 1 m
x (.... (..)))
Epor
7k +
.k
7k
+
7
7 +
m
1
1
1
7k
1
?
)rr)
1 ((1
7) m
1
((1
1 7)
m
12

---

Under review as a conference paper at ICLR 2021
In decentralized equilibrium, every step in S₁ has m probability to transfer to S2, and every step in S2 has
probability get reward. Similarly:
7
1
dde ((1 m
mn-
Therefore:
Berr
m=- -1) + m(1 7) +
m m= -7) m(1 7) + my
mm mnn-11p 7)2
Edec
Detailed derivation of Proposition Consider the step game with is states S Sk.. Agents and action spaces are the same as Proposition The discount factor is 7. r(s,a) 0,
{
That is, the team can only receive reward in the last step, which is common case in RL environments. The deterministic state tran-
{S1,... Sk-1), Va: r(Sk, a)
0 otherwise
a1 02
an
sition function is a) S
14+1
a2
On i = 1... k, where Sk+1 denotes the
otherwise
game ending.
Let's prove that Beork
2 (n-n2)) 7²
Proposition shows that Bean2 Edec.2
22(n-2)(1 - 7)2 Now suppose
Beard 2 m mn(n=2)(1 - 7)", by Baec,
using similar derivation in Proposition
[ we can get
1
BBoo
Boorit1 ((1 11) 7)
7 BBdec,
Eetecit1 1 ((1 2-1) )
mn
So,
Ecorit+1
m" mn
(11 - -7) + Ecors m n- (1 - 7) Ecor, m(1 7) + my Edec,
Edec,i11
m(1 7) + Edec,
= =m= n- - Bcars
2 m (n - 7)1
Edec,
Given state since the action space is discrete, suppose each agent has m actions, then we can list all possible joint actions ai ... a, a sequence {m Consider the optimal correlated policy x ((a,.... s). Let pt = <axa k = 1......
Derivation of Proposition 3
Let Z be the value range of z. Since z is continuous random variable, we can divide the support set of Z into m" disjoint subset {2x 1k=1 which satisfies P(z Zz) Ps.
Next, let each agent performs the following deterministic policy:
[+(8,2) z) (at); if E Za
That is, agent performs action (a,); if E Zg. All agents will perform joint action if 2 E Z and the team will perform joint action a with probability P(2 E 7x) = Pk- This is exactly the optimal correlated policy.
A.3 SMAC MAP AND AGENT SELECTION
In SMAC environment, not all maps are suitable for evaluating out method. Because:
In some map(s), QMIX's performance is not good in non-robust settings (e.g. 3s5z_vs 3366z 6h vs 8z, 27m vs 30m, comridor). These results can be found in SMAC paper (Samvelyan et al. 2019)
13

---

Under review as a conference paper at ICLR 2021
8m
8m
253z
2s3z
3m
Map and Agent
Agent 4
Random Agent
Agent 2 0.5
Random Agent
Agent 2
Adv Eval Rate
0.5
0.5
0.3
NP
1.9
36.9
5.7
45.6
17.6
NG
9.9
50.9
5.6
59.6
22.9
Gap
8.0
14.0
-0.1
14.0
5.3
VA
53.8
59.4
78.5
60.9
32.3
GM
68.2
79.6
90.5
67.5
42.6
Gap
14.4
20.2
12.0
6.6
10.3
Table 2: Additional comparison of NG/NP and GM/VA
In some map(s), the policy trained by QMI in non-robust settings is already robust with most agents (e.g. www.aa_aannliinne 3s 3s_vs__33_ (3,4,555) 1c3s5z except agent Colossi, MMM).
some hard map(s), may be difficult for agents to achieve robust policy, (e.g. 8m vs 9m or 10m_vs_1 Im. our team already has less agents than the enemy; 1c3s5z agent Colossi, Colossi a large agent, thus it makes mistakes, other small agents are hard to make up.)
Since big map may more likely be robust when only one agent makes mistakes, we decide to focus on small maps. By examining several maps about QMIX's performance and robustness, choose four maps to evaluate our method: Sm, 2s3z, 3m, 3s5z.
Since many agents in these maps are homogeneous, only select some representative agents to evaluate.
3m map: All agents are homogeneous. We select the agent that is the least robust one in normal QMIX policy: agent (The test winning rates of normal QMIX policy with 1000 episodes are 18.9%, 21.2%, 17.6% respectively, when agent 0, 1, has 30% probability to select the worst action.)
8m map: All agents are homogeneous. Since this map has more agents than 3m, we select the most robust agent and the least robust agent in normal QMIX policy: agent and agent 6. (The test winning rate of normal QMIX policy with 1000 episodes are 2.3%, 3,4%, 5.4%, 10.3%, 1.9%, 7.3%, 14.4%, 7.1% 1% respectively, when agent 1, 2, 4, 5, has 50% probability to select the worst action.)
3s5z map: During training we found that this map is difficult to train. We have to train more steps to make it converge, and the training process is slow. Therefore only eval- uate the least robust agent in normal QMIX policy: agent 3. (The test winning rate of normal QMIX policy with 1000 episodes are 27.5%, 31.5%, 80.5%, 9%, 9.5%, 14.5%, 18%, 22.5% respectively, when agent 0, 1, 2, 3, 4, 5, 6, has 50% probability to select the worst action.)
2s3z map: During training we found that this map is a bit difficult to train. We want to only evaluate the least robust agent in normal QMIX policy. But in this map, agent 2 and agent 4 are almost the same robust, so we evaluate both of them. (The test winning rate of normal QMIX policy with 1000 episodes are 6.1%, 8.1% 5.7% 6.4%, 5.7% respectively. when agent 0, 1, 2, 3, 4 has 50% probability to select the worst action.)
A.4 ADDITIONAL EXPERIMENT RESULTS
In this section, we present the additional experimental results, including the mean winning rate during training (Figure 4a), the adversarial testing results (Figure 4b). the comparison of NG/NP and GM/VA (Table and the random testing results (Figure 4c
The results shows that, the performance of GM is better than VA and GV in most settings, except for 2s3z's 's random agent, which GV better than VA but GM similar to GV. Also, in most settings, the performance improvment of NG/NP is less than GM/VA.
14

---

Under review as a conference paper at ICLR 2021
Agent
Agent
100
0.7
Thoe Wirning Fare 1000 Eooddde (75)
0.6
20
50
0.5
0.4
0.3
40-
70-
0.2
0.1
20-
0.0
50
100
150
220
0.0
0.1
0.7
0.3
0.0
6.5
0.0
0.1
0.7
0.3
0.0
0.5
Timandapo (x30000)
Adversatial Rata
Nandom Rutt
8m Random Agent
3m Random Agent
8m Random Agent
0.8
Text Kltrning Rare oP3005 Eprortes (14
0.7
20-
20-
60
0.3
0.3
VA
20
GV
20
CV
0.1
50
1.00
150
200
0.1
0.2
6.5
0.0
6.3
0.4
(c10000)
Advensantal Rale
Rendom
2:37 Agent 2
263r Agent
2537 Agent
GVV
Test Wirnnng Rate of 1000 Ep-ettees 1h4
60-
80
VA
VA
20
CV
CV
GM
Ter
GM
0.2
0.2
253z Random Agent
253z Random Agent
253e Random Agent
1:0
100
506
CVV
GM
Teah Wimting Fate or 10000 E2000-ppor-ee dee
0.6
0.5-
60-
20-
0.2
0.1
GM
GM
0.0
NP
30
Timestops (110000)
Agent
Agent
Agent
1:0
100
0.5
Terk Miteting Park of 1200 Eptoiees 1h0
GVV
GV
GM
GM
GM
0.4
0.3
0.-
40
70-
1.00
150
0.1
0.3
0.2
Timautepa (c20000)
Adversarial Rate
Randum
(a) Training Curves
(b) Adversarial Testing Results
(c) Random Testing Results
Figure 4: Additional Testing Results
15

---

Under review as a conference paper at ICLR 2021
Table Adversarial Testing Results: Mean test winning rate
0%
10%
20%
30%
40%
50%
NP
87.6
72.1
47.1
22.0
8.6
1.9
VA
96.4
94.6
89.5
78.8
65.7
53.8
Agent 4
GV
97.9
96.7
93.3
85.0
71.3
59.6
GM
99.3
98.7
96.7
90.2
79.1
68.2
NP
87.6
79.7
67.0
53.3
31.2
14.4
VA
95.4
93.1
87.2
74.8
59.4
50.2
8m Map
Agent 6
GV
92.7
90.8
84.6
74.9
62.7
52.1
GM
98.0
96.0
91.5
84.0
73.7
63.6
NP
87.6
80.4
68.7
60.1
47.0
36.9
VA
90.2
87.9
82.8
69.4
69.6
59.4
Random Agent
GV
94.6
94.1
90.9
85.3
77.9
67.7
GM
99.4
98.7
97.1
93.1
88.8
79.6
NP
97.5
92.7
75.0
50.8
21.4
5.7
VA
82.8
85.1
85.0
86.6
82.5
78.5
Agent
GV
89.4
92.0
92.9
91.4
86.0
79.4
GM
97.6
98.0
97.3
96.0
93.9
90.5
NP
97.5
85.7
65.0
39.1
20.9
5.7
VA
68.6
70.7
72.7
72.2
64.9
53.9
2s3z Map
Agent 4
GV
92.0
91.4
88.2
81.9
71.3
58.5
GM
93.4
93.0
92.5
90.2
81.6
72.6
NP
97.5
94.7
87.9
77.9
63.5
45.6
VA
89.9
90.0
88.1
82.1
74.5
60.9
Random Agent
GV
96.8
97.4
94.1
89.4
80.6
69.6
GM
91.6
93.1
91.4
87.5
81.1
67.5
NP
94.4
90.7
78.7
56.5
28.6
9.0
VA
78.8
76.7
71.6
66.2
56.8
45.7
3s5z Map
Agent 3
GV 79.4
67.9
68.3
66.2
60.6
51.3
GM
88.9
88.1
87.2
83.4
78.9
73.5
NP
96.1
78.2
40.8
17.6
VA
95.2
81.0
58.8
32.3
Agent
GV
90.2
77.0
57.2
34.8
GM
96.7
86.1
64.7
42.6
3m Map
NP
96.1
80.6
56.3
32.8
VA
96.1
86.0
68.0
44.4
Random Agent
GV
97.1
86.1
67.2
44.2
GM
97.8
90.6
72.6
51.0
A.5 SMAC EVALUATION TABLES
The mean testing winning rates of adversarial testing results are shown in Table The mean testing winning rates of random testing results are shown Table 4
16

---

Under review as a conference paper at ICLR 2021
Table 4: Random Testing Results: Mean test winning rate
0%
10%
20%
30%
40%
50%
NP
87.6
85.3
77.5
65.8
54.4
43.9
VA
96.4
96.4
94.5
93.3
89.8
84.7
Agent
GV
97.9
97.4
97.6
96.3
94.8
93.1
GM
99.4
99.4
98.9
98.3
97.1
94.8
NP
87.6
87.2
85.5
83.8
83.1
75.8
VA
95.8
94.5
93.5
91.2
87.5
83.9
8m Map
Agent 6
GV
92.7
93.0
91.7
90.2
87.9
81.5
GM
98.0
96.9
96.4
94.7
93.8
90.5
NP
87.6
83.3
81.7
80.0
74.0
71.0
VA
90.2
89.6
88.8
87.2
84.7
83.6
Random Agent
GV
94.6
94.8
94.1
94.1
93.0
91.3
GM
99.4
99.2
98.9
98.5
97.9
97.0
NP
97.5
96.0
89.3
78.4
58.6
40.5
VA
82.8
85.5
84.2
86.1
86.0
85.8
Agent 2
GV
89.4
90.6
91.8
90.1
92.3
90.5
GM
97.6
97.4
97.7
97.6
97.4
96.7
NP
97.5
94.4
86.1
78.4
66.3
52.6
VA 68.6
68.3
68.6
68.6
68.9
69.7
2s3z Map
Agent 4
GV 92.0
85.1
86.1
81.8
79.5
76.7
GM 93.4
92.8
92.9
92.6
91.7
91.1
NP
97.5
96.1
93.6
90.5
86.4
81.6
VA
89.9
89.7
89.4
88.3
85.5
82.5
Random Agent
GV
96.8
97.3
95.8
94.6
93.0
89.8
GM
91.6
93.0
91.9
90.8
89.8
87.9
NP
94.4
92.9
86.0
79.6
68.5
57.7
VA
78.8
76.2
76.3
73.6
69.4
65.7
3s5z Map
Agent 3
GV
79.4
66.1
64.9
65.6
63.3
62.4
GM
88.9
89.7
88.0
87.6
86.3
83.8
NP
96.1
90.3
79.7
65.0
VA
95.2
91.0
81.9
74.7
Agent 2
GV 90.2
85.9
80.5
71.5
GM
96.7
94.3
88.2
79.1
3m Map
NP
96.1
92.2
84.6
74.1
VA
96.1
92.3
87.1
80.4
Random Agent
GV
97.1
94.1
87.6
81.6
GM
97.8
96.1
91.0
87.7
17


Reviews:
[{"review_id": "oWZCThCWmM", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper addresses the issue of robustness in cooperative multi-agent RL setups, where the inclusion at test time of an agent that makes error or is even adversarial can drastically decrease performance. The main idea is to compute a correlated equilibrium, by allowing all agents policies to depend on a common signal. To encourage the actions of the agents to correlate, they add a mutual information loss (i.e. a retrodiction that encourages the global latent to be predictable given the action taken). \n\nHigh level review:\nThe paper introduces a natural solution to the studied problem. However, it suffers from a number of flaws that precludes acceptance at ICLR in my mind.\n\n- The proposed method is extremely similar to previous work (Chen et. al and Kim et. al). In particular Chen et al. also suggest something essentially equivalent to the Infomax loss (see equation 5). Their particular definition of the distribution is solely needed for the theoretical proof that correlated policies can be obtained by having a global latent variable (theorem 2.2 in their paper, which is essentially equivalent to proposition 3 in this paper; both theorems are fairly standard use of latent variables to represent correlated distributions). I fail to appreciate the genuine contribution of this paper over Chen et. al. Note that neither approaches (Chen et. al, Kim et. al) are used as baselines in this paper, while they clearly should be. \n\n- The writing is poor and can often be very confusing.\n\n- The theoretical section does not connect well to the rest of the paper ; section 3 is almost entirely disconnected from what the authors suggest doing (equation 4 is effectively not solved at all, so why introduce this hierarchy of relaxation if the proposed method only poorly relates to it? In the end, the authors train instead an agent in a regular MARL setup with a global variable, and test it in the robustness setting; this is a heuristic that entirely replaces the maximin approach of equation 4 as far as I can understand). Is the counterexample in section 4 novel, or folklore? (It feels very familiar). I would not couch it in terms of a proposition (which is oddly informal - 'can be much larger' - for a theoretical statement), and instead study it more carefully as an intuitive example for why correlation is needed.  A team could play in a decentralized way and obtain very good reward by using the (1,1,1,1,1) action with probability one (which does not require correlation). The issue is that this strategy is not robust to adversaries. This should be explained in more detail.\n\nReviewer familiarity with the work: I am moderately familiar with the MARL literature (MADDPG, COMA; I worked on on MARL paper), and with game theory (correlated equilibrium). The precise setting of the paper I was not familiar with (robustness with respect to switching from cooperative to mix-cooperative; QMIX algorithm). I was not aware of the work of Kim et al.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jTpXZpimtgU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a new method for making cooperative MARL algorithms more robust to teammate mistakes. The approach learns correlated equilibria that depend on a global random variable. The setting is discussed with some theoretical results and experiments are provided to compare versions of their approach in Starcraft (SMAC). \n\nSafe and robust MARL is an important topic. There may be settings in which agents make mistakes or can be in dangerous situations. Making agent policies robust to such circumstances will be necessary in real-world scenarios. Also, the idea of using mutual information to improve agent performance is promising. Sometimes, such information is available (or can be available cheaply) so why not take advantage of it?\n\nThere are some questions about the exact robust MARL setting that is approached in this paper. In particular, the paper assumes the setting where only one agent makes a mistake at each timestep and that mistake-making agent is adversarial (i.e., chooses the worst possible action for the team). The specific setting in Eq 5 has the agent choose the minimizing action with a small probability and the maximizing action with the remaining probability. When is such a setting realistic? If all the agents are assumed to be trained centrally with all the agent policies known, why would an agent become adversarial in this way? The setting should be motivated to better understand how realistic and general it is. \n\nThe theoretical results are extensions of the normal form case to the stochastic game case. The theory appears novel and as a result useful, but not a major contribution. \n\nThe algorithm is somewhat straightforward, but there are novel aspects and the idea is promising. As the paper points out, other approaches have used global random variables, but not in this robust context and not with the form of mutual information that is added to improve the use of the global signal. \n\nThe experimental results show the method works well, but they are not sufficiently comprehensive. For example, how does it perform if *no* agents make mistakes? This appears to be shown in Figure 2, but the results aren't clear. It seems like sometimes the proposed methods do better than the normal policy. This shouldn't be possible in the fully observable case, but may be due to the fact that SMAC is partially observable. Using policies that are dependent on a global random variable can improve performance in partially observable settings. This is known (e.g., see the paper below), but never discussed in the paper. SMAC domains are partially observable so it isn't clear how much that is a factor. Because the paper and theory are described from a fully observable perspective, the paper should also have results for the fully observable case. The SMAC results are useful, but the proposed method should also be discussed in terms of partial observability. \n\nBernstein, Daniel S., et al. \"Policy iteration for decentralized control of Markov decision processes.\" Journal of Artificial Intelligence Research 34 (2009): 89-132.\n\nThe writing is generally understandable, but it should be polished. There are some issues that are not clear such as the fact that a policy in the partially observable case (defined in 3.1) could do better by conditioning on the history. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "b4sYwbUpRQH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Overview\n\nThis paper considers robustness in the context of multi-agent setting with decentralized execution, in the scenario where one agent may behave sub-optimally (either by being adversarial or simply taking random actions) on a fraction of the time-steps.\n\nThe solution relies on attempting to find a correlated equilibrium by providing agents with a common source of randomness.\n\nThe paper then exposes some experimental results on Starcraft. The experiments seem sound and prove the point. However, some details seem to be missing, potentially raising concerns for reproducibility, and the exposition could potentially be improved for better clarity.\nDespite these short-comings, which should be easily addressable, the results are convincing and the simplicity of the approach warrants its acceptance.\n\n## Method\n\nThe paper starts by theoretically justifying the need of finding a correlated equilibrium, and thus do a good job at motivating the approach.\n\nThe method relies on augmenting the agents by providing access to a shared source of entropy, in the form of a shared low dimensional random variable $z$.\n\nTo encourage the network to rely on this $z$, the authors add a variational-lower bound type of loss that experimentally helps the performance.\n\n## Experiments\n\nThe authors carry out experiments on some Starcraft environments of the SMAC suite.\nThey build on an established algorithm for these environments, namely QMIX, and provide a reasonable \"vanilla\" adversarial training scheme as baseline.\n\nGlobally, the full method using the variational lower bound loss seems to be performing consistently better than both the adversarial training baseline and the version of the training with only access to z but no variational lower bound loss.\n\nThe reporting of the results could probably be improved:\n* Figure 2 and 3 use different vertical axis range, making direct comparison difficult\n* Training curves are indeed interesting to give an intuition of the general behavior, but I think it would be valuable to provide in the main text an aggregated table summarizing the results in table 1 and table 2 from the appendix, with proper confidence intervals (something like an average performance over maps and agents). This would better convince the reader that the better performance of QMIX-GM is a general trend in all the possible settings.\n\n\nThe paper specifies that the method uses a 3-dimensional uniform distribution in $[0,1]^3$ for $z$, however the design decisions behind such a choice are not discussed or experimentally backed. In particular, it is not clear if the dimensionality of this variable plays any role in the final performance, and if/how practitioners may have to tune this knob when applying the method to their own problem.\n\nFinally, some details are lacking in the architecture used:\n* The text says that the latent code $z$ is added as input to the Q network, but doesn't specify the architectural changes that are made to made this possible.\n* As for the variational approximation, the text reads \"[we] apply a neural network to output its mean and variance\". However, not details are provided on this network, and in particular whether it shares parameters with the aforementioned Q network\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "76pAhboo6Z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers a case in the cooperative multiagent reinforcement learning where one single agent can behave adversarially. It claims that the performance of the whole MARL system  would deteriorate significantly  and demonstrate this phenomenon both theoretically and empirically. Then the author proposes a solution, i.e. , solving the correlated equilibrium in the game.  To seek  the correlated equilibrium, the author introduces a global random variable z and adds a mutual information regularization term.  Indeed this idea is easy to follow since it directly follows the definition of the correlated equilibrium. At last, it tests the algorithm in the SMAC environment and compares it with QMIX algorithm.\n\n\n\n\nQuestion 1: How do you get the global random variable z in Q(o_i, z,a_i) in the execution. Is it sampled from the posterior distribution q(z|o,a)? \n\nQuestion 2: my main concern is about the novelty of this paper. Can you discuss the difference between your method and MAVEN[1], where the author also introduces the latent variable z and uses the mutual information on z and observed trajectory.  The idea seems to be almost the same especially when it is applied on QMIX.\n\nQuestion 3: The author should include more baselines. Here I just see the result from QMIX. The author should test this idea on other classical methods, e.g., COMA and many others.\n\n[1] MAVEN: Multi-Agent Variational Exploration\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-6PC_OJtCzY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nRobustness in the multi-agent setting is a nuanced concept, as (a subset) of agents can act adversarially, while the non-adversarial agents can be trained to be more robust. The authors propose to solve a max-min problem, in which some agents are optimizing their rewards taking into account some agents maybe acting suboptimally/adversarially. \n\nThe authors implement learning using QMIX and evaluate on SMAC. They train a model p(z | a, o) for a global latent variable (using a variational lower bound on mutual info between z and a) that each robust agent conditions its policy on. \n\nThe authors claim this method can yield significant performance improvements over vanilla adversarial training.\n\n\nStrengths\n\n- Robustness is not well explored in the multi-agent setting, and using a correlated equilibrium seems like an interesting way to model coordinated robust training for (a subset) of MARL agents.\n\nWeaknesses\n\nThroughout, the writing is not very clear. \n\n- The notation in (3) and (4) is sloppy: which indices are \"team\" and which ones are \"mis\"? \n- Sect 4.1 talks about differences between correlated eq and decentralized equilibria, but the authors never formally define what the difference is between \"cooperative agents\" (which they claim emerges under centralized training, decentralzied execution), and a correlated equilibrium (in which some agents cooperate, presumably, but others don't). I assume the authors are getting at the normal definition of correlated eq: there is some (global) random variable that the policies are conditioned on, but the text is very unclear about this.\n- The propositions 1 - 2 seem out of place without a formal defn of correlated eq (see above).\n- The text around prop 3 never mentions that agents presumably now learn a policy pi(a | s, z). This should be clarified.\n- Eq 7 is missing indices i, or should clarify the \"a\" refers to a joint action, etc. \n\nConceptual issue:\n\n- Adversarial agents are defined as taking actions that minimize their *own* Q-value (i.e., agent i executes argmin_i Q_i). But shouldn't the worst mistake be the action that minimizes other agents' value? It seems this definition of \"adversarial\" agent is rather weak, and counter to what is defined in Eq (1 - 3), where adversarial agents want to minimize other agents' value as well? This is where the notation again causes confusion.\n\nThe experimental results are too thin to conclude the authors' method is indeed effective. \n\n- What is the significance (if any) of having different agents (6, 4, etc) as the \"adversarial\" ones in Figures 1, 3? \n- The authors do not explain what the structure of SMAC is (the experiment environment). \n- The authors should visualize the distribution of actions executed by the robust/adversarial agents to give some intuition of the qualitative differences. \n\nQuestions\n\n- What is a \"flat maximum\" (above Eq 3)?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the interesting area of cooperative multi-agent learning and presents a promising method to make MAL robust to mistakes of teammates, while learning correlated equilibria. Reviewers find the presented setting and theoretical contributions limited and the experiments not extensive enough; also some technical details about the architecture are lacking, and the notation and writing can be substantially improved upon. As such the paper does not seem ready for publication at this stage.
Decision: Reject