PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration
Authors: Pengyi Li
Hongyao Tang
Tianpei Yang
Xiaotian Hao
Sang Tong
YAN ZHENG
Jianye HAO
Matthew E. Taylor
Jinyi Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-Agent Reinforcement Learning
Collaboration

Abstract: Learning to collaborate is critical in multi-agent reinforcement learning (MARL). A branch of previous works proposes to promote collaboration by maximizing the correlation of agents’ behaviors, which is typically characterised by mutual information (MI) in different forms. However, simply maximizing the MI of agents’ behaviors cannot guarantee achieving better collaboration because suboptimal collaboration can also lead to high MI. In this paper, we first propose a new collaboration criterion to evaluate collaboration from three perspectives, which arrives at a form of the mutual information between global state and joint policy. This bypasses the introduction of explicit additional input of policies and mitigates the scalability issue meanwhile. Moreover, to better leverage MI-based collaboration signals, we propose a novel MARL framework, called Progressive Mutual Information Collaboration (PMIC) which contains two main components. The first component is Dual Progressive Collaboration Buffer (DPCB) which separately stores superior and inferior trajectories in a progressive manner. The second component is Dual Mutual Information Estimator (DMIE), including two neural estimators of our new designed MI based on separate samples in DPCB. We then make use of the neural MI estimates to improve agents' policies: to maximize the MI lower bound associated with superior collaboration to facilitate better collaboration and to minimize the MI upper bound associated with inferior collaboration to avoid falling into local optimal. PMIC is general and can be combined with existing MARL algorithms. Experiments on a wide range of MARL benchmarks show the superior performance of PMIC compared with other MARL algorithms.

Under review as a conference paper at ICLR 2022
PMIC: IMPROVING MULTI-AGENT REINFORCEMENT LEARNING WITH PROGRESSIVE MUTUAL INFORMA- TION COLLABORATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Learning collaborate is critical in multi-agent reinforcement learning (MARL). A branch of previous works proposes to promote collaboration by maximizing the correlation of agents' behaviors, which is typically characterised by mutual infor- mation (MI) in different forms. However, simply maximizing the MI of agents' behaviors cannot guarantee achieving better collaboration because suboptimal col- laboration can also lead to high MI. In this paper, first propose a collab- oration criterion to evaluate collaboration from three perspectives, which arrives at a form of the mutual information between global state and joint policy. This bypasses the introduction of explicit additional input of policies and mitigates the scalability issue meanwhile. Moreover, to better leverage MI-based collaboration signals, we propose a novel MARL framework, called Progressive Mutual Infor- mation Collaboration (PMIC) which contains two main components. The first component is Dual Progressive Collaboration Buffer (DPCB) which separately stores superior and inferior trajectories in progressive manner. The second com- ponent is Dual Mutual Information Estimator (DMIE), including two neural es- timators of our new designed MI based on separate samples in DPCB. We then make use of the neural MI estimates to improve agents' policies: to maximize the MI lower bound associated with superior collaboration to facilitate better collabo- ration and to minimize the MI upper bound associated with inferior collaboration to avoid falling into local optimal. PMIC is general and can be combined with existing MARL algorithms. Experiments show the superior performance of PMIC compared with other MARI. algorithms.
wide range of MARL benchmarks
INTRODUCTION
With the potential to solve complex real-world problems, Multi-Agent Reinforcement Learning (MARL) has attracted much attention in recent years (Rashid et al., 2018; & Sha 2019 Wang
ct al. 2019b) and been applied to many practical domains like Game AI (Peng et al. 2017)) robotics control (Matignon et al. 2012), transportation (Li et al. 2019). However, efficiently achieving col- laboration and learning optimal policies still remains challenging in MARL (Liu et al.) 2020 Wen
cet al. 2019). Prior efforts on improving collaboration of MARL agents mainly rely on efficient com-
munication Eccles et al. 2019 Das et al. [2019] Kim et al. 2019 2021), or resort to value function
factorisation (Sunehag et al. 2017: Rashid et al. 2018 Son et al. 2019)
A popular paradigm of cooperative MARL is Centralized Training, Decentralized Execution (CTDE) (Rashid et al., (2018: Sunchag et al., 2017). During centralized training, agents are granted access to other agents' information and possibly the global state, while during decentralized exe- cution, agents make decisions independently based on their individual policies. However, although global information is incorporated during centralized training of CTDE, guiding the decentralized policies of multiple agents only by reward signals is often inefficient, especially when the reward signal is stochastic or sparse. This requires additional mechanisms to facilitate multi-agent collabo- ration (Roy et al. 2019 Kim et al., 2020). Later, another branch of works which complementary to the above two branches focuses leveraging the correlation influence among agents for better collaboration (Jaques et al. 2018: 2019; Xie et al., 2020; Liu et al., 2020 Merhej & Chetouani] 2021).Tha The intuition behind this is that if agents make decisions with the awareness of the influence

---

Under review as a conference paper at ICLR 2022
or behaviors of other agents, the non-stationary problem could be mitigated, thus agents are more likely to achieve collaboration.
Motivated by this, recently, several works (Chen et al. 2019 Mahajan et al. 2019 Kim et al. 2020) propose to maximize the correlation of agents' behaviors to promote collaboration. The correla- tion is commonly quantified by the mutual information (MI) of agents' behaviors. Signal Instructed Coordination (SIC) (Chen et af.] 2019) incorporates a shared latent variable = (sampled from a prede- fined distribution) into agents' policies as a signal of collaboration. During training, the MI between latent variable 2 and agents' joint policy is maximized. By taking = as input, the correlation of each agent's 's decision to the joint policy is enhanced. Similarly, Multi-agent Variational Exploration (Maven) (Mahajan et al. (2019) establishes an encoder network to encode the initial global state into the latent variable z instead of sampling from a predefined distribution. Then Maven maximizes the MI between z and the resulting trajectory of all agents. However, one common drawback of both methods is, the shared variable required during decentralized execution violates the CTDE paradigm. This makes algorithms fail in some real-world deployment scenarios where global com- munication is not available. Apart from the above methods, VM3-AC (Kim et al. 2020 measures the MI between any two agents' policies to capture the correlation of agents' behaviors, where is also introduced to guarantee the MI of any two policies is positive. Through maximizing the MI of any two agents" policies, VM3-AC reduces the uncertainty of agents' policies and thus facilitates multi-agent collaboration. However, VM3-AC also violates the CTDE paradigm due to the require- ment of the shared variable z. Furthermore. VM3-AC calculates the MI between any two agents, which can be computationally infeasible with the increase of the number of agents. Another critical thing which is neglected by previous works is that, although agents with high degree of collaboration usually generate highly correlated behaviors, this does not necessarily results in high rewards. Thus, simply maximizing the MI of agents' behaviors cannot guarantee desired collaboration. Worse still, maximizing MI exacerbates the problem: the agent can easily overfit its strategy to the behaviours of other agents (Zhang et al. 2019 Lanctot et al. 2017). An example is shown in Figure I where agents are easily falling into sub-optimal collaboration.
In this paper, we propose a new collaboration criterion to evaluate the collaboration of multiple agents in terms of the joint policy, individual policy, and other agents' policies. Based on the crite- rion, we propose a new form of MI between the global state and joint policy to facilitate collabo- ration which is free of the reliance of explicit additional input as introduced in previous works and addresses the scalability issue. Moreover, to avoid falling into sub-optimal collaboration, we pro- pose a novel framework, called Progressive Mulual Information Collaboration (PMIC), which con- tains two main components. The first component is Dual Progressive Collaboration Busfer (DPCB) which includes a positive and a negative buffer to dynamically keep the superior and inferior col- laboration separately. The second component is Dual Mutual Information Estimator (DMIE) which has two MI neural estimators to estimate the MI of global state and joint action based on DPCB. Specifically, one estimator is trained on the positive buffer to provide the lower bound of MI, the other is trained on the negative buffer to provide the upper bound of MI. By maximizing the lower bound and minimizing the upper bound of MI, the agents could progressively break the current sub- optimal collaboration and learn towards better performance which promotes an efficient and stable learning process. PMIC is general and can be easily combined with existing MARL algorithms. Our experiments show that PMIC significantly accelcrates existing MARL algorithmss, outperforms other related algorithms wide range of MARL benchmarks.
2 PRELIMINARIES
We consider a fully cooperative multi- aggent task where a team of agents are situated in a stochas- tic, partially observable environment. it can be modeled as a decentralised partially observ- able Markov decision process (Dec-POMDP) (Oliehoek & Amato (016), defined by a tuple (I,S, T,T,T.T... R, Here I {1,..., N} denotes the set of N agents. In Dec-POMDP, the full state of the environment 8, E cannot be observed by agents at each time step t. Each agent i E I can only observe its individual observation of determined by observation function O(st, i), each agent é uses a stochastic policy ग. to choose actions uj ~ TT(-lo{), yielding the joint action u. After executing de in state st, the environment transits to the next state 8141
u, = ( = Nu according to transition function T(s,, u) and agents receive a common reward r, from R(s;, u.), with a discount factor 7. In cooperative MARL,, the collaborative team aims to find joint pol-
2

---

Under review as a conference paper at ICLR 2022
18
Pattern
Brastre Dure
BI0
Agent
Pattern
Dascino
Reward
VM3
B
Agent
Pattern
600
pobera
potors
Tima Sruss (16))
(a) Motivating Example
(b) Count of patterns
(c) Training curve
Figure 1: (a) A motivating environment. Effective patterns are patterns that can get positive re- wards. The optimal pattern here is to achieve target A coordinately, while achieving other targets leads to sub-optimal patterns. (b) The count of different collaboration patterns learned by different algorithms, evaluated over 10k (Ik x 10 seeds) episodes. (c) The training curves of episodic average rewards over 10 seeds with 95% confidence regions for different algorithms.
icy π (u) = II (u that maximizes the total expected discounted cumulative reward E, 1 r're] where y the discount factor.
3 PROGRESSIVE MUTUAL INFORMATION COLLABORATION
3.1 MOTIVATION
In this section, we show the motivation of our work to explain why blindly maximizing cor- relation of agents' behaviors may make agents stuck into suboptimal collaboration. Firstly, we give motivating example shown in (a) of Fig- ure 1 where agents need cooperate to catch the targets A, and C with different sparse rewards and punishments. Capturing A is more reward- ing than B and C. Agents may receive penal- ties for capturing different targets. We denote the behaviors generated by the joint policy of achieving a certain target concurrently as col- laboration pattern. In this setting, promoting collaboration with simply maximizing the MI of agents' behaviors cannot guarantee a better collab- oration. As shown in Figure 2 the patterns joint distribution is composed of patterns which capture targets A, B and C. This distribution is chaotic and contains various collaboration patterns. In this case, maximizing MI on any patterns of the distribution may enhance the inferior collaboration pat- terns, for example maximizing MI on the pattern B or C may make agents converge to a sub-optimal collaboration pattern. In fact, this is very common situation. In complex environments, there is a wide variety of collaboration, maximizing MI could only stimulate agents towards a certain col- laboration, which may be sub-optimal and in turn prevents from discovering better collaborations. This happens because, as learning preceeds, a seemly good collaboration will become sub-optimal once better one is discovered. However, if excessively maximize the MI using sub-optimal samples, the agents may be accelerated to fall into sub-optimal policies. Thus the motivation of our minimization MI. As shown in the right of Figure 2 we divide the distribution into the superior and inferior distributions. then we maximize the MI associated with superior collaboration patterns and minimize the MI associated with inferior collaboration patterns. In this manner, the current patterns strengthen towards superior patterns and are away from inferior patterns.
Superior Patterns A
Inferior Patterns Joint Distribution
Superior Pattems
Pattoms e,c
Joint Distribttion
Patterms Joint Distnbution
Current Pattern
Figure 2:
The left represents maximizing MI on all patterns and the right represents maximizing MI with superior patterns and minimizing MI with inferior patterns.
method is to solve the limitation with the maximization-miniiizztooo
Next we conduct experiments with four methods on the environment: MADDPG (Lowe et al., 2017) SIC-MADDPG (Chen et al.] 2019). VM3-MADDPG (Kim et al.) 2020) and our method where VM3 and SIC both promote collaboration by maximizing MI. Results of the count of algorithms converging to different collaboration patterns and overall performance are shown in (b) and (c) of We can that both SIC and VM3 have greater probability of falling into the sub-
Figure 1
3

---

Under review as a conference paper at ICLR 2022
optimal patterns than that into the optimal one. This demonstrates that maximizing MI only cannot guarantee the optimal collaboration, it may also stuck into sub-optimal collaboration. In addition, method outperforms other methods in both the number of effective patterns and the optimal patterns, which demonstrates that our method is more effective in facilitating collahoration while avoiding falling into sub-optimal collaboration patterns than other methods. In addition to the above problem, VM3-AC (Kim et al. (020) has scalability issue which need to compute the MI of policies between any two agents. Besides, all previous methods (Kim et al. 2020; Chen et al. 2019; Mahajan et al. 2019) need additional inputs, e.g., latent variable 2 in (Chen et al. 2019 Kim et al., 2020) or global initial state in (Mahajan al., 2019), during the decentralized execution phase, which violates the rule of CTDE paradigm.
To this end, we propose a new collaboration criterion as well as a novel framework to solve the above problems. Specifically, we propose new collaboration criterion to evaluate the collaboration of multiple agents in terms of the joint policy, individual policy, and other agents' policies. Based on the criterion, new way that calculates the MI of the global state and joint policy is proposed to promote collaboration which does not require an additional latent variable as input and addresses the scalability issue. To solve the problem of easily falling into sub-optimal collaboration by maximiz- ing MI only, we then propose PMIC framework which leverages two distinct mechanisms to guide agents in a progressive manner. Finally, to solve the inefficiency of being guided only by rewards (e.g., stochastic or sparse), we use the estimated values of the new MI which are provided by DMIE as intrinsic rewards. To the best of our knowledge, we are the first to propose a combination of maximizing the lower bound of MI and minimizing the upper bound of MI to promote collaboration in MARL. In the following section, we first introduce the new collaboration criterion, then we show the details of PMIC framework, finally we give an combination of our method with MADDPG.
3.2 COLLABORATION CRITERION
In this section, we propose new collaboration criterion to evaluate the collaboration quality in multi-agent systems. We suppose good collaboration should contain the following three properties 1) the joint policy should explore in the joint action space to enrich the diversity of the collaboration patterns and avoid model collapse. 2) Each agent i should be deterministic enough about what behaviours it should adopt given its observation. 3) Meanwhile, given an agent i's policy and other agents' observation, the uncertainty on other agents' policies should be low, thus more easily to achieve better collaboration. Based on this, the new learning objective considers the above three aspects: 1) maximizing H (mT-ss)); 2) minimizing the entropy of each agent's individual policy: (T; | s). where T. is the policy of agent i; 3) minimizing each agent's 's uncertainty on other agents' policies: H ,y, s). where -i is other agents except agent and _ is the joint policy of other agents. Thus our final objective can be written as:
H( ((m: 8) H(T=: n, 8) H (T((ss) H(T s) [(s;
(I)
The objective ultimately can be defined the mutual information of the global state and joint policy. Thus we can maximize the MI of global state and joint policy to promote collaboration. This MI objective is simple and general that does not rely on any latent variable or the scalability issue. However, simply maximizing this MI can hinder algorithm learning as discussed above. Therefore, we propose novel framework PMIC to solve the problem, which is detailed in the following section.
3.3 PMIC FRAMEWORK
In this section, we introduce our framework, Progressive Mutual Information Collaboration (PMIC), to leverage the collaboration criterion proposed above facilitate multi-agent collaboration. The key idea of PMIC framework is to identify superior and inferior collaboration patterns, and en- courage multiple agents to achieve superior patterns while prevent agents from performing inferior patterns. Such guidance of joint policy is imposed in progressive manner during the learning process, since collaboration patterns are not known in advance with an oracle view. Intuitively, with the help of PMIC, agents alter their collaboration patterns progressively towards better ones while avoid being trapped at sub-optimal collaboration patterns.
To be specific, Figure illustrates PMIC framework, consisting of two main components. The first component is Dual Progressive Collaboration Buffer (DPCB) which separately stores supe- rior and inferior trajectories progressively. DPCB maintains historical samples that reflect superior
4

---

Under review as a conference paper at ICLR 2022
Superior Collaboration Patterne
Dual Progressive Collaboration Buffer
Dual Mutual Information Estimator
Pasitive Buffer
Agent
store
(R:)
train
MINE
Agent
Tra (Rs)
Neural estimation
bound
max (Ra...)
Negative Buffer
Agent
Pattern
(R2)
train
CLUB
Agentt
Traj
Neural estimation upper bound of
Figure 3: An overall illustration of PMIC framework, consisting of two main components. Dual Progressive Collaboration Buffer (DPCB) maintains superior and inferior trajectories separately in progressive manner. Dual Mutual Information Estimator (DMIE) includes two neural estimators of the collaboration criterion (Equation based on separate samples in DPCB.
and inferior patterns progressively. The second component is Dual Mutual Information Estimator (DMIE), including two neural estimators of our MI collaboration criterion based on separate sam- ples in DPCB. DMEE provides quantitative measurements of state and joint policy variables based on their samples, which used improve agents' policies by maximizing the MI estimates asso- ciated with superior collaboration pattern and minimizing the MI estimates associated with inferior ones. In the following, we introduce the two components in details.
Dual Progressive Collaboration Buffer (DPCB). DPCB consists of a positive buffer and a neg- ative buffer to store superior and inferior trajectories respectively. To identify the superior tra- jectories, we use the average return R over the last M episodes as a baseline. We denote the trajectory with the lowest return the positive buffer as Rlow- For episode k, we store the tra- jectory with return R& if Ra > max( R); when the positive buffer is full, DPCB over- writes the trajectories with return Rjow is overwritten. This ensures the quality of collaboration patterns reflected by the positive buffer monotonically increases during the learning process. For the opposite case Rs & max( max(Ruww, R),, the trajectory is stored in the negative buffer, which maintained in First-In-Firtt-Out (FIFO) manner. This to make agents break recent in- ferior collaboration patterns, thus driving agents to adjust and explore consistently. The posi- tive buffer and negative buffer store the samples that reflect superior and inferior collaboration patterns encountered during the learning process.
Dual Mutual Information Estimator (DMIE). Based on the samples in DPCB, quantitative estima-
Encoder Fe,
tion of collaboration criterion proposed in See 3.2 obtained by training neural estimators of T(s; DMIE consists of a MINE (Belghazi al., 2018) es- timator of the lower bound of ((+s)) for max- imization based on the positive buffer, and a CLUB (Cheng et al. 2020) estimator of the upper bound of (-Is)) for minimization based on the negative buffer. In concrete, MINE approximates the lower bound of ((s;((-<>) below, based on states & and joint action samples u T(-(s)::
Action
Encoder
T..
Figure 4: Architecture of MINE for lower- bound estimation of TMINE ~ (-1s))) (St, u.) IPsu and (st, ug) IPs s Pu-
(-1s)) 2 (sINE(;; (s:s()))) = sup Epsu [-sp (-To, wi Wiel
2 (8, w)) Er, [sp (Tu, (s, u))] (2)
-L(w))
where Psu is state-action joint distribution s D.(.) and u ~ (-1s)) with some buffer D; and Ps and Pu are the marginal distribution of Psu. where the samples can be obtained by D.(.)
and u ~ (.). Tw, (s, u) is neural network with parameters W1 E Ω that outputs scalar and sp(2) = log (1 + exp(2)). Similarly, CLUB approximates the upper bound of I [(; x (-[s)):
[(s; ( (s)) < (cLUB(s;<(11a))) EEsu [log Tw (u s)] &F [log Twy (u s)],
(3)
[(wg)
where Tw,, (u|s) neural network with parameters wa that models the conditional distribution.
5

---

Under review as a conference paper at ICLR 2022
The training losses of MINE and CLUB neural esti- mators are [(w1) and (ww)) defined in above equa- tions. Thus the total loss of DMIE is:
5r)
Action
Prediction
Network
LDME (w))) + ((w)).
(4)
For calculation, the expectation of state-action joint distribution is replaced by sampling s D.(-) and u ~ <(-1s) from the positive buffer and negative buffer respectively. Detailed architecture of MINE and CLUB are illustrated Figure 4 and We pro- vide complete process of training and calculation in Appendix C After training, given state and joint-action samples, we can use MINE and CLUB neural estimators calculate the MI estimates according to Equation [2 and
Tag
Figure 5: Architecture of CLUB upper- bound estimation of (1; ~ <--ls)). Twy (wel [se) and Tw, (ux())) are the proba- bilities of inferring u, and ug based on St-
MINE and CLUB are trained with the samples from the positive buffer and negative buffer respec- tively. Intuitively, only the behaviors that obey superior collaboration patterns or inferior collabo- ration patterns have large MI estimates calculated by MINE or CLUB, since [(wi) and ((w2) are optimized. Therefore, given the interaction samples of current joint policy of agents, MINE and CLUB can calculate the MI estimates which provide effective signals in guiding agents' behavior towards or away from superior and inferior collaboration patterns progressively. We introduce how to integrate PMIC framework with MARL for more efficient learning in the following.
3.4 INTEGRATION OF PMIC AND MARL
Recall the key idea of PMIC framework: leverage superior and inferior collaboration patterns encountered during the learning process to provide effective learning signals that encourage agents to achieve better collaboration progressively while avoid being trapped by sub-optimal ones. With the quantitative MI estimation introduced in previous subsection, we aim to find the joint policy that maximizes the MI estimate from MINE and minimizes the MI estimate from CLUB in addition to maximizing expected discounted return conventionally. Formally, the objective function of PMIC- MARL is: ((x) = E, IEx= y'rr]= + M (s; (+1s)) ICLUB (s; π (-|s)). Note all three terms to optimize in the equation
expectation over trajectories generated by policy и. We then transform the two MI estimation terms into fractions of each time step and arrive at the following objective with per-step MI rewards:
J (TT = -E,,, 2U
Σ E PMIL)) where ri
DMIE a. (86 us) Blcl.u (8:3 uz)).
(5)
a and B are the hyperparameters that control the dependence on MI guidance. In principle, PMIC- MARL a general framework which can be implemented with different MARL algorithms.
For representative implementation, we introduce PMIC-MADDPG by integrating PMIC frame- work with MADDPG (Lowe al., (017). Other PMIC-MARL implementation is also studied in our experiments. The pseudo-code of PMIC-MADDPG is shown in Algorithm [ In each episode, agents interact with the environment and store the trajectory samples into experience replay (Line 3-7 in Algorithm|D D. The trajectory is added to the positive buffer or the negative buffers in DPCB according its return (Line in Algorithm| []. In frequency of i steps, DMIE trained with the samples from DPCB (Line 10-11 in Algorithm T to consistently reflect current superior and inferior collaboration patterns, At last, PMIC-MADDPG updates Q networks and the policies of agents based on DMIE (line 12 in Algorithm D. (o(#) and L, (0) are the objective functions of Q and policy which are defined as:
(Ø E. E. [0 (o-Qo (s, u))" C- = E.-ol - To (o(us)))
where à =
- r +r p DIME + sub-optimas behavioral patterns and learn towards better collaboration, which promotes efficient and stable learning process.
or (s', 1 (s))). With PMIC, the agents can progressively break the current
EXPERIMENTS
In this section, we design experiments answer the following questions:

---

Under review as a conference paper at ICLR 2022
Algorithm PMIC-MADDRG
1: Initialize: Q networks o, N actor networks 0y..., and corresponding target networks d', .i..... On's
DMIE with parameters wi and w2 and DMIE update frequency DPCB; experience replay buffer D for each episode do
3: for each time steps t do for each agents i do
4:
Select action u, *e, (of) for each agent Execute joint action u; and observe observations { (or global state 8(41) reward Ft 7: Store interaction trajectories to D
5:
6:
8:
Add interaction trajectories to the positive buffer or the negative buffer in DPCB (according Sec 3.3 for each training step step
do
10:
if m%k == then
11:
Update DMIE with DPCB according to Equation
12:
Update Q and actor networks according to Equation
Pray agenns
Pray (6 (gageet
MACOYIC
MASAS
PAID-MMOOPG
00
Preducior (hey (12 agente)
Predator Pray (24 aegeraa))
Naw gation
-22
201
Stpp 196
(4.0
Figure 6: Comparisons of averaged return on MPE.
RQ1 (Performance) Can PMIC effectively improve algorithms and outperform other methods?
RQ2 (Effectiveness of maximizing and minimizing / (s; ग ()) Whether 1 (ss, (u]s)) is effec- tive? can both mechanisms bring performance improvements?
RQ3 (Effectiveness of DPCB) Is DPCB necessary component for PMIC?
Benchmark & Baselines. For a comprehensive comparative study, we evaluate our algorithms on both discrete and continuous action spaces. For the continuous action space, two kinds of do- mains are considered: Multi-Agent Particle Environment (MPE) and Multi-Agent MuloCo bench- mark, and we compare PMIC-MADDPG with six advanced algorithms as baselines: SIC-MADDPG (Chen et al. 2019). VM3-AC (Kim et al. 2020), MASAC (Kim et al. 2020)) MADDPG (Lowe et al.) 2017), FacMADDPG (de Witt et al. 2020), COMIX (de Witt et al. (0200) where FacMADDPG and COMIX are the sfate-of-thh-ar (SOTA) algorithms in Multi-Agent MuJoCo benchmark (de Witt let al. 2020). For environments in discrete action space, the challenging StarCraft II micromanage- ment (SMAC) benchmark is considered which has high complexity of control and requires learning policies in large discrete action space, and compare PMIC-RÓDE with the current SOTA algo- rithm RODE on SMAC. Environment description is provided in Appendix B
code provided by their authors build the algorithm according to the original papers where the hyperparameters have been fine-tuned on all environments. Further implementation details can be found in Appendix H
For all baseline algorithms,
4.1 PERFORMANCE (RQ1)
We first evaluate the performance on environments of MPE with 10 different random seeds: Wildlife Reseue, Cooperative navigation, Partial Observation Cooperative Predator Prey with 3, 6, 12, and 24 predators respectively, where the agents control predators for collaboration and the policy of prey is fixed. The results shown in Figure PMIC-MADDPG has significant improvement

---

Under review as a conference paper at ICLR 2022
4000
2-Agert Hos Kchwetsh
2-Agert Willer
GAAgest HellChwwah
2003
+330
600
Ravwer
1983
2010
KASAC
MADIPO
60)
PMR--0020PG
VMSACC
12 12 15
1. 1477 (lar
Figure 7: Comparisons of averaged return on Multi-Agent MuJoCo.
for MADDPG and outperforms other methods across all tasks. On these environments, a higher reward means a better collaboration pattern. We can see that both VM3-AC and SIC-MADDPG get lower rewards than PMIC-MADDPG which indicates maximizing MI makes agents fall into sub- optimal collaboration pattern quickly and cannot guarantee better collaboration pattern. Besides, PMIC can still deliver significant performance gains as the number of agents becomes larger, while some other algorithms cannot learn to collaborate effectively. For example, Predator Prey with 24 agents, only PMIC-MADDPG and VM3-AC can learn collaborative behaviors, where VM3-AC requires three times more time consumption than PMIC-MADDPG.
We further evaluate PMIC on 3 tasks of Auent MuJoCo benchmark with 10 different random seeds. In these tasks, agents need to cooperate in robot control and different agents control different joints of the robot. In our experimental setting, agents do not share information, which is the most difficult setting Multi-Agent MuJoCo. The experimental results in Figure show that PMIC- MADDPG performs significantly better than MADDPG on the tasks. On the other hand, PMIC- MADDPG outperform other baselines (COMIX, DEI--MAAPP.SS-SADDEE
VMBAC,MASAC), which demonstrates the benefits of PMIC for improving performance in challenging cooperative tasks with partial observation and continuous action space.
10
10m, ve. 11m
3x, ve
PMSC RODE
H0007
VWW Rath
0.3
0.1
(4-)
fhom (e6
2532
NMNV
MMN2
02
02
62
533
42
0.6 880
Tiaa 280k (41)
Time (14)
Figure Comparisons of averaged test win rate on SMAC.
For SMAC, we evaluate PMIC 6 maps with 5 different random seeds. Note that RODE is the SOTA algorithm on SMAC thus we mainly compare our method with RODE. The results are shown in Figure 8 PMIC can also provide an improvement for RODE with faster rate of convergence and higher performance. In summary, we show that PMIC is an effective framework that can be integrated with most algorithms and can provide significant improvements both in continuous action space and discrete action space. More experiments are included in Appendix
4.2 EFFECTIVENESS OF MAXIMIZING AND MINIMIZING I(s; (RQ2)
To answer RQ2, we first provide an ablation study on maximizing and minimizing [(s; ा (u(s)) to investigate the effect of the two mechanisms. As shown in (a) and (b) of Figure using maximizing and minimizing [(s; ι (u)s)) can achieve faster convergence and higher performance. than only using maximizing or minimizing [(s; T (u]s)).
To verify whether the first estimator MINE that maximizes [(s; T (u]s)) can guide agents correctly, collect optimal and sub-optimal collaboration patterns train MINE respectively, then the
8

---

Under review as a conference paper at ICLR 2022
Cooperal
Visile
Viliilo Resius
MA-DPIG
Eiaand E. 2aviir
MADOPG- MAK-
-00
Ditand B1
Rloard
Mintriza
(14
15
50
20 20
30
(a)
(b)
Figure 9: Ablation study and effect about maximizing and minimizing [(;; ( (ulls))
trained MINE to guide an initialized MADDPG from the beginning. As shown in (c) of Figure y training MINE with optimal collaboration patterns can quickly guide the algorithm to learn the op- timal patterns, which indicates that maximizing I(s; π (u[s)) on superior collaboration patterns can guide agents correctly and quickly. However, with the guidance of MINE trained on sub-optimal patterns, the policy only converges to the sub-optimal patterns. This indicates that only maximizing MI without distinguishing the quality of collaboration patterns will make agents fall into inferior col- laboration patterns. This further verifies our assumption in the motivation section and also indicates the necessity of designing DPCB.
Next we examine whether the second estimator CLUB that minimizes I(s; (u|s)) can break the current sub-optimal collaboration pattern and help agents achieve better collaboration. We test on Wildlife Rescue and find policy which has trapped into suboptimal patterns(e.q., rescue animals with lower rewards). After million time steps, we add minimizing [(s; क (u(s)) for the policy with different B. The result in (c) of Figure 9 shows that the training curve gradually increases and achieves higher rewards than baseline. This indicates that minimizing / (s; i (u(s)) can help the
algorithm break the current sub-optimal pattern and discover better collaboration patterns. Due to space limitation, more ablation experiments are putted in Appendix F
4.3 EFFECTIVENESS OF DPCB (RQ3)
We finally examine whether the pos- itive buffer and the negative buffer in DPCB can be replaced with a normal replay buffer. The results are shown in Figure TO Leveraging DPCB is the most effective way. In fact, lever- aging normal replay buffer can cause patterns conflicts which means train- ing MINE and CLUB with same pat- terns and can not guarantee the cor- rect guidance.
Widlf Rescue
Cooperative Navigaton
-109
Bpad
-12
Reward
NAD3PO
140
-102
00
11.
20
30
Shock
Figure 10: Ablation experiments on DPCB.
CONCLUSION
In this paper, we propose a novel collaboration criterion in a form of the mutual information between global state and joint policy. This bypasses the introduction of explicit additional input of policies that violates CTDE paradigm and mitigates the scalability issue meanwhile. Moreover, to address the potentially detrimental encouragement induced by mutual information maximization, we pro- poses a framework called PMIC. consisting of: DPCB which progressively maintains superior and inferior collaboration patterns; and DMIE which has two MI neural estimators of our MI collabora- tion criterion based on separate samples in DPCB. In additional to maximizing expected discounted return, PMIC-MARL aims at maximizing the MI estimates associated with superior collaboration to guide agents to facilitate better collaboration and minimizing the MI estimates associated with infe- rior collaboration to encourage exploration and avoid falling into sub-optimal collaboration. In our experiments, we evaluate several implementations of PMIC-MARL in a wide range of cooperative environments with both continuous action space and discrete action space. The results demonstrate that the effectiveness and generalization of PMIC.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540, 2018.
Liheng Chen, Hongyi Guo, Yali Du, Fei Fang, Haifeng Zhang, Yaoming Zhu, Ming Zhou, Weinan Zhang, Qing Wang, and Yong Yu. Signal instructed coordination in cooperative multi-agent reinforcement learning. arXiv, pp. arXiv-1909, 2019.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning, pp. 1779-1788. PMLR, 2020.
Abhishek Das. Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 1538-1544 PMLR, 2019.
Christian Schroeder de Witt, Bei Peng, Piexre-Alexanrre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709 06700, 2020.
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emer- gent communication in multi-agent reinforcement learning. arXiv preprint arXiv:1912.05676, 2019.
Yijie Guo, Junhyuk Oh, Satinder Singh, and Honglak Lee. Generative adversarial self-imitation learning. arXiv preprint 11X1v:1812.009500, 2018.
Shariq lqbal and Fei Sha. Actor-atrention-critte for multi-agent reinforcement learning. In Interna- PMLR, 2019.
tional Conference on Machine Learning, pp.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Çaglar Gülçehre, Pedro A. Ortega, DJ Strouse, Joel Z. Leibo, and Nando de Freitas Intrinsic social motivation via causal influence in multi-agent RL. CoRR, abs/1810.08644) 10.08647, 2018. URL http://aax.iv.org/aab...
08647
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International Conference on Machine Learning, pp. 3040-3049. PMLR, 2019.
Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning schedule communication in multi-agent reinforcement learning. arXiv preprint arXiv:1902.00 2019.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. maximum mutual informa- tion framework for multi-agent reinforcement learning. arXiv preprint arXiv:2006.0277 02736.022222220 2020.
Woojun Kim, Jongeui Park, and Youngchul Sung. Communication in multi-agent reinforcement learning: Intention sharing. In 9th International Conference on Learning Representattinns ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https: //openreview. www. peet/forrm?iddggae 12dR 9twy]
Mare Lanctot, Vinieius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent rein- forcement learning. arXiv preprint arXiv:1711.008: 2017.
Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The World Wide Web Conference, pp. 983-994, 2019.
Minghuan Liu, Ming Zhou, Weinan Zhang, Yuzheng Zhuang. Jun Wang, Wulong Liu, and Yong Yu. Multi-agent interactions modeling with correlated policies. arXiv preprint rrXiv:2001.03415 2020.
10

---

Under review as a conference paper at ICLR 2022
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-compeiite environments. In Advances in neural infor- mation processing systems, pp. 6379-6390 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7613-7624
2019.
Laëtitia Matignon, Laurent Jeanpierre, and Abdel-Illah Mouaddib. Coordinated multi-robot explo- ration under communication constraints using decentralized markov decision processes. In Jörg Hoffmann and Bart Selman (eds.), Proceedings of the Twenty-Sixs AAAI Conference on Artificial Intelligence, 2012.
Ramona Merhej and Mohamed Chetouani. Lief: Learning to influence through evaluative feedback. In Adaptive and Learning Agents Workshop (AAMAS 2021), 2021.
Frans A Olichock and Christopher Amato. Springer, 2016.
A concise introduction to decentralized POMDPs.
Peng Peng, Ying Wen, Yaodong Yang. Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent lidirectionally-corrinne nets: Emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint 1rXiv: 2017.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foer- ster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. arXiv preprint arXiv: 1485, 2018.
Julien Roy, Paul Barde, Félix G Harvey, Derek Nowrouzezahrai, and Christopher Pal. Promoting coordination through policy regularization in multi-agent deep reinforcement learning. arXiv preprint (908::/990..222222 2019.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Interna- fional Conference on Machine Learning, pp. 5887-5896. PMLR, 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinieius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. delopoeccomooiitoon networks for cooperative multi-agent learning. arXiv preprint arXiv:/706.05296, 2017.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence-based multi-agent explo- ration, 2019a.
Weixun Wang, Tianpei Yang. Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. Action semantics network: Considering the effects of actions in multiagent systems. arXiv preprint urXiv::9077 1461, 2019b.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for 2019.
multi-agent reinforcement learning, arXiv preprint
Annie Xie, Dylan P Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent repre- sentations to influence multi-agent interaction. arXiv preprint arXiv:2011.06619 2020.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. arXiv preprint 1xXiv:1909.02663 2019.
11

---

Under review as a conference paper at ICLR 2022
Predator
Predator
Agent
landmark
Predator
Agent
Predator
Prey
Agent
Prey
(a) Predator-Prey
(b) Cooperative Navigation
Wildlife Rescue
Figure 11: Multi-Agent Particle Environment.
A RELATED WORK
We have divided previous works on facilitating multiple agents collaboration into three branches in See In this paper, we mainly pay attention the third branch, where the previous works pro- mote agents collaboration by increasing the correlation of agents. We divide the previous works in two categories.
the first category, the algorithms take holistic view and facilitate collaboration by increasing the correlation (i.e., mutual information) of the agent' own behaviours and the joint policy. For example, SIC [Chen et al. 2019) extracts the information of the joint policy into the latent variables, which are then used the input of agents' policy networks, then SIC maximizes mutual information of each agent own behaviors and the latent variables to improve the correlation of agents. Based SIC, the agents will know what kind of the joint policy the whole team is tak- ing and what kind of action should be selected based on the latent variables. MAVEN MMahjjan al. 2019 has similar idea with SIC, except that MAVEN extracts joint policy information from the initial global state and maximizes the mutual information of future trajectories and the latent variables. In the second category. the algorithms facilitate collaboration from the perspective of individual agents by enhancing the correlation among different agents. For example, EITI (Wang al. 2019a leverage MI to captures the influence between an agent's current actions/states and other agents' next states in grid environments. SI-MOA (aques et al. 2019 proposes social influ- ence intrinsic reward to captures
mutual information between multiple agents' actions to achieve coordination. VM3 (Kim et al. 2020 has similar idea with SIC-MOA except that VM3 modifies policy iteration based the MI and introduces additional input to explicitly represent the relation of pro-
agents' policies. Finally, VM3 achieves better performance than SIC-MOA. In this paper, pose a new collaboration measurement by mutual information between the global state and the joint policy. Besides, our method consider the problem introduced by maximizing MI and propose the maximization-miniiiii
MI to solve the problem of falling into inferior collaboration patterns.
B ENVIRONMENT DETAILS
Predator Prey: N slower cooperating agents chase the faster adversary around randomly generated environment with L large landmarks impeding the way. In our setting, agents control the predators to chase the prey, the policy of prey is fixed. The agents are partially observable, the observation radius of each predator is 0.25. Only when the predator captures prey, predator can get the reward 10. set N to 3, 6. 12. 24 separately. Each game has 25 steps.
Cooperative Navigation: Agents must cooperate through physical actions to reach set of L land- marks. In setting, agents receive shared reward which is the of the minimum distance of the landmarks from any agents, and the agents who collide each other receive negative reward -1. Besides, all agents receive 1 if all landmarks occupied. Each game has 25 steps.
Wildlife Rescue: agents must cooperate to rescue M wildlife with different risks and rewards. We provide control time T.. When agent catches up with animal, the agent can control T,
12

---

Under review as a conference paper at ICLR 2022
seconds to wait for other agents arrive. Each game has T steps. We set T. to 8, T to 60. The specific reward at the end each episode is set in Table [
Table I: The reward matrix of the rescue agents at the end of each episode. Both agents receive the same reward.
reward
agent 2
tiger
deer
cat
the road
agent
tiger
-30
0
-30
deer
-30
7
6
-30
cat on the road
0
6
5
0
-30
10
0
0
Multi-Agent MuJoCo (de Witt et al. 2020) is range of challenging continuous multi-agent control tasks of Multi-Agent MuJoCo bench- mark suite. Multi-Agent MuJoCo is designed for decentralized cooperative continuous multi- agent robotic control. The structure is shown in Figure 12] For evaluation, we use the re- ward setting of the original paper but we to zero for all environments. k controls how much information each agent can observe from its adjacent agents. When i is zero, it means each agent can only observe information about its own joints, which is the hardest setting for Multi-Agent MuJoCo.
[A]
(8)
[D]
(E)
[F]
Figure 12: The structure of Multi-Agent MuloCo. Different colored parts of the robot controlled by different agents.
For SMAC, we use the latest version 2.4.10 and default settings for all maps.
C PROCESS OF TRAINING MINE AND CLUB
In this section, we detail the process of training MINE and CLUB.
First we introduce the process of training MINE. We train MINE based on the positive buffer of DPCB. Figure Caauur//iilustaattt illlstratess the detailed process of estimating the lower bound. We design the net- work T with parameters W1 state encoder Ew, and action encoder E MINE takes the (8t, u4) ~ IP Pss and (st, ux) ~ Ps 6 Pu as input and encodes the data as vectors (s,,(e),, (8t), Fus, (ut). Fu,, (ux) with the same dimension. Then take the inner product of Fu,, (st) and Fus, (u.), E., (st) and Ew,, (ux) to get Tw,, (st, uz) and Tw, (St, ux) (e.q., scores) separately. Finally we can Equation (2) the estimate of IMIEE St. ut). In summary, MINE is trained by sampling data from positive buffer of DPCB to minimize [(w)) in Equation (2).
plug the scores
Then we details the process of training CLUB. We train CLUB based on the negative buffer of DPCB. Figure iillustrates the detailed process of estimating the upper bound. We design the net- work T' with parameters W2 as action prediction network. The train process of CLUB different from MINE, we only need to sample data from the joint distribution Psu- Then we directly mini- mize the loss L(w2) based on the samples improve the accuracy upper bound of MI estimated by CLUB.
D INTEGRATE PMIC WITH RODE
RODE is a role based algorithm which decomposes joint action spaces into restricted role action space to reduce the primitive action-observation spaces. RODE, each task can decomposed into sub-task which has smaller action-observatinn space, each sub-task is associated with role p. To demonstrate the generalization and effectiveness PMIC, we design EMIC-RODE (e.q. integrate PMIC with RODE). To integrate with PMIC, we take the joint role p into consideration. Thus
13

---

Under review as a conference paper at ICLR 2022
measure the MI between the joint policy and global state with the joint p (e.q., [(u: p))). Since the action is selected based on the observation and the selected role. [(u; p. s) means the correlation between the global state, the selected joint role and joint policy which has the same effort with T(u; s) in PMIC-MADDPG Different from PMIC-MADDPG, we set rf ,MMM = a ( Pt; ut) ATCLuB St. Pe; u-))- rf r.MI is used in the same way in PMIC-MADDPG. To achieve the goal in Equation we only need to modify the Qtor as follows:
L ED [ Qtoc(s, (s,,))) 2))²
where Brode = r DMIE + max, Qtot (s', Tgr (s))) and Qiot a QMIX-style (Rashid al. 2018) mixing network to estimate the global action-value and Qtat is the target network of Q DMIE updated according to Equation Since RODE is QMIX-based method, credit assignment is also applied to /MIN and ICLUB. The MI superior collaboration patterns and inferior collaboration patterns can reward and punish the agents based on their contribution, which can better help to find the optimal collaboration patterns.
To integrate PMIC with other algorithms, we simply replace MADDPG with other MARL algo- rithms. For example, To integrate with RODE, only need to initialize the networks of RODE in line I of Algorithm [ and retain other components. Then we use RODE to select actions in line 4-5. All other processes remain the same except changing T(u; s) to I(u; p. s). Finally, we update the value network according to Equation and others are the same with RODE to replace the line 12 in Alg Algorithm The same applies in combination with other algorithms.
E EXPERIMENTAL SETTINGS
On MPE and SMAC, we use episode reward as the criterion for adding data to DPCB.
On Multi-Agent MuJoCo, we can more accurately collect superior collaboration patterns. Since when the episode reward of one trajectory is high, there no guarantee that there are no poor sub- trajectories which affect DPCB. Thus we leverage a more fine-grained filtering method to filter the collaboration patterns on Multi-Agen Agent MuJoCo. We leverage sub-trajectories to replace the complete trajectory to accurately filter superior and inferior collaboration patterns. In particular, we set the length of sub-trajectories to 50 steps except 100 steps for 2-Agent Walker and use the total reward of each sub-trajectory the criterion for adding data DPCB. The update frequency of MINE and CLUB is on all tasks.
The experiments MPE are carried out Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz. The experiments on Multi-Agent MuJoCo are carried out on NVIDIA GTX 2080 Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. The experiments on SMAC are carried out on Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz and Intel(R) Xeon(R) CPU ES-2679 v4 @ 2.50GHz.
F ADDITIONAL EXPERIMENTS
In this section, present more experiments help better understand method. Four questions are raised and more experiments on Multi-Agent MuloCo are provided.
RQ1: Is DPCB buffer size sensitive?
RQ2: Is maximizing I (ss; π (-1s)) better than other MI forms?
RQ3: What is the time consumption of PMIC-MADDPG?
RQ4: Is a and 33 PMIC sensitive?
To answer RQ1, we design experiments on MPE and adjust the size of DPCB. Specifically, we test 5 different buffer sizes (100, 500, 1000, 5000, 10000) for positive buffer and negative buffer of DPCB. The results shown in Figure 13 indicate that both too large buffer size and too small buffer size have a negative effect on the performance. A buffer size that is too small prevents MINE and CLUB from carving patterns well, while too large buffer size hinders the optimality of DPCB which makes it impossible for MINE and CLUB to adapt quickly to the new collaboration patterns. Therefore an appropriate buffer size is important.
14

---

Under review as a conference paper at ICLR 2022
Cooperative Navigation
Predator Prey (3 agents)
-100
-120
-140
Reward
Reward
-160
100
100
1000
1000
10000
10000
-200
500
500
5000
-220
5000
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Time Steps (1e6)
Time Steps (1e6)
Figure 13: Influence of DPCB size.
Predator Prey agents)
2-Agent HalfCheetah
5000
25
4000
Reward
Reward
3000
10
2000
PMIC version
PMIC-version
5
SIC version
1000
SIC-version
VM3 version
VM3-version
0
Time Steps (1e6)
Time Steps (1e6)
Figure 14: Comparison maximizing mutual information with different forms. PMIC-version leverages MI of global state and joint action. VM3-version leverages the MI of any two agents' policies among agents. SIC-version leverages the MI of z and the joint policy.
To answer RQ2, we directly maximize MI with different forms to give a comparison. avoid differences caused by some mechanisms (e.g., double Q) or structures (e.g., MINE), we apply dif- ferent forms MI to MADDPG and leverage MINE as the MI estimator and other settings remain the same. The hyperparameters have been fine-tuned. The results mize MI of the global state and the joint action better than other MI forms in terms of the final performance. This demonstrates that proposed method of maximizing the MI of global state and joint action is more effective. Besides, we also find that maximizing MI on environments with sparse reward brings performance degradation the original algorithm. The reason is: Due the difficulty of exploration on environments with sparse reward, mutual information will play a greater role guiding compared to sparse reward. Since maximizing MI has the problem of easily making agents fall into suboptimal collaboration, agents quickly fall into suboptimal collaboration with the guidance of MI.
shown in Figure [4] Maxi-
To answer RQ3, we evaluate the time consumption of different algorithms on 6-Agent HalfCheetah. The experiment carried out on NVIDIA GTX 2080 Ti GPU with Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz. We evaluate the time consumption of each algorithm individually with additional programs running on the device. Each result is the average of ten time-consuming calculations. The time consumption includes the time consumption of the execution phase and the time consumption of the centralized training phase. The results are shown in Table PMIC-MADDPG brings less time consumption than other methods.
To answer RQ4, we give ablation experiments of and B. The results are shown in Figure [5 The experiment proves that appropriately sized values of and B are critical for algorithm improvement.
15

---

Under review as a conference paper at ICLR 2022
Table Time consumption of different algorithm on 6-Agent HalfCheetah every 1000 time steps.
algorithm seconds
MADDPG PMIC-MADIPGG
SIC-MADDPG
VM3-AC 182.58
30.96
24.43
31.01
algorithm
COMIX
MRC-MADDPG
MASAC
seconds
34.43
87.13
41.68
Cooperative Navigation
Cooperative Navigation
-100
-100
-120
Reward
Reward
-140
180
0.0
0.0
0,0001
160
0.0001
-80
0.001
0.001
0.01
-180
0,01
-200
0.1
0.1
-220
MADDPG
-200
MADDPG
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Time Steps (1e8)
Steps (1e8)
Figure 15: Ablation experiment of and B.
The performance loss occurs if 3 or is too large too small. Thus for different environments, we need adjust a and achieve the best performances.
Predator Prey (3 agents)
2-Agent HalfCheetah
6000
40
5000
W
4000
Reward
Reward
25
20
2000
MADDPG
MMCC-MADPGG
1000
5
PMIC-MADDPG with normal estimator
0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
2
3
5
6
9
10
11
12
13
14
15
Time Steps (1e8)
Time Steps (1e6)
Figure 16: Comparison of PMIC with MINE and PMIC with normal estimator.
Later we evaluate the impact of MINE on the performance of the algorithm. Because we are using MINE for the first time in the field of multi-agent reinforcement learning, thus need provide comparison of MINE and the normal mutual information estimation method. The results are shown in Figure 16 We can see that the performance of both methods similar, which demonstrates that the effectiveness of PMIC is introduced by the maximization minimization MI, rather than MINE.
We further analyze whether MINE trained with the positive buffer in DPCB could act as a good guide, which requires the MINE to estimate large value to superior collaboration patterns and small value to inferior collaboration patterns. We train MADDPG in 2-Agent Walker. During training, we save the positive buffer every 100000 steps, which ensures that the collaboration patterns saved each time better than the previous time. Then we save the MINE every 100000 steps and use it to estimate MI of collaboration patterns in saved positive buffer. The estimated MI of different positive buffer by different MINE are plotted on Figure I7 The larger the index, the newer the MINE and buffer. When line, indicating that MINE has a higher estimate for the better collaboration patterns. But when train MINE in the same way with VM3-AC (Kim et al. 2020) MINE estimates the same value of MI for
use DPCB to train MINE, the color gradually darkens from left to right in each
16

---

Under review as a conference paper at ICLR 2022
MINE index
0.6
34 13 32 31 10 9
0.4
MINE
positive buffer index
positive buffer index
Figure 17: Visualization results of the MI estimated by different MINE for different positive buffer. X-axis denotes the index of positive buffer. Y-axis denotes the index of MINE. The value in coordi- nates (z, y) represents the MI estimated by the MINE with index for the buffer with index x.
both inferior and superior collaboration patterns. From the perspective of visualization, the MI of superior coordination estimated by MINE with DPCB is large that such coordination can be more encouraged, which can give accurate guidance to superior collaboration patterns.
2-Agent Humanoid
2. 2.Agent HumanoiddStand
4 Agent Art
630
150000
140000
120000
9000
Reward
120000
Reward
750
#00
110000
350
100000
MADDPG
90309
-200
PMIC-MADOPG
10003
230
10
Figure 18: Comparison of more tasks on Multi-Agent MuloCo.
In addition to the above experiments, we add more experiments on Multi-Agent MuJoCo. The re- sults shown in Figure 18 PMIC has also significantly improvement the original algorithm for some more difficult robot control tasks such as humanoid and humanoid-standupp These experi- ments comprehensively demonstrate the effectiveness and generality of PMIC.
Pray (3 agents)
2-Agent HaltChoetah
6-Agent
2000
9502
5000
1500
Roward
Rewwrd
Reward
1000
30
9935
500
10
PNIC MASAC
MASAC
2505
86
Time Papa (148)
Tire Stoza
Sapo (1e8)
Figure 19; Comparison of MASAC-based methods on Multi-Agent MuloCo.
Finally, we integrate PMIC with MASAC to further verify the generalisation and effectiveness of
PMIC. The results are shown in Figure 19 PMIC has also significantly improvement MASAC. Beside, PMIC-MASAC is better than other MASAC-based method VM3-AC. The effectiveness and generalisation of PMIC is more convincing by the above experiments.
17

---

Under review as a conference paper at ICLR 2022
G COMPARISON WITH OTHER RELATED ALGORITHMS
2-Agent HalfCheetah
6-Agent HaifCheetah
6000
2000
5000
1500
4000
Reward
Reward
3000
1000
2000
500
1000
GASIL-MADDOG PMIC-MADDPG
13
10
13
Time Steps (1e6)
Time Steps (1e6)
Figure 20: Comparison between leveraging PMIC and GASIL.
In this section, we give new question: Whether PMIC can be replaced by discriminator which can also achieve the purpose guiding agents to better collaboration. To answer question, more experiments are carried out. This idea using discriminator is similar to Generative Adversarial Self-Imitation Learning (GASIL) (Guo et al., 2018) where a discriminator is trained to discrimiate between superior trajectories and inferior trajectories, while the policy learns to fool the discrimina- tor by imitating superior trajectories. Specifically, GASII. maintains a good trajectory experience. The discriminator scores the good trajectory to 1.0 and scores the trajectory generated by current policies to 0.0. The results are shown in Figure [0 which indicates that our MI based architecture PMIC is more effective than GASIL, which may be mainly due to the fact that the discriminator is difficult to train and can suffer from pattern collapse.
H HYPERPARAMETERS
The hyperparameters for network architecture:
two fully connected layers comprised of units 64 with ReLU nonlinearity and a final layer with tanh to output actions as the policy network for each agent, use the critic network with the same architecture the policy network except tanh for the final layer.
I: On MPE,
2: For Multi-Agent MuJoCo, the policy networks and critic network use the same architecture with those MPE, but with 200 and 100 units for two fully connected layers.
3: For SMAC, we use code provided by RODE, the parameters and details are the same with the original paper.
All algorithms' the comparison is fair. For MADDPG, For MINE, we use two fully connected layers (100 units on Muti-Agent MuJoCo and 64 on MPE) with Leaky ReLU nonlinearity encode global states and joint actions respectively, the results obtained by dot product of the embeddings of global state and joint action. For CLUB, we use two fully connected layers (50 units on Muti-Agent MuJoCo and 32 on MPE) with Leaky ReLU nonlincarity to encode global states. For SMAC, we need to encode p. thus CLUB uses two fully connected layers (64 units) to encode p and the global state, then the outputs are concatenated to form vector of 128 dimensions, and use the vector to predict the mean and variance of the global action. MINE uses two fully connected layers (32 units) to encode p and the global state and uses a fully connected layer (64 units) to encode the joint action. the results are obtained by dot product of the embeddings of new vector (e.q., combine the vectors of the global state and the joint role p) and joint action). The architectures used to calculate MI in other MI-related algorithms (SIC-MADDPG and VM3-AC) use the same number of units and activation function with MINE and the other parts consistent with the setting in original papers.
network architecture (on MPE and Multi-Agent MuJoCo) is the same to ensure that
centralized Q network and N policy networks.
18

---

Under review as a conference paper at ICLR 2022
For SIC-MADDPG and VM3-AC, there are two parameters to adjust: the dimension of 2 and a. For SIC-MADDPG, we select z's dimension from [2,3, 5, 8, 10, 15, 20) fol- lowing the setting in original paper MPE, 22,,44558, 10, 15] Multi-Agent MuJoCo and select a from (0.1,0001 0.001, 0.0001, 0.00001] following the setting in original paper. For VM3-AC,
select 2's dimension from [2, 224 following the setting in original pa- per and select a from [0.1,0.0L 0,001, 0.0001, 0.00001]. For MASAC, we adjust B from [1.0, 0.1, 0.01, 0.001, 0.0001], 0.00001 control the entropy. For PMIC, need adjust and 8, we select from [1.0. 0.1, 0.01, 0.001, 0.0001]. The final choice of B and dimension of is shown in Table 3 For FacMADDPG and COMIX, we use the official code and the parameters of the original paper.
For other hyperparameters on MPE and Multi-Agent MuloCo, 10-3 for critic and 10-1 for actor on MPE except 1 x 10-2 on Wildlife Rescue and use Adam optimizer with learning rate 1 10-3 for critic and x 10-4 for actor on Multi-Agent MuJoCo. For MINE and CLUB, the learning rate is x 10-4 on all environments except 1 10-3 on Wildlife Rescue. The discounted factor 7 and are 0.99 and 0.002 on Multi-Agent MuJoCo and 0.95 and 0.001 on MPE. Replay buffer size is 10 on MPE and Multi-Agent MuJoCo excpet x 10 on Wildlife Rescue. Batch size 1024 on MPE and 100 on Multi-Agent MuJoCo. The size positive buffer and negative buffer of DPCB is 1000 MPE except 6000 Wildlife Rescue. 5000 for positive buffer and negative buffer on Multi-Agent MuloCo. 500 for positive buffer and 3000 for negative buffer on SMAC.
For hyperparameters of RODE, all parameters remain the same as in the code provided in the original paper. RODE has main adjusted hyperparameters: Number of role clusters and role interval. Number of role clusters is used to control the number of role types. The role interval decides how frequently the action spaces change and may have a critical influence on the performance. All parameter settings are consistent with the original paper. The selection of a and / is shown in Table. 4 Batch size 128 to update MINE and CLUB, We apply maximization and minimization of MI after 100000 time steps on SMAC.
Table 3: Selection of a, 3 and dimension of for different algorithms MPE and Multi-Agent MuloCo.
SIC-MADDPG
PMIC-MADDPG
VM3-AC
MASAC
Env name
a|z dim
a
a/2 dim
Predator Prey (3 Agents)
0.00001 2
0.01
0.1
0.1 4
0.1
Predator Prey (6 Agents)
0.0001
0.01
0.1
0.01
0.1
Predator Prey (12 Agents)
0.0001 3
0.1 0.1
0.01 4
0.1
Predator Prey (24 Agents)
0.0001
0.01 0.1
0.01
0.1
Cooperative Navigation
0.0001 2
0.0001 0.01
0.1
0.01
Wildlife Rescue
0.001
0.001 0.1
0.001 2
0.001
2 agents HalfCheetah
0.0001 3
0.1 0.0001
0.1
0.01
6 agents HalfCheetah
0.0001 2
0.1 0.001
0.1 2
0.01
2 agents Walker
0.0001
0.1 0.001
0.1
0.01
Table 4: Selection of 3 on SMAC.
PMIC-RODE
Map name
a a
MMM2
0.0001
0.0001
MMM
0.0001
0.001
2s3z
0.001 0.1
3s vs 5z
0.001 0.01
3s5z.
0.001
0.01
10_vs_ 11m
0.01
0.01
19

---

Under review as a conference paper at ICLR 2022
disubble betem0.0 withed cluurie Q
BRUNHG-MAADDPS MADDPG
betem2 with double
Reward
01
Rewwerd
Figure 21: Ablation experiments of BH and double Q Qn Wildlife Rescue.
I EXPERIMENTS ABOUT MASAC-RELATED ALGORITHMS ON WILDLIFE RESCUE
Through experiments, find that the MASAC-related algorithms not learn positive rewards on Wildlife Rescue environment. Thus we experiment MASAC on Wildlife Rescue environment to find the reason.
There are two differences between MASAC and MADDPG: (*) and double Q mechanism. Firstly, we make adjustments to 3 and find that matter how adjust B, rewards, even if 3 is set to 0.0. Secondly, we change the network architecture of MASAC by removing the double mechanism, then find that the algorithm can get positive rewards. Thus hypothesize that the main reason why MASAC-related algorithms can not learn to cooperate on Wildlife Rescue is caused by double Q mechanism.
not get positive
To further verify our hypothesis, we add double Q to MADDPG and find significant performance degradation and slower convergence which proves that double Q mechanism is the main factor. We give some explanations, the Wildlife Rescue is characterized by high punishment for miss- coordination and low positive rewards. Double prevents overestimation but can lead to underes- timation, thus using double Q might ignores less frequent positive rewards, leading underestima- tion.
20


Reviews:
[{"review_id": "y46qzi5YNs3", "reviewer": "Reviewer_iVbL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n1. The writing of this paper is generally clear.\n2. The motivation of this paper is clear that aims to solves the problem that is observed from the existing state-of-the-art algorithms.\n3. The experiments are sufficient, including the discrete/continuous action cases, different environments/tasks and multiple ablation studies to verify the effectiveness of the proposed framework.\n4. The proposed framework can be incorporated into any MARL algorithms, so it is a general method.\n\n## Weaknesses\n\nThis paper does not have obvious weaknesses, but I have some concerns that the authors need to clarify.\n1. Can the authors give more details about how $\\mathbb{P}\\_{ \\mathcal{S} \\mathcal{U} }$, $\\mathbb{P}\\_{\\mathcal{U}}$ and $\\mathbb{P}\\_{\\mathcal{S}}$ implemented?\n2. In experiments, the authors claim that the results from the first two graphs in Figure 8 shows that the proposed maximizing and minimizing MI can improve the stability. However, the stability of the proposed method is not well, i.e., the variance of the proposed method is much larger than other competitors. Can the authors give more explanations on this?\n3. For the abalation study in Figure 9, the authors did not show the case that PMIC-MADDPG-normal\\_buffer that can directly show the effectiveness of the proposed replay buffer. Can the authors show the result for it?\n4. If I understand correctly, $T\\_{\\omega\\_{1}}$ is designed with the state and action encoders while $T\\_{\\omega\\_{2}}$ is without. Can the authors explain the reason for the inconsistent design for these two models? It is better that the authors can provide the results for MINE without state/action encoders to get rid of the suspect that the performance improvement relies on the encoders rather than the main contribution claimed in the paper.\n\n## Minors\n\nThere are multiple writing typos in the paper.\n\n1. \"Similarly, CLUB approximates the lower bound ...\" where lower bound should be upper bound. \n2. $I(s; \\pi(u|s)$ appearing in multiple places should be $I(s; \\pi(u|s) )$.\n3. Above Equation (3), $D\\_{a}(\\cdot)$ should be $D\\_{u}(\\cdot)$.\n\nThe above are just the ones I found. I urge the authors to check the paper writing again, and revise these writing typos in the discussion stage.", "summary_of_the_paper": "This paper aims to solve the collaboration among agents (that is usually called forced coordination in other literatures). The common method to enhance the correlations between agents by mutual information (MI) could lead to the sub-optimal collaboration. To address this issue, the author propose Progressive Mutual Information Collaboration (PMIC). This new MARL framework is composed of two components: (1) two replay buffers that respectively collect superior trajectories and inferior trajectories; (2) two mutual information estimators that respectively estimate the upper bound and the lower bound of MI that are trained with the data stored in the two replay buffers introduced above. These two estimators (MINE and CLUB) are from the prior works. The novelties of this work are applying these two estimators to collaboration problem for MARL and learning the mutual information that evaluates the correlation among agents constrained within these two bounds. In my view, the idea of appropriately modelling these MI bound estimators to the actual phenomenon in the MARL problem is novel. Seen from the algorithmic framework type, this algorithm add the learned MI term to the reward function that plays the role of auxiliary reward. From the perspective of optimization, this is a kind of implementation of Lagrangian method, to transform the constraints stated in Equation (2) and (3) to a unconstrained terms involved in the objective function. Similar to many prior works, the tuning of the multipliers could be a potential issue. The authors also conduct the ablation studies on these two multipliers and the results seem like it would not affect the performance in the case of Cooperation Navigation.", "summary_of_the_review": "In summary, this paper has some novolty but stays at the application level, though this is not my concern. My main concern is the effectiveness of the main contribution claimed in the paper, i.e. the maximization-minimization MI. I concern that whether the performance improvement mainly comes from the specific architecture $T$ used in the framework.\n\nFor the current version of paper, I can only recommend reject (but marginally below the threshold) due to the weakneses and concerns above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "r4Qxy1BOpik", "reviewer": "Reviewer_5fQS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe proposed method is novel and shows good performance.\n\nWeaknesses:\n1. While the results generally look good, some reward curves in Figure 5, 6, 7, 8, and 9 are not converged. It would be better to compare the converged performance between baselines. \n2. I am not quite following why minimizing the MI in inferior trajectories is helping improve the corporation. Generally, I would suggest more explanations on the motivation of the PMIC.", "summary_of_the_paper": "This paper proposes PMIC a MARL framework for improving multi-agent collaboration through mutual information. PMIC uses two separate buffers to store the superior and inferior trajectories, and uses them to train two estimators for the upper and lower bounds for the mutual information between global state and joint policy in two sets of trajectories. The estimated mutual information bounds are then used as additional rewards for training individual policies. The experiments presented in the paper show the advantage over several baselines on several benchmark environments.", "summary_of_the_review": "I think the submission overall is good for its novelty and experiment results. I only have some minor concerns about the theoretical backup of the proposed method. Thus, I suggest accepting the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gbvnlJ3MgW", "reviewer": "Reviewer_vXVk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-Rebuttal\nI now fully understand the algorithm. Thank the authors for the detailed response. I misunderstood the motivation of using the two bounds. Now the algorithm design looks pretty clear and intuitive to me, which is simply reducing MI on poor trajectories while increasing MI on good trajectories.\n\n**some suggestion on writing**\n1. I think the position of Sec. 3.2 makes it a bit misleading since it is presented after the motivating example (Sec. 3.1).  Sec. 3.1 has already assumed the optimizing objective for cooperation is MI and presents the motivating example based on this assumption. Note Sec 3.2 simply re-stated why MI can be a good optimization objective for learning, which is, in fact, the assumption in the previous subsection! Wouldn't it be more appropriate to justify the assumption before assuming it? This cyclic statement would make the presentation less consistent for the readers (including me) to follow. From the best readability, I would suggest the authors have a separate preliminary/motivation section, which contains the current content of Sec 3.2 first (to first show why optimizing MI is good) and then the content of Sec 3.1 (to show the issues of directly optimizing MI when having multiple patterns).\n\n1. I would still encourage the authors to make the best efforts to add the related work section back to the main paper. This paper is related to a lot of good ideas in existing works. It would be critical to mention them directly. The authors may possibly make the experiment sections a bit more concise and shrink the conclusion section to make some space. \n\n\n=======================================================================\n\n### Strong Points\n1. The proposed algorithm is, to the best of my knowledge, novel and it is indeed interesting to see such a separation of buffer with mutual information computation significantly boosts the performance of a collection of MARL algorithms, **with the assumption that the derivation is correct**.\n2. There are a lot of ablation studies conducted in the experiment section as well as the appendix, which is appreciated. The experiment section is clear and easy to follow.\n\n### Weak Points\nIn general, before moving into more detailed comments, I would strongly encourage the authors to substantially re-organize the paper to make it easier for the readers to follow. Also, most of the arguments and motivations in the paper are very hand-wavy, which makes me very concerned about the correctness of the proposed algorithm. It could be because of the writing issue, but, at least from the current statements in the paper, the correctness cannot be rigorously justified. More comments follow.\n\n1. A related work section would be encouraged.\n\n    Sec. 3.1 and the introduction section have a substantial overlap describing the existing literature, which should be put into a separate related work section. Fig.1 also looks strange as it appears in a motivation section. Fig.1 seems to only serve as a role of toy example showing that PMIC works better than some baselines. This is an experiment result rather than a motivation example! It doesn't make sense to me to motivate an algorithm by simply saying hey it works well. As a research paper, a reader would expect insights before moving into empirical evidence of your algorithm being good. I would appreciate it if the authors could rewrite Sec. 3.1 such that some more concrete examples of how the algorithm is working, for example, intuitively what kind of behavior leads to high MI; what characteristics are those sub-optimal strategies intuitively so we have a sense of what kind of policies we want to avoid; and so on. The current Sec.3.1 is simply a replication of the introduction section. \n\n2. I couldn't find how sec. 3.2 and sec 3.3 are related\n\n    > However, simply maximizing this MI can hinder algorithm learning as discussed above.\n\n    Eq.(1) definitely makes a lot of sense to me and there are existing works that consider similar objectives for MARL learning (not exactly the same objective but somewhat similar, like this one for example https://arxiv.org/pdf/2106.02195.pdf ). But why does this objective hinder learning? Is this really discussed? I have made my best efforts on understanding the discussions in Sec.3.1 but I could find any convincing arguments of how this objective hinders learning. \n\n   > Therefore, we propose a novel framework PMIC to solve the problem\n\n   Okay, even if this objective does have hindered learning, then why does the proposed PMIC algorithm solves tackles the issue in this objective? What is the relationship between Eq(1) and PMIC? To be concrete, for example, Sec.3.1 argues that the policy should be diverse (well, the terminology isn't really appropriate but let's use it for now) as suggested by using the entropy bonus but I couldn't find any corresponding term in PMIC for doing so. So what problem does PMIC solve? \n\n   There are so many hand-wavy arguments in this paper like the two above.\n\n\n3. There are no rigorous justifications on the relationship between high-reward trajectories and MINE/CLUB\n\n    > Intuitively, only the behaviors that obey superior collaboration patterns or inferior collaboration patterns have large MI estimates calculated by MINE or CLUB since $\\mathcal{L}(\\omega_1)$ and $\\mathcal{L}(\\omega_2)$ are optimized.\n\n    - *I think this statement is the foundation of the correctness of the method*. If this statement is correct, the whole algorithm becomes neat. Yes, I do have witnessed empirical analysis supporting this argument in Fig 9 and Fig 15. But, I want to emphasize that those two figures are empirical evidence rather than rigorous derivation. Typically, as a research paper, the conventional way of algorithm development is to first provide rigorous deviation (e.g., mathematical proof; equation analysis and approximation; or explanation and discussion on concrete toy examples), then develop an algorithm that solves the formulated problem and finally present empirical supports. The authors simply present this key argument by using the term ''__intuitively__'', which I failed to catch with my best efforts. This argument is the most critical point of the method and if this argument is wrong, the whole paper becomes problematic. Hence, I strongly encourage the authors to provide formal and rigorous justification or proof to this statement. \n\n    - Here is what confuses me a lot. Note that the two buffers are categorized according to their _rewards_ while both MINE and CLUB are bounds for _mutual information (MI)_. So what is the mathematical connection between rewards and MI? Why should superior strategies have higher MINE value? Can you mathematically prove or justify this? To the best of my knowledge, I cannot find any related proof in the existing literature.\n\n    - MINE is a generic lower bound for MI. So wouldn't optimizing MINE with the entire replay buffer strictly lead to a higher lower bound than only using the positive buffer (you have more data!)? Similarly, why not optimize CLUB using the entire buffer as well? Fig 9 does not include such ablation either. Assuming the correctness of all the arguments of MINE and CLUB, using the entire replay buffer should lead to at least the same or probably even better behavior compared with PMIC since you have both tighter lower bound and upper bound? \n\n### Other Issues\n1. > the joint policy should be diverse enough to avoid being stuck into sub-optimal collaboration.\n\n    Typically the term \"_diverse_\" refers to a set of (or a distribution of) different policies rather than a single joint policy. Also, according to the phrase \"_avoid being stuck into sub-optimal_\", it seems that diverse in fact means that the policy maintains high entropy _throughout the training process_ rather than describing the characteristics of the _final policy_ (the final policy can have low entropy if it converges but during training, it would better have high entropy for the purpose of exploration). So I would suggest the author direct state the purpose of avoiding poor local optimum rather than using the term \"_diverse_\" (you may be encouraged to refer to the literature of learning diverse skills/strategies in deep/MA RL literature)\n\n2. > 3) Meanwhile, given an agent i’s policy on the global state, the uncertainty on other agents’ policies also should be low, thus more easily to achieve better collaboration\n\n    I don't get why this point is necessary when we have 2) already. I couldn't find any corresponding objective that minimizes the intra-agent policy uncertainty in Sec 3.4. I don't know why this point is presented here. \n\n4. minor issues\n    - The predator-prey environment is not a cooperative one, for which the one-sided reward wouldn't become a convincing metric. Are you referring to the cooperative version of predator-prey? I didn't find any content confirming this. \n\n    - I would encourage the authors to have a sub-index for each plot in Fig 8 rather than saying sth like \"the third plot in fig.8\".\n\n    - A possibly related paper also using mutual information formulation for cooperative MARL: https://arxiv.org/abs/1910.05512\n \n    - Fig 15 is okay but it is not really super convincing. I can still observe some color differences in the right plot (I guess the values range from 0.6 to 1.0 maybe?) So probably you just need some tuning of beta or value normalization to also make VM3-AC work. \n", "summary_of_the_paper": "This paper proposes a new exploration scheme for general multi-agent reinforcement learning called PMIC. PMIC maintains two separate buffers by keeping trajectories with high rewards and trajectories with less satisfying rewards. PMIC additionally computes a lower bound of mutual information between state and policy over the positive buffer and an upper bound of the negative buffer. Finally, an intrinsic reward in the form of the lower bound minus the upper bound is introduced as the exploration bonus, which improves a collection of different MARL algorithms on a few benchmark testbeds. ", "summary_of_the_review": "Post-rebuttal: \nAfter carefully re-checking the paper with the feedback from the authors, I have fully convinced by the methods and therefore changed my recommendation to acceptance. \n\n==================================\n\nThe paper proposes a seemingly neat algorithm and presents a good amount of experiments. However, some critical correctness flaws cannot be rigorously justified based on the current content of the paper due to too many hand-wavy arguments. The readability of the paper should be improved as well.\n\nI cannot recommend acceptance based on the current form of the paper. But, I would still believe in the potential of the algorithm based on the experiment results. That being said, the paper still has the potential to become a really strong one if the algorithm can be rigorously (or mathematically) justified.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VFhndk9ngV2", "reviewer": "Reviewer_QvKR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- This paper addresses an important issue of the MI-based MARL framework where the joint policy converges to sub-optimal.\n- The proposed idea, which maximizes MI based on superior samples and minimizes MI based on inferior samples to avoid the joint policy falling into the sub-optimal, is novel.\n- The authors provide various ablation studies for better understanding.\n\n\nWeakness and Questions\n\n-There is a lack of explanation for the collaboration criterion (section 3.2), e.g, as I understand, $s$ is a state variable whose distribution is conditioned on $\\pi$. \n-For the interpretation of the proposed MI form, the authors decompose the proposed MI into the entropy of join policy ($H(\\pi(\\cdot|s))$, the negative entropy of an agent's policy ($-H(\\pi_i|s)$) and conditional negative entropy of the agent's policy given other agents' policies ($-H(\\pi_{-i}|, \\pi_i, s)$). The authors claim that maximizing $-H(\\pi_{-i}|, \\pi_i, s)$ minimizes uncertainty on other agents' policies. However, $H(\\pi_{-i}|, \\pi_i, s)$ reduces to $H(\\pi_{-i}|s)$ since $\\pi_{-i}$ and $\\pi_i$ are independent given the state. Thus, it seems that the interpretation of the proposed MI form, $I(s;\\pi(\\cdot|s))$, is not proper. In this regard, can we say that the proposed MI form causes the collaboration?\n-The authors minimize $I(s;\\pi(\\cdot|s))$ for experiences which have low returns, so it encourages the joint action to explore for inferior experiences. This is because the joint action should be uniform to minimize $I(s;\\pi(\\cdot|s))$. On the other hand, maximizing $I(s;\\pi(\\cdot|s))$ for experiences that have high returns encourages the joint action to exploit for superior experiences. Thus, in my opinion, the effectiveness of the proposed method comes from the further exploration and exploitation strategy rather than the collaboration strategy. To see where the performance gain comes from, it would be great if the authors compare the proposed method with MA-SAC+DPCB and VM3-AC+DPCB.", "summary_of_the_paper": "This paper proposes to maximize the MI between the global state and joint policy for promoting collaboration. To achieve better collaboration, a Dual Progressive Collaboration Buffer (DPCB) which stores superior and inferior samples separately is introduced, and then the proposed method maximizes the lower bound on MI using the samples in superior buffer and minimizes the upper bound on MI using the samples in inferior samples. ", "summary_of_the_review": "-The proposed idea is novel, but the author should provide further explanation on collaboration criterion and further experiments to see where the performance gain comes from. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper includes an interesting idea of pushing towards good, and away from bad, trajectories, in a natural clean way.

The main problem of the paper is one of clarity.  The paper could be written to be more concise and clear, which would allow, for instance, for sufficient space for the figures (which are currently sometimes rather tiny) as well as not having to fiddle with the margins and spaces quite as much as the current submission seems to do (which would be strictly disallowed at most conferences).  The issue of clarity was also clear during discussion, where sometimes multiple rounds of clarifications were needed to allow the reviewers to correctly interpret parts.

For these reasons, I recommend that the authors resubmit a new, cleaned-up, version of the work, with all the changes neatly incorporated.  Then I think this could make for a nice addition to the literature.

I appreciate this will be a disappointment to the authors, but I think ultimately it will make their work more impactful, and longer-lasting.
Decision: Reject