Mutual-Information Regularized Multi-Agent Policy Iteration
Authors: Jiangxing Wang
Deheng Ye
Zongqing Lu
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-Agent Reinforcement Learning

Abstract: Despite the success of cooperative multi-agent reinforcement learning algorithms, most of them focus on a single team composition, which prevents them from being used in more realistic scenarios where dynamic team composition is possible. While some studies attempt to solve this problem via multi-task learning in a fixed set of team compositions, there is still a risk of overfitting to the training set, which may lead to catastrophic performance when facing dramatically varying team compositions during execution. To address this problem, we propose to use mutual information (MI) as an augmented reward to prevent individual policies from relying too much on team-related information and encourage agents to learn policies that are robust in different team compositions. Optimizing this MI-augmented objective in an off-policy manner can be intractable due to the existence of dynamic marginal distribution. To alleviate this problem, we first propose a multi-agent policy iteration algorithm with a fixed marginal distribution and prove its convergence and optimality. Then, we propose to employ the Blahut–Arimoto algorithm and an imaginary team composition distribution for optimization with approximate marginal distribution as the practical implementation. Empirically, our method demonstrates strong zero-shot generalization to dynamic team compositions in complex cooperative tasks.

Mutual-Information Regularized Multi-Agent Policy Iteration
Jiangxing Wang
Deheng Ye Tencent Inc.
Zongqing Lut School of Computer Science Peking University BAAI zongqing 1w@p.ul.ppkeeee. edu. en
School of Computer Science
Peking University jianxxivdstu. pku edu.
dericye@tencent.com
Abstract
Despite the success of cooperative multi-agent reinforcement learning algorithms, most of them focus single team composition, which prevents them from being used more realistic scenarios where dynamic team composition is possible. While some studies attempt solve this problem via multi-task learning in fixed set of team compositions, there is still a risk of overfitting to the training set, which may lead catastrophic performance when facing dramatically varying team com- positions during execution. To address this problem, we propose to use mutual information (MI) as an augmented reward prevent individual policies from rely- ing too much on team-related information and encourage agents to learn policies that are robust different team compositions. Optimizing this MI-augmented objective in an off- off-policy manner can be intractable due to the existence of dynamic marginal distribution. To alleviate this problem, we first propose a multi-agen policy iteration algorithm with fixed marginal distribution and prove its conver- gence and optimality. Then, we propose to employ the Blahut-Arimoto algorithm and an imaginary team composition distribution for optimization with approximate marginal distribution as the practical implementation. Empirically, our method demonstrates strong zero-shot generalization to dynamic team compositions in complex cooperative tasks.
Introduction
The cooperative multi-agent reinforcement learning (MARL) problem has attracted the attention of many researchers for being well-abstracted model for many real-world problems, such as traffic signal control (Wang al., 2021), autonomous warehouse (Zhou et al., 2021), and even AutoML (Wang et al., 2022) the feedback to the machine learning community. In cooperative MARL problem, we aim to train group of agents that can cooperate to achieve a common goal. Such common goal is often defined by global reward function that is shared among all agents. Although fully decentralized
this objective is naturally centralized, we want agents to be able to execute manner. Under such requirement, Kraemer and Banerjee (2016) proposed the centralized training with decentralized execution (CTDE) framework, where centralized critic is learned to evaluate the performance of the joint policy in terms of the global reward and group of decentralized individual policies are learned via the centralized critic to realize decentralized execution.
With centralized critic, multi-agent policy gradient methods directly use it to guide the update of each decentralized individual policy. Based on this idca, a series studies (Lowe et al., 2017: Kuba al., 2022; Yu et al., 2022; Ye et al., 2020) have been proposed with different optimization
Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

techniques for policy improvement. On the other hand, as the centralized critic is used to guide the learning of decentralized individual policies. many CTDE algorithms choose to factorize the centralized critic into decentralized individual utilities via the mixer network. This line of research is called value decomposition (Sunehag et al., 2018). Based on different design choices of the mixer network, a variety of value decomposition methods (Sunehag et al., 2018; Rashid et al., 2018; Wang et al., 2020; Zhang et al., 2021) has been proposed and achieved great success in cooperative MARL problems.
Despite the success of CTDE methods, previous research mainly focuses on training agents under a fixed team composition, which has been shown to exhibit serious overfitting issues (Wen et al., 2022), and leads to catastrophic performance when facing unseen team compositions. One natural way to solve this problem is to introduce multiple team compositions during training to alleviate overfitting. However, due to the existence of the mixer network and the network structure of individual utilities and policies, such a strategy cannot be simply applied to many existing CTDE methods.
REFIL (Iqbal al., 2021) attempts to address this problem and proposes to use multi-head atten- tion (Vaswani et al., 2017) in the mixer network and individual utilities to handle dynamic team compositions. It further proposes an imaginary objective based on random sub-group partitioning to accelerate the training process given a fixed set of team compositions. Although REFIL is able to handle dynamic team compositions, as it is still trained on a fixed set of team compositions, there is still a risk of overfitting to not single team composition, but set of team compositions, which has been demonstrated in several previous studies (Liu et al., 2021; Shao et al.. 2022).
The overfitting issue can be attributed to the lack of robustness in different team compositions. When facing an arbitrary team composition during execution, one agent's ss observed information about team composition can arbitrarily vary from what it experienced during training. If the agent puts too much credit on this highly varied information make decisions, may fail achieve robust behavior. As the variation of team composition is uncontrollable, one way to achieve robust behavior to reduce the reliance on team-related information. Based on this intuition, we propose MIPI (Mutual Informatinn Regularized Multi-Agent Policy Iteration), minimizing the mutual information between the policy of the agent and the ream-related information encourage robust behavior of each agent. Inspired by SAC (Haarnoja et al., 2018), we combine the global reward of environment and mutual information of each agent as a new objective and learn individual policies to optimize them optimization
at the same time. As the incorporation of mutual information imposes a challenge due to the existence of dynamic marginal distribution, we first propose a multi-agent policy iteration algorithm with fixed marginal distribution and prove its convergence and optimality. Then, we propose to utilize the Blihut-Arimoto algorithm (Cover, 1999) and an imaginary team composition distribution for optimization under an approximate dynamic marginal distribution as the practical implementation.
To empirically justify our algorithm, we first evaluate the performance of MIPI in a simple yet challenging matrix game. Compared with the other two basclines using pure environmental reward and pntropy-augmenter reward, using mutual information an augmented reward can help the agent find the policy that can achieve consistent performance across different team compositions. Then, we move to more complicated scenario, StarCraft Micromanagement Tasks. While having the same level of performance in the training set, MIPI achieves better zero-shot generalization results in unseen team compositions during evaluation.
2
Related Work
2.1 Centralized Training with Decentralized Execution (CTDE)
CTDE methods can be categorized into value decomposition and multi-agent policy gradient, de- pending on whether a centralized critic is decomposed or not. For value decomposition methods, a centralized critic decomposed into decentralized utilities through the mixer network. Different mixers have been proposed as different interpretations of the et al., 2018) (IGM) principle or its equivalence, which ensures the consistency between optimal local actions and optimal joint action. VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 201 8) give sufficient conditions for IGM by additivity and monotonicity, respectively. QPLEX (Wang et al., 2020) and FOP (Zhang et al., 2021) take advantage of duplex dueling architecture to guarantee IGM.
(Rashid
2

---

In multi- policy gradient, a centralized critic function is directly used to guide the update of each decentralized individual policy. Most multi-agent policy gradient methods can be considered as an extension of the policy gradient method from RL to MARL.. For example, MAPPDG (Lowe et al., 2017) extends DDPG (Lillicrap et al., 2016), HATRPO (Kuba et al., 2022) extends TRPO (Schulman et al., 2015), MAPPO (Yu et al., 2022) and CoPPO (Wu et al., 2021) extend PPO (Schulman et al., 2017).
2.2 Dynamic Team Composition
While classical CTDE methods mainly focus on fixed team compositions, in real-world applications, agents that can adapt to dynamic team composition preferable. To address this problem, RE- FIL (Iqbal et al., 2021) incorporates multi-head attention (Vaswani et al., 2017) into the networks and further introduces an imaginary objective based random sub-group partitioning accelerate the training process on a fixed set of team compositions. While REFIL learns policies that can handle dynamic team compositions, studies (Liu et al., 2021; Shao et al., 2022) suggest that it generalizes poorly on unseen team compositions. The necessity of training agents that can generalize to unseen team compositions is evident using the automated warehouse as an example, where is very common to add more agents (more agents got purchased) or delete some agents (some agents got broken). Therefore, agents have to deal with different teams in the application scenario, which can not be fully covered during training. In order to learn policies that can adapt to unseen team compositions, many studies choose to sacrifice the requirement of decentralized execution. For example, a centralized agent is assumed in COPA (Liu et al., 2021). which has global view of the environment and coordinates agents by distributing individual strategies. 2022), a communication channel is assumed to elect conductors, so that the corresponding groups are constructed with coodduucoo-fooloowee consensus. Unlike these methods, we do not assume any kind of centralization during execution, such that the ability of decentralized execution is fully preserved. In CollaQ (Zhang et al., 2020), the decentralized utility function is decomposed into two terms: the self-term that only relies on the agent's 's own state, and the interactive term that is related to states of nearby agents. By using an additional MARA loss to constrain the contribution of the interactive term in the decentralized utility function, CollaQ solves the generalization on the dynamic team composition problem with CTDE being preserved. Unlike this method, MIPI uses mutual information to constrain the contribution of team-relate information in the agent's 's policy.
SOG (Shao et al.,
2.3 CntemattionTThorrdde Principles in RL
In the standard RL problem, the objective is to solely optimize the environmental reward. However, in many problems, we not only want to optimize the cumulative rewards but also want the learned policy to exhibit some other properties. Therefore, line of research across single-agent and multi-agent domains has been proposed using different information-theoretic principles for policy optimization. For example, SQL (Haarnoja et al., 2017) and SAC (Haarnoja et al., 2018) incorporate the maximum entropy principle to encourage exploration and diverse behavior. FOP (Zhang et al., 2021) further extends this idea to MARL and proves its convergence and optimality. As the augmented entropy term distorts the original objective in the original MDP, which may lead to undesirable behavior in some scenarios (Eysenbach and Levine, 2019). To solve this problem, DMAC (Su and Lu, 2022) proposes to use the divergence between the current policy and previous policy to replace entropy, which yields a bound of the discrepancy between the converged policy and optimal policy in the original MDP. Using divergence to guide the policy optimization is also very popular in offline RL (Levine et al., 2020), for example, F-BRC (Kostrikov et al., 2021) and ICQ (Yang et al., 2021), where the divergence is used to control the similarity between learned policy and behavior policy. While the entropy and KL divergence can be seen as a measurement of the distance to a fixed policy, the mutual information is about the distance to a dynamic marginal policy. In MIRL (Grau-Moya et al., 2019) and MIRACLE (Leibfried and Grau-Moya, 2020), the environmental reward is combined with mutual information to encourage the learned policy to be close to an optimal prior policy, which is also dynamically learned during the RL process instead of being fixed throughout, Unlike (Grau-Moya al., 2019; Leibfried and Grau-Moya, 2020) that aim at generalized version of SAC in single-agent RL, the purpose of our work is to solve generalization on dynamic team composition in MARL.
Mutual information (MI) has been widely used in previous MARL research for various purpose, e.g.. exploration (Mahajan et al., 2019; Wang et al., 2019; Zheng et al., 2021), coordination (Konan et al., 2021; Kim et al., 2023), individuality (Jiang and Lu, 2021), diversity (Li et al., 2021) and social
3

---

influence (Jaques et al., 2019). Unlike these works that mainly focus on the performance of agents in a fixed team, we focus on the generalization ability of agents over different or even unseen teams. Also, these works mainly try to increase the mutual information between two variables to enhance the dependency between variables. However, in our work, we try to decrease the mutual information between the agent's s policy and team-related information to reduce the dependency between these two variables and avoid overfitting. Being an exception, PMIC (Li et al., 2022) maximizes the MI associated with the superior trajectories and minimizes the MI associated with the inferior trajectories at the same time. However, it still focuses on the training of a fixed team, while our work focuses on the training of dynamic team and the generalization over unseen teams.
3 Background
In this paper, formulate cooperative MARL with dynamic team composition multi-agent Markov decision process (MMDP) with entities. MMDP with entities can be defined by a tuple {E, S, A, U, 7)r & is the set of entities in the environment, S is the set of states, and each state 8 is composed by the state of each entity a = [se]. It is worth noting that except agents A, there are also other entities in the environments (e.g., landmarks, obstacles, agents with fixed behavior). U = U1 X .... x A| is the joint action space, where U, is the individual action space for each agent For the rigorousness of proof, we assume full observability such that each state 8 E S. each agent i receives state 8, chooses an action u, E Ug, and all actions form a joint action U. The state transitions to the next state s' upon according to the transition function P(s' s. u) : S x U x S - [0, 1]. and all agents receive a shared reward r(ss, u) S x U + IR The objective is learn an individual policy T. (u,(s)) for each agent such that they can cooperate to maximize the expected cumulative discounted return, E y'rel, where y [0, 1) is the discount factor. In CTDE, from a centralized perspective, a group of local policies can be viewed as a joint policy Tj (u)(s)) For this joint policy, we can define the joint state-action value function Qj u+) = F1 MM E 7'rt+e/#t, uz]. Note that although we assume full observability for the rigorousness of proof, we use the trajectory of each agent T; Ti (Y x U:)" to replace state s for each agent to settle the partial observability in practice, where Y is the observation space.
Since we are discussing dynamic team composition in this paper, we further denote 5f" as team- unrelated information for agent i (agent's own information), and use s; to denote team-related information that varies along team composition (e.g., information of other agents, landmarks, obsta- cles). Although we assume full observability for each agent (i.c., 8 is the same for all agents), sf and si can be different as the circumstance of each agent per se is different. One can easily conclude that {s s, for each agent i.
4 Method
In this section, we present our method, MIPI, as follows. In Section 4.1, we introduce the mutual information (MT) augmented objective for regularizing the reliance on 8, for each agent. However, due to the existence of the dynamic marginal distribution, direct optimization on this objective can be intractable in practice. Therefore, in Section 4.2, we first discuss a multi-agent policy iteration with fixed marginal distribution and prove its convergence and optimality. Then, in Section 4.3, we discuss how to use an imaginary team composition distribution to achieve an approximate dynamic marginal distribution and how to use the Blahut-Arimoto algorithm (Cover, 1999) to optimize the corresponding objective. Finally, in Section 4.4, we summarize the learning framework of MIPI.
4.1 MI-Augmented Objective
The learning objective of standard MARL can be formulated as follows,
T >r (s u.)
arg max E Jj
Σ
(I)
1=
4

---

where p(su) is the distribution of initial state. We can further rewrite as:
arg max
Σ
Puys (o1)
Begi(ucle))
[7*r(*s un)
where Pay. (8t) is the the marginal distribution over states at timestep t. Recall that the conditional mutual information between and y given 2 can be expressed follows,
log p(rlyy 2)
MI yl=) = p(..))
Ep(aly.e)
p(æ(2)
The conditional mutual information can be used to measure the dependency between variable r and with given. Therefore, goal is to reduce the reliance of T. (u,lss) m. (u,ls, . on s can formulate conditional mutual information as follows,
(u ( S
MI s; (s E p((tte7) (07)
E Eaitadet (o7)
log
o. (2
J. (u, (u,
log
where )((4ls()=-) ( sT). Incorporating the conditional mutual information of agents into the standard MARL objective, we now have the MI-augmented objective used in this paper:
Emi(uelse) [2'(1(8t,))) ut) Σ log Te(ui Ti (u 1 }
arg max Tjs
B Boo (mr)
(3)
s.t. mu(uelltt))
Σ Payi (87, (:sslst ls sT.))
(4)
where the coefficient is used to determine the trade-off between maximizing global reward and minimizing mutual information.
4.2 Multi-Agent Policy Iteration with a Fixed Marginal Distribution
As we can see in (4), the optimization of (3) is highly coupled with a dynamic marginal distribution Ty (u,,ls,)) What's even worse is, this marginal distribution determined by Pxy. (s,,1s);)) which is related to Tjt. making the optimization problem even harder. However, one may notice that, if such a marginal distribution T. (us, is given and fixed, this problem becomes much easier and can be solved in an off- off-policy manner. Therefore, in this section, introduce aaett policy iteration with fixed marginal distribution and prove its convergence and optimality, and in the next section, we discuss how approximate the dynamic marginal distribution by integrating constraints similar to (4) into this multi-agent policy iteration. First, let us define the joint value function Vj and joint state-action value function Oj+ as follows.
>Log log Te(uielse)) |s0 Ty (uuelsst)
(x) = -Eaje
Σ
r (1(( ui) > 1=
070² (s. u) = r(s, u) + Eara [Vi((s)].
where n, (ugu is the fixed marginal distribution for each agent i. With the above definition, we further deduce that:
(e(u, s) TE (uuu(s))
V Erzy.
O (s, u) a Σ log
We can then define the joint policy evaluation operator as
T TTy. (s, u) == r(s, u) + E.r [V)(
(5)
and have the following lemma.
5

---

Lemma (Joint Policy Evaluation). Consider the modified Bellman backup operator my (5) and a mapping Qj : 8 x U I with [U] oc, and define Q+++1 Fa Oj. Then, the sequence Qj: will converge the joint Q-function of Th. as k OG.
Proof. See Appendix A.I.
Using Lemma 1, we can get 2j. for any joint policy Tjt. However, is hard for us to use Qj for individual policy improvement. To solve this problem, many value decomposition methods choose to factorize the joint state-action value function Qj into the utility function Qi of each agent and use Q: to guide the individual policy improvement of me In this paper, we factorize the joint state-action value function into the following form, which is shared by many value decomposition methods (Zhang al., 2021; Su and Lu, 2022; Wang et al., 2023):
01tH (s, = Σ wi(s) QT, u.) + b(s).
(6)
After the evaluation of the joint policy and the decomposition of Qit, we construct the following optimization problem for individual policy improvement.
T/ (1: ny (4: ss
arg max Eyg [ot (8, log
Based on the above optimization problem, we have the following lemma for individual policy improvement.
Lemma 2 (Individual Policy Improvement). Let wew be the optimizer of the maximization problem
in (7), Then, we have Oj (s, u) 2 Q7= oid
(s, for all (8, u) E (SS] x [U] with |U| < Do, where
old (uls) II, n (us(s) and πe (u(s)) II. (n: Te
Proof. See Appendix A.2.
Combining Lemma and 2, we can have the following theorem which proves the convergence and optimality of multi-agent policy iteration with fixed marginal distribution.
Theorem 1 (Multi-Agent Policy Iteration with Fixed Marginal Distribution). For any joint policy Ty. if we repeatedly apply joint policy evaluation and individual policy improvement. Then
the joint policy Tj(uls)) = II Tee (u(u)))) will eventually converge to Tje such that O7," (s, u) 2 2321" (s, for all Tit, assuming |U| < cc.
Proof. See Appendix A.3.
4.3
Approximation for Dynamic Marginal Distribution
With the multi-agent policy iteration above, we can have Qj+ and corresponding Q: for each agent, however, under fixed marginal distribution. In this section, discuss how to approximate the dynamic marginal distribution to decouple Pa,, (s from Tjt, and introduce the Blahut-Arimoto algorithm for the corresponding optimization.
Notice that the original objective (3) comes with a constraint (4). Section 4.2, what we did is to remove this constraint for an easier optimization process. What we are going to do here, is to add similar constraint back. First, consider the meaning of Pry (s, | ), represents the potential team composition given team-unrelated information. Therefore, inspired by REFIL (Iqbal et al., 2021), we randomly partition team composition under = {,,t 8 } into different subgroups, yielding a set of imaginary team compositions and corresponding imaginary distribution pO* (xt) for imaginary team-related information. With this imaginary distribution, propose the approximate objective

---

[(10:]
|4]
i=1 ))
L(os)
Mixer Network
Qila, + u m((e,)) ((ws)
Qu(..u)
11
Q1 )
MLP
( .*)
Critic
Critic [A]
H
W2
(.)
m(ai)
GRU
MLP
H
S
Agent
Agant (A]
H
[s]
MHA
MLP
MLP
Q1
Q/A
Environment
Figure 1: Learning framework of MIPI, where each agent i has three modules: a utility function
Qi ue; 0,), policy (u: od), and marginal policy Ti ; ;w))
for (3) as follows,
Baiiuelee) (7' (r(()(), ut) - a Σ log
Ti Ti(uiolse) Ii (u
arg max Tig
B Bany (ec)
(8)
s.t. me(ueelo)t))
(9)
%
Therefore, we can have the approximate optimization problem for (7) as follows,
arg max Ew? [Q old (s, u:) - a log (u (s)
new
(10)
<T(u: (5
x
s.t. <((u(s))= = Σ p(>)1sp ) sf).
(11)
The objective above exhibits similarities with the rate-distortion problem (Cover, 1999), which could be solved using the Blahut-Arimoto algorithm. Although with the approximation above we break the convergence of Theorem 1, by using the Blahut-Arimoto algorithm we can have the following theorem, indicating the convergence of (10) as shown in Leibfried and Grau-Moya (2020).
Theorem 2 (Convergence of Constrained Individual Policy Improvement). The optimization problem induced by (10) can be solved by iterating an alternate fashion through the following two equations:
T." (uglst) = Σ p(s)1s yo?" (u:ss ,
(12)
क.. (u.s, (u.(ss))aaaa) + (uilst 387) En, क..h (,:|os exp (8, ux)/a))
(13)
where refers to the iteration index. Denoting the total number of iterations M, the presented scheme converges at rate of (1/M) to an optimal policy पद for any given bounded utility function Qi and any initial policy TQ.
Proof. See Appendix A.4.
4.4 MIPI Framework
In Section 4.2 and 4.3, we propose our learning algorithm in theory. In this section, we discuss how to implement algorithm in practice, which can be summarized in Figure 1.
7

---

In MIPI, each agent has a utility function Qi(s, uy; 0,), policy Ta Tz(U; s; di), and marginal policy Tu(u,ls jwy). ;wy)) For joint policy evaluation, with the utilities of agents, we use a mixer network Mixer(:, 8; ④) to get the joint state-action value function Qj as follows,
(s u) Mixer([Q:(s. un; 0.) 8;
(14)
|A| Σ w. (.,(s)((sss..) un 0.) b(s),
(15)
Where (8) 2 0 is positive weight used linearly decompose Qj with the IGM principle being preserved. Same as REFIL, wy (s)s) () is computed via the attention mechanism to handle dynamic team composition. With Qjt. we can update the utilities and the mixer network by minimizing the following TD error:
[(10)) 0) = EDD @i((s, -()r))((s u) 7 (Qii(sQ u²)
(r(ss
[4l E log (ufls'') Te me(u(ss"")
(16)
where D is the replay buffer, Qit is the target network and uf is sampled from the current policy (u; (x; of each agent. To accelerate the training with a fixed set of team compositions, we also incorporate the same imaginary objective based on random sub-group partitioning REFIL for joint policy evaluation.
As we described in Section 4.3, the constrained individual policy improvement is achieved via an iterative update of π. (u,) s; (b) and n (u,u|s7 1w;). For n. (u, ist w:), we update it via the maximum likelihood estimation:
[(w:) ED
E B (o
[Iog [og
(17)
For (u:ls; 6t), update it by minimizing the KL- KL-divergence as follows,
[a(10g [og (u(uss)) Ti n((uIlot)) 2:(3.11)
((d:) = Ep
E Busoi(ud)s)
(18)
Experiments
In this section, we evaluate MIPI in two different scenarios. One is simple yet challenging matrix game, which we use to illustrate how mutual information may help to learn generalizable policies. Then, we evaluate MIPI in a complicated cooperative MARL scenario: StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), comparing against REFIL, AQMIX (Iqbal al., 2021), CollaQ (Zhang et al., 2020) and MAPPO (Yu et al., 2022). More details about experiments, hyperparameters, and the learning curve of each algorithm are included in Appendix B and C. All results are presented using the mean and standard deviation of five runs with different random seeds.
5.1 An Illustrative Example: Matrix Game
We first use matrix game explain how mutual information works in solving generalization problems. In this game, we have two agents, and each of them can take two actions {0, 1} and take of the types {A, B}. During training, train these two agents under team compositions ( (A, B) and (B, A), where the corresponding payoff matrices are shown in Figure ((a) and 2(b). However, during evaluation, we test their performance on team composition (B, B). and have Figure 2(c) as the payoff matrix, which is different from training scenarios.
As we can see in Figure 2(a), 2(a), 2(c), we have different optimal joint actions in different team compositions. However, there exists a generalizable joint action (az = 0, 02 = 0) that can achieve consistent performance regardless of team compositions, even if it is not optimal joint action in any team composition.
In Figure 2(d), we plot the evaluation results of three algorithms during training, which is evaluated on team composition (B, B). These three algorithms are all the same except they receive different re- wards: pure environmental reward, environmental reward combined with entropy, and environmental

---

a2
42
a2
01
Q1
8
10
0
0
8
0
Muszll Ietermation
0
10
0
10
(a) payoff matrix: (A, (b) payoff matrix: (B, A) (c) payoff matrix: (B, B)
learning curves
Figure 2: A matrix game with different team compositions: (a) (b) (c) payoff matrices for different team compositions; (d) learning curves of different methods on team composition (B, B).
Table 1: Final performance on all SMAC maps. MIPI outperforms REFIL, AQMIX, and CollaQ in out of 9 evaluation maps. We bold the best mean performance for each map.
#Agent
Training
Evaluation
Algorithmss
Tasks
3-5
8
MIPI
0.659-10.02
0.453-10.08
0.404-10.062
0.276:10.076
REFIL.
0.674-00.038
0.441++0.100
0.352-+00078
0.236-H0. 103
SZ
AQMIX
0.528:10.004
0.343:10.105
0.291110.084
0.182:10.058
CollaQ
0.508
0.366:10.086
0.314:10076
0.198:10.097
MАPPO
0.206
0.129:10.019
0.148
0.036:10.015
MIPI
0.548+10.032
0.42:10.10 102
0.297
0.261+10.09
REFIL
0.568
0.348400.057
0.009
0.164
CSZ
AQMIX
0.50940.054
0.323
0.216
0.152:20.071
CollaQ
0.4591000061
0.362:10.13
0.267:10.09
0.231+0.0095
MАPPO
0.248
0.12::0.029
0.06
0.054
MIPI
0..023
0.405:1-0.05
0.447--0.041
0.467+10.067
REFIL
0..057
0.437:10.118
0..19
0.224
MMM
AQMIX
0.0016
0.447:10.043
0.34410.071
0.251
CollaQ
0.5891000027 0.5131:0.07
0.423:10.026
0.286110.083
MAPPO
0.289
0.32:10.102
0.25::00063
0.275#0.009
reward combined with mutual information. As we can see in Figure 2(d), with the help of mutual information, agents are able to resist the temptation of overfitting the specific team composition and learn behavior that can generalize across different team compositions.
5.2 StarCraft Micromanagement Tasks
5.2.1 Performance
Further, we evaluate MIPI on SMAC with the map designed by Iqbal et al. (2021). We customize three different types of scenarios (SZ. CSZ, and MMM) based on this map for our experiments. In S7. scenarios, agents can take two different unit types, in CSZ. and MMM, agents can take three different unit types. During training, the maps randomly initialize 3-5 agents and the same number of enemies at the start of each episode. During the evaluation, we use 6-8 agents and 6-8 enemies. Results are shown in Table In general, MIPI outperforms the baselines in 8 out of 9 evaluation scenarios. When the evaluation scenario is similar to the training scenarios, the gap between MIPI and other baselines is relatively small, whereas, in the evaluation scenario that is very different from the training scenarios, the gap between MIPI and other baselines becomes larger. In terms of the training performance, REFIL achieves the best result in all three scenarios, as it does not consider overfitting issue at all. However, the performance of MIPI is still at the same level as REFIL, which indicates that MIPI can achieve better zero-shot generalization without sacrificing the performance on the training set.

---

Table 2: Final performance on all SMAC maps. MIPI compared with ablation baselines. We bold the best mean performance for each map.
#Agent
Training
Evaluation
Algorithmss
Tasks
3-5
8
MIPI
0.659
0.453
0.404
0.276
SZ
Value
0.621
0.336:10.075
0.275:10.114
0.154:10.052
Entropy
0.105:10.035
0.02411.001
0.015:10.011
0.01
MIPI
0.048
0.42:10.10 102
0.297:10.112
0.2611:0.09
CSZ
Value
0.542:00.099
0.368:10.083
0.207
0.172:10.112
Entropy
0.316::0.04
0.237:00000
0.07610.0041
0.066:10.041
MIPI
0.548
0.495=t0.054
0.447
0.467::0.067
MMM
Value
0.545
0.505-+0.058
0.3008
0.319:E0. 105
Entropy
0.2034
0.24:0.05
0.16-H0.064
0.075++0.044
5.2.2 Ablation
Although MIPI uses the random sub-group partitioning as in REFIL, an actor-critic structure, whereas REFIL uses only critic. Therefore, one may question whether the improved generalization of MIPI is due the introduction mutual information, or simply due the introduction of the actor. To eliminate such concern, we build two ablation baselines, Value and Entropy, where all other perspectives are the same as MIPI, except they use pure environmental reward and harrepy-augmented reward, respectively. As we can see in Table 2, MIPI also outperforms these two baselines, which demonstrates the importance of MI-augmented reward MIPI.
Conclusion
In this paper, we propose MIPI, MI-regularized multi-agent policy iteration algorithm improve the generalization ability of agents under unseen team compositions. We first prove the convergence and optimality of algorithm given fixed marginal distribution, then propose use imaginary distribution to approximate the dynamic marginal distribution to better approximate the original objective and incorporate the Blahut-Arimoto algorithm into the multi-agent policy iteration to optimize this approximate objective. We evaluate our algorithm in complex coordination scenarios, SMAC, and demonstrate that MIPI can achieve better zero-shot generalization results, without sacrificing the performance on the training set.
One potential limitation of this work is the introduction of approximation distorts the original mutual information augmented objective and breaks the convergences of multi-agent policy iteration. One possible solution to this problem is seek alternative solutions using on-policy optimization methods (Schulman et al., 2015, 2017; Grudzien et al., 2022) to optimize the augmented objective.
Acknowledgments and Disclosure of Funding
This work was supported in part by NSF China under grant 62250068 and Tencent. The authors would like to thank the anonymous reviewers for their valuable comments.
10

---

References
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Benjamin Eysenbach and Sergey Levine. If maxent the answer, what is the question? arXiv preprint arXiv:1910.01913, 2019.
Jordi Grau-Moya, Felix Leibfried, and Peter Vrancx. Soft q-learning with mutual-informattin regularization. In International conference on learning representations, 2019.
Jakub Grudzien, Christian A Schroeder De Witt, and Jakob Foerster. Mirror learning: A unifying framework of policy optimisation. In International Conference Machine Learning, pages 7825-7844, 2022.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep -neegy-baseed policies. In International conference on machine learning, pages 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
Shariq Iqbal, Christian A Schroeder De Witt, Bei Peng, Wendelin Böhmer, Shimon Whiteson, and Fei Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4596-46000 PMLR, 2021.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z. Leibo, and Nando De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference machine learning, pages 3040-3049. PMLR, 2019.
Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference Machine Learning, pages 4992-5001. PMLR, 2021.
Woojun Kim, Whiyoung Jung, Myungsik Cho, and Youngchul Sung. A variational approach to mutual information-based coordination for multi-agent reinforcement learning. arXiv preprint 00X10:2303.00451,, 2023.
Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay, Iterated reasoning with mutual information in cooperative and byzantine decentralized teaming. International Conference on Learning Representations, 2021.
llya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pages 5774-5783. PMLR, 2021.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94 22222944222 2016.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International conference on learning representations, 2022.
Felix Leibfried and Jordi Grau-Moya. Mutualinfformaion regularization in markov decision pro- cesses and actor-critic learning. In Conference on Robot Learning, pages 360-373. PMLR, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. multi-agentrrinfoceemmnn learning. Advances in Neural Information
Celebrating diversity in shared multi-agent reinforcement Processing Systems, 34:3991-4000 2021.

---

Pengyi Li, Hongyao Tang, Tianpei Yang, Xiaotian Hao, Tong Sang, Yan Zheng, Jianye Hao, Matthew Taylor, Wenyuan Tao, and Zhen Wang. Pmic: Improving multi-agent reinforce- ment learning with progressive mutual information collaboration. In International Conference on Machine Learning, pages 12979-12997. PMIR, 2022.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International conference on learning representations, 2016.
Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandkumar. Coach-player multi-agent reinforcement learning for dynamic team composition. In International Conference on Machine Learning, pages 6860-68777 PMLR, 2021.
Ryan Lowe. Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent environments. Advances in neural information
actor-critic for mixed processing systems, 30, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019.
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 43044. PMLR, 2018
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farguhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.0400 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference machine learning, pages 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Self-organized group for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:5711-5723, 2022.
Kefan Su and Zongqing Lu. Divergence-regullriize multi-agent actor-critic. In International Conference on Machine Learning, pages 20580-20600 PMLR, 2022.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decommosition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized execution: Multi-agent conditional policy factorization. In International conference on learning representations, 2023.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent g-learning. In International Conference on Learning Representations, 2020.
12

---

Tong Wang, Jiahua Cao, and Azhar Hussain. Adaptive traffic signal control for large-scale scenario with cooperative group-base multi aulti--geet reinforcement learning. Transportation research part C: emerging technologies, 105:10304,, 2021.
Tonghan Wang, Jianhao Wang. Yi Wu, and Chongjie Zhang. Influence-based multi-agent exploration. arXiv preprint arXiv:1910.055/10 0,,20,,,.512... 2019.
Zhaozhi Wang, Kefan Su, Jian Zhang. Huizhu Jia, Qixiang Ye, Xiaodong Xie, and Zongqing Lu. Multi-agent automated machine learning. arXiv preprint arXiv:2210.09084 2022.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi- agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35: 1659--1116,,,, 2022.
Zifan Wu, Chao Yu, Deheng Ye, Junge Zhang, Hankz Hankui Zhuo, et Coordinated proximal policy optimization. Advances in Neural Information Processing Systems, 34:26437-26448, 2021.
Yigin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang. Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu, al. Towards playing full moba games with deep reinforcement learning. Advances in Neural Information Processing Systems, 33:621-632, 2020.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo cooperative multi-agent games. Advances in Neural Information 2022.
Processing Systems, 35:24611-22
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International Conference Machine Learning, pages 12491- 12500. PMLR, 2021.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang. Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Multi-agent collaboration via reward attribution decomposition. arXiv preprins arXiv:2010.08531, 2020.
Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34:3757-3769, 2021.
Tong Zhou, Dunbing Tang, Haihua Zhu, and Zequn Zhang. Multi-agent reinforcement learning for online scheduling in smart factories. Roboties and Computer-Integratee Manufacturing, 72: 102202, 2021.
13

---

A Proofs
A.1 Proof of Lemma!
Lemma (Joint Policy Evaluation). Consider the modified Bellman backup operator TT (5) and a mapping Of: : - R. with [U] < oc, and define Q Fa Of. Then, the sequence Qj will converge to the joint Q-function of Ty. as k - oc.
Proof. First, define the augmented reward as:
Ty (1ullss')
Ty (s, u) = r(s, ui - a Es'
B Σ log
(uilo,
Then, rewrite the update rule as:
Qj u) t- Tri (s, u) + Rg myn (ir , zz)).
Last, apply the standard convergence results for policy evaluation (Sutton and Barto, 2018).
A.2 Proof of Lemma
Lemma 2 (Individual Policy Improvement). Let π.!ew be the optimizer of the maximization problem
in (7), Then, have Qit 1e (s, u) 2 Q]: (s, f all (s, u) E (S] x |0] with |U|| co, where old jt. (uls) = II, old (us(s)) and new jj. (u(s) 11. new (uslouull ())
Proof. As x.e optimizes (7), can have:
Exgew [o (s, us) a log N..e" (u, (ui
2 E pipld Th (s. a log
T: Told(us (us (s) (u, 85
(19)
n7 (u 185t)
Since assume that:
01tH us) Σ wi(s) (s, u4) + b(s).
we can have:
log n.ew (w: 8)
E Euw new
Qft Tit (8
it
Ti (ugl $
n (uiss)
Σ wi( Q (s, wi) + b(8)) 2 log
Eu-rpon now
Jy (2; (2111
wi(() Q oold (s,
Tupo(us)s)
E Euiwatee
(s,u,) u;) aolog (u |s
+ b(s))
nold (us
E ourooa
(x) Q.: (8. log
+ b(())
n, (1: 1s
Q Rit (8,u) Σ log (u+ls)
E Eurrpit
ne. (u (sxt)
V. jt
(s),
(20)
where the inequality is from plugging in (19).
We assume क.. (u:j (s) and m. (u:ls)) ) to be e-soft policy (Sutton and Barto, 2018) to avoid the log term being
undefined.
14

---

Last, considering the modified Bellman equation, the following holds:
Told
(8, u) r(8, + EB
V
(s')
T.e (u(ls')
S r1(8, u) tyB
E Buewayto
[ot (s',
Σ log
Dew
<Ofe
(8, u),
where we have repeatedly expanded Qj²" and the inequality in (20).
the RHS by applying the modified Bellman equation
A.3 Proof of Theorem
Theorem 1 (Multi-Agent Policy Iteration with Fixed Marginal Distribution). For any joint
policy Tjs if we repeatedly apply joint policy evaluation and individual policy improvement. Then the joint policy (ij(uss) = II (u(uis)) will eventually converge to Tgo such that Q (s. u) 2 Qjth (s, for all Tit, assuming |U] co.
Proof. First, by Lemma 2, the sequence (mftt monotonically improves with Oj a
jj
Since
the augmented reward is bounded, then 2 is bounded. Thus, this sequence must converge to some Tj.- Then, at convergence, we have the following inequality:
x.. (2;1s) 3.. (u;
2 E,, [Q77 (s, u.) log (u; (s)
Em; Q (s, u.) a log
VT, * ग.. Tu,
Using the same iterative argument in the proof of Lemma 2, get Qj. u) 2 Oite (s, u.) for all (8, u) E |S] x (U]. That is, the state-action value of any other policy Tjt is lower than or equal to that of the converged policy Tjt- Therefore, Tze is the optimal joint policy.
A.4 Proof of Theorem
Theorem 2 (Convergence of Constrained Individual Policy Improvement). The optimization problem (10) can be solved by iterating in an alternate fashion through the following two equations:
T (u.lst) = Σ p(5))s)) yo?" (uilst st)
TI"' (uslo,) exp (is(QQ((suu)/o)
T (uglst .5)
En, म." (u:l s exp(Qi(s,
where refers to the iteration index. Denoting the total number of iterations M, the presented scheme converges at a rate of O(1M) to an optimal policy पद for any given bounded utility function Qi and any initial policy nQ.
Proof. First, we notice that for a fixed Tr(uslst, ;8{), we can have its optimal marginal as constrained in (1
(u:ls())) = Σ ((5) (uslss s().
Then, for fixed marginal oy (uu(ss))) we can have the optimal ग (u:ls; s;) by solving (10) via standard variational calculus:
T(ulst)) exp((I(ss. ui)a)
<(ugss7 S,
Σn, (u,ls, ) exp(Q)(s,
Lastly, with the above two equations, we can apply Theorem Leibfried and Grau-Moya Moya (2020) to finish our proof.
15

---

B
Experiment Settings and Implementation Details
B.1 Matrix Game
In the matrix game, we use learning rate of 3 10 for all algorithms. For the algorithm that uses mutual information as the augmented reward, we set the number of Blahut-Arimoto iterations to 1. For algorithms that use mutual information and entropy as the augmented reward, we fix o as 0.5. The batch size used in the experiment is 64. Critics and polices used in the experiments consist of one hidden layer of 64 units with ELU non-linearity. For the mixer network, we use hypernetwork similar to QMIX (Rashid et al., 201 8), except no non-linearity is used. The environment and model are implemented in Python. All models are built by PyTorch and are trained via I Nvidia RTX 1060 GPU to conduct all the experiments. Each experiment takes roughly I hour.
B.2 SMAC
In StarCraft II, we use a learning rate of 5 x 10-4 for all algorithms. The structure of the critic network and the mixer network of MIPI are the same as REFIL (Iqbal et al., 2021) except no oon-aiieyy is used in the mixer of MIPI. The number of Blahut-Arimoto iterations is set to 4 for MIPI in this experiment. The policy network of MIPI shares all layers with the critic network except the last layer of the policy network being a different fully-connected layer. The target networks will be updated once every 200 training episodes for all algorithms. The temperature parameters a and a; are fixed as 0.03 in SZ and CSZ and fixed as 0.1 in MMM for MIPI and Entropy. For REFIL, AQMIX, and CollaQ, we use their default settings. For CollaQ. as the original implementation is based on different SMAC environment where the entity-level observation is not available, we re-implement CollaQ with minimum changes to adapt the entity-level observation based on the framework provided in REFIL to ensure fairness of comparison. For MAPPO, as there is no published version of MAPPO for dynamic team compositions, we choose to implement MAPPO following Papoudakis et al. (2021), with additional attention modules used in the policy and the critic handle dynamic team compositions. The environment and model are implemented in Python. All models are built by PyTorch and are trained via mixture of 4 Nvidia A100, 4 RTX 3090, and 1 RTX 2080 TI GPUs to conduct all the experiments. Each experiment takes to 32 hours depending on the algorithms and scenarios. Our implementation of MIPI is based on REFIL (Iqbal et al., 2021) with MIT license. It is worth noting that, although we assume full observability for the rigorousness of proof, the trajectory of each agent is used to replace state for each agent imput to settle the partial observability in all SMAC experiments.
B.3 Resource Collection
In Resource Collection, we use a learning rate of 5 x 10-4 for all algorithms. The structure of the critic network and the mixer network of MIPI are the same as REFIL (Iqbal et al., 2021) except no non-linearity is used in the mixer of MIPI. The number of Blahut-Arimoto iterations is set to I for MIPI in this experiment. The policy network of MIPI shares all layers with the critic network except the last layer of the policy network being a different fully-connected layer. The target networks will be updated once every 200 training episodes for all algorithms. The temperature parameters a and ag are fixed as 0.05 in Resource Collection for MIPI. For REFIL, AQMIX, and CollaQ, we use their default settings. For CollaQ, the original implementation is based on different SMAC environment where the entity-level observation is not available, we re-implement CollaQ with minimum changes to adapt the entity-level observation based on the framework provided in REFIL to ensure fairness of comparison. For MAPPO, as there is no published version of MAPPO for dynamic team compositions. we choose to implement MAPPO following Papoudakis et al. (2021), with additional attention modules used in the policy and the critic to handle dynamic team compositions. The environment and model are implemented in Python. All models are built by PyTorch and are trained via 4 Nvidia RTX 3090 GPUs to conduct all the experiments. Each experiment takes roughly 20 hours. Our implementation of MIPI is based on REFIL (Iqbal et al., 2021) with MIT license. is worth noting that, although we assume full observability for the rigorousness of proof, the trajectory of each agent is used to replace state s for each agent as input to settle the partial observability in all SMAC experiments. As suggested by previous research (Liu et al., 2021; Shao et al., 2022), random sub-group partitioning does not work well in Resource Collection, therefore we choose not to use for MIPI in this experiment.
16

---

C Training performance on SMAC
In this section, we additionally provide the learning curves of all algorithms used in Section 5.2. As we can see from Figure 3, these algorithms achieve similar training performance, except Entropy.
3-5sz symmetric
3-5csz symmetric
3-5MMM symmetric
1.0
MIPI
Value
Value
0.8
Value
Entropy
Entropy
Entropy
REAIL
REFIL
0.6
REFIL
AQMIX
AQMIX
AQMIX
CollaQ
CollaQ
0.4
CollaD
0.0
0.0
0.4
0.8
1.6
2.0
0.0
0.0
0.4
0.0
12
1.6
2.0
0
0.4
0.8
1.2
2.0
Figure Learning curves of all the methods in SMAC, where the unit of x-axis is 1M timesteps and y-axis represents the win rate of each map.
D More Experiments
D.1 Resource Collection
In this section, We further evaluate MIPI Resource Collection, which is scenario in terms the level of collaboration used by COPA (Liu et al., 2021). During training, the will have 6-8 agents. We plot
challenging
map randomly initializes 3-5 agents, and during the evaluation, the curve training and evaluation performance Figure 4. As we can see, MIPI outperforms the baselines by large margin, which indicates that MIPI also perform well in scenarios requiring strong collaboration.
resource collection(train)
resource, colleectiooo((ellaatiin)
300
MIPI
MIPI
REFIL
400
REFIL
AQMIX
AQMIX
200
CollaQ
300
CollaQ
MAPPO
MAPPO
200
100
100
0
0.0
0.4
0.8
1.2
1.6
2.0
0.0
0.4
0.8
1.2
1.6
2.0
Figure Learning curves of all the methods in Resource Collection, where the unit of x-axis is 1M timesteps and y-axis represents the return.
D.2
Ablation on Alpha
In this section, We further include the ablation study on the impact of alpha. We train MIPI with different alpha on 3-5 agents scenarios and evaluate their performance 6-8 agents scenarios. We use alpha=oc to represent the case where team-related information is completely removed (it is worth noting how this is different from actually set alpha=co in MIPI), Results are summarized Table 3. As we can see, unless alpha is set unreasonably, MIPI can always achieve better generalization ability without sacrificing the training performance. It's worth noting that alpha=oc outperforms Value here, which further indicates that reducing the dependency on team-related information promotes generalization, even when the team-related information completely removed. However, this strategy not widely effective and sacrifices the training performance too much in some cases (see MMM), which further leads to a decay in both training and evaluation. contrast, our method uses alpha to control the degree of dependency ream-related information, which provides more flexibility.
17

---

Table 3: Final performance on all SMAC maps. MIPI is compared with the ablation baseline. We bold the best mean performance for each map.
#Agent
Training
Evaluation
Alpha
Tasks
3-5
7
8
0 (Value)
0,62120,042
0.33610.007
0.275
0.154
0.01
0.672:1.002
0.394::0.065
0,365:0.006
0.261
0.03 (MIPI)
0.659#0.02
0.453++0.08
0.404=00006
0.276=000076
SZ
0.05
0.643:00002
0.447:10.062
1..054
0.313110.069
0.1
0.475:10.073
0.277:10.12.
0.26=20093
0.146:10.075
0.5
0.175:10.053
0.056:10.015
0.129:0.026
0.04310.021
00
0.546:10.069
0.429:000036
0.38910.038
0.221
O (Value)
0.542:10.059
0.368:10.083
0.207
0.172:10.112
0.01
0.592:10002
0.378+00003
0.364.10.03
0.304
0.03 (MIPI)
0.508+00002
0.4
0.297=0.115
0.261#0.09
CSZ
0.05
0.506-10.046
0.417:+0.02
0.223+00.094
0.192+0.001
0.1
0.3441:0.05
0.218:10116
0.11310.079
0.09810.086
00
0.50620.076
0.368110.04
0.30910.056
0.27
O (Value)
0.545++0.048
0.505-+0.058
0.391+00.88
0.319+00. 105
0.05
0.59+00.008
0.59:40.053
0.526-00.055
0.426:+0.152
MMM
0.1 (MIPI)
0.508:10.023
0.405:10.0054
0.447:10.041
0.467:10.067
0.5
0,277=00004
0.199++0.094
0.18
0.139=00005
0.396+00.07
0.4324:0.018
0.383+0.041
0.315+00006
692 symmetric
7sz symmetric
8sz symmatric
MIPI
REAIL
0.4
REFIL
0.30
0.25
AQMIX
CollaQ
0.3
CollaQ
0.3
MAPPO
MAPPO
MAPPO
0.2
0.1
0.05
0.0
0.0
0.00
0.8
2.0
0.0
2.0
6cs2 symmetric
7csz symmetric
Besz symmetric
0.35
MIPI
0.4
RE FIL
REFIL
0.30
AQMIX
0.3
AQMIX
CollaQ
CollaQ
MAPPO
MAPPO
0.2
0.15
0.1
0.10
0.1
0.05
0.0
0.00
0.0
0.4
1.2
2.0
0.0
2.0
6MMM symmeetric
7MMM symmetric
8MMH symmetric
0.5
MIPI
RE FIL
0.5
RETIL
REFIL
AQMIX
0.4
AQMIX
Collaq
0.4
CollaQ
CollaQ
MAPPO
0.3
MAPPO
0.3
MAPPO
0.1
0.1
0.0
0.0
0.0
0.8
0.8
1.6
2.0
0.0
1.2
2.0
Figure 5: Learning curves of all the methods in SMAC, where the unit x-axis is 1M timesteps and y-axis represents the win rate of each map.
D.3 Performance on Higher Level Driver
When working on the follow-up project of this paper, we noticed that REFIL. can achieve better generalization results in MMM with higher-level NVIDIA driver without any code-level change.
18

---

The results are shown in Figure 5, where all algorithms are trained in single platform that REFIL achieves better results. As we can see, although REFIL achieves better generalization results in some cases, MIPI can still outperform these baselines in terms of both speed and final performance by properly setting a and og (0.01 for sz, 0.015 for esz and 0.05 for MMM).
19


Reviews:
[{"review_id": "L8NKp3ErJB", "reviewer": "Reviewer_NEb7", "summary": "To address the issues of overfitting in Multi-Agent Reinforcement Learning (MARL) and dynamic team formation, this paper proposes a new approach of using mutual information regularization as intrinsic rewards to prevent individual policies from excessively relying on team-related information. Experimental results demonstrate that the proposed method outperforms relevant baselines on several SMAC tasks.", "questions": "1. Could the authors perform algorithm comparisons on tasks that require higher levels of collaboration?\n2. What would be the impact on performance if the team-related information is completely removed in SMAC tasks?\n3. are the network architecture and basic hyperparameters of LIMI the same as those of the baselines?\n4. Can the authors provide experimental analyses regarding the hyperparameter alpha? Additionally, what are the specific settings of alpha for each task?", "limitations": "See Weaknesses.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper is easy to read.\n2. The method proposed in this paper, which utilizes mutual information to address the dynamic team formation problem, appears to be novel. However, I have some questions that need to be addressed by the authors.", "weaknesses": "1. This paper aims to reduce the reliance of agents on team-related information in decision-making. However, one potential issue may exsit with this approach. It may result in fewer cooperative behaviors among agents, leading to a greater emphasis on individual performance than team collaboration. As a result, this method may not perform well in tasks that require high levels of cooperation. I suggest that the authors evaluate their algorithm on tasks provided by COPA, which require strong collaboration.\n\n2. The introduction of mutual information constraints introduces some human knowledge, as it requires decomposing the observed information into team-related and unrelated components.\n\n3. The paper lacks some related works. In section 2.3 \"Information-Theoretic Principles in RL,\" the authors should provide a more extensive overview of relevant MI-based MARL literature [1-10], which is more closely tied to collaboration.\n\n---\n\n[1] Maven: Multi-agent variational exploration.\n\n[2] A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning.\n\n[3] An Adaptive Entropy-Regularization Framework for Multi-Agent Reinforcement Learning.\n\n[4] Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning.\n\n[5] Influence-Based Multi-Agent Exploration.\n\n[7] Episodic multi-agent reinforcement learning with curiosity-driven exploration.\n\n[8] The Emergence of Individuality.\n\n[9] PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration.\n\n[10] Celebrating Diversity in Shared Multi-Agent Reinforcement Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This paper aims to reduce the reliance of agents on team-related information in decision-making. However, one potential issue may exist with this approach. It may result in fewer cooperative behaviors among agents, leading to a greater emphasis on individual performance than team collaboration. As a result, this method may not perform well in tasks that require high levels of cooperation. I suggest that the authors evaluate their algorithm on tasks provided by COPA, which require strong collaboration.\n\n> Could the authors perform algorithm comparisons on tasks that require higher levels of collaboration?\n\n> What would be the impact on performance if the team-related information is completely removed in SMAC tasks?\n\n> Can the authors provide experimental analyses regarding the hyperparameter alpha?\n\nThanks for your valuable suggestion. We have carried out additional experiments on the task that requires strong collaboration and ablation studies for the questions above. Please refer to the common response for more experiment results. Additionally, we would like to point out that reducing the reliance on team-related information will not lead to a greater emphasis on individual performance than on team collaboration. This is because, in cooperative MARL problems, agents can only receive a joint reward signal which reflects the joint performance of all agents. There is no individual reward signal for agents to optimize. Therefore, even if we reduce the reliance on team-related information, they will still be optimizing the joint reward, which is about the performance of the whole team. The reason to reduce the reliance of agents on team-related information is just to prevent agents from overfitting to team compositions used as the training set.\n\n> The introduction of mutual information constraints introduces some human knowledge, as it requires decomposing the observed information into team-related and unrelated components.\n\nThanks for pointing this out. As we mentioned in our paper Line [151-154], we will decompose observed information into team-unrelated information $s_{i}^{+}$, which includes information of agent per se and other non-agent information, and team-related information $s_{i}^{-}$, which includes information of other agents. To achieve this kind of decomposition, agents only need to distinguish different entities in the environment, which is commonly assumed in previous research, see [1] and [2].\n\nHowever, your comment still makes sense in scenarios where such decomposition is not trivial. For example, when we have an image as the agent's input. In such cases, we can further relax the constraint of information decomposition, where team-unrelated information $s_{i}^{+}$ will only include information of agent per se, and team-related information $s_{i}^{-}$ will include all other information. With such a decomposition, agents only need to distinguish themselves from other information, which is much easier than the previous one. We use this setting in our Resource Collection experiment, where $s_{i}^{+}$ of LIMI is just the information of the agent itself, and $s_{i}^{-}$ includes information about other agents and landmarks. As suggested by the result, LIMI can still achieve better performance than other baselines with this relaxed way of information decomposition. This kind of information decomposition is also consistent with our setting in SMAC, as there are no non-agent entities in SMAC. So, there is no need to redo the experiments in SMAC.\n\n> The paper lacks some related works. In section 2.3 \"Information-Theoretic Principles in RL,\" the authors should provide a more extensive overview of relevant MI-based MARL literature [1-10], which is more closely tied to collaboration.\n\nThanks for your comment. We will add corresponding discussions of these papers in our related works section. In our previous version of related works, we mainly focus on research where mutual information (and other informational measurements) does not simply serve as additional loss terms, but alters the form of policy iteration. But we will surely add more texts and citations in the later version of our paper based on your recommendations, as these works also account for a large part of MARL research where informational measurements are used. Unlike these works that mainly focus on the performance of agents in a fixed team, we focus on the generalization ability of agents over different or even unseen teams. Also, these works mainly try to **INCREASE** the mutual information between two variables to enhance the dependency between variables (except PMIC). However, in our work, we try to **DECREASE** the mutual information between the agent's policy and team-related information to reduce the dependency between these two variables and avoid overfitting.\n\n> Are the network architecture and basic hyperparameters of LIMI the same as those of the baselines?\n\nYes, as we implement our algorithm based on REFIL. The network architecture of the agent and mixer, also the basic hyperparameters of LIMI are the same as REFIL. Except as required by our theory, we remove the non-linearity of the hypernetwork produced by the mixer.\n\n> Additionally, what are the specific settings of alpha for each task?\n\nAs we mentioned in Line [468-469] in Appendix, $\\alpha$ and $\\alpha_i$ are fixed as 0.03 in SZ and CSZ and fixed as 0.1 in MMM for LIMI.\n\n[1] Randomized entity-wise factorization for multi-agent reinforcement learning, Iqbal et al., ICML 2021.\n\n[2] Self-Organized Group for Cooperative Multi-agent Reinforcement Learning, Shao et al., NIPS 2022.\n\n---\n\n**We hope the above response can address the concerns of the reviewer. If the reviewer still has questions, please feel free to discuss with us! Thanks again to the reviewer for providing us with helpful suggestions and feedback.**"}, {"review_id": "Qf2MOXJ9P9", "reviewer": "Reviewer_KG7B", "summary": "This paper proposes a method which use the mutual information as intrinsic reward to prevent the agents over-rely on its collaborators. Meanwhile, the proposed method can encourage the agents to learn robust policies in different team compositions. Moreover, this paper utilizes Blahut–Arimoto algorithm and an imaginary team composition distribution for optimization. The experiments show the effectiveness of the proposed method", "questions": "1.\tMutual information is widely used in MARL. What is the main difference or improvement of the proposed one compared with the related works[1-4]?\n2.\tHow the model will perform compared with MAPPO?\n\n[1] Celebrating diversity in shared multi-agent reinforcement learning. 2021.\\\n[2] Influence-Based Multi-Agent Exploration. 2019.\\\n[3] Social influence as intrinsic motivation for multiagent deep reinforcement learning. 2019\\\n[4] The emergence of individuality. 2021.\n", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper proposes a novel method for improving multi-agent policy iteration.\n2.\tExperiment results on several scenarios in SCII show the efficiency of the proposed LIMI.\n", "weaknesses": "1.\tSome important baselines are missed. Since the authors have mentioned them in the related work, the new published and common used baselines (e.g. MAPPO, HATRPO) should not be neglected.\n2.\tPerformance on more challenging scenarios iof different benchmarks deserve to be verified.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Some important baselines are missed. Since the authors have mentioned them in the related work, the new published and common used baselines (e.g. MAPPO, HATRPO) should not be neglected.\n\n> Performance on more challenging scenarios of different benchmarks deserve to be verified.\n\n> How the model will perform compared with MAPPO?\n\nThanks for your valuable suggestion. We have carried out additional experiments on more challenging scenarios and included MAPPO as a baseline in our experiments. Please refer to the common response for more experiment results.\n\nHowever, as you may have noticed, there is no published version of MAPPO for dynamic team compositions. We choose to implement MAPPO following [1], with additional attention modules used in the policy and the critic to handle dynamic team compositions as our baseline in our experiments.\n\n> Mutual information is widely used in MARL. What is the main difference or improvement of the proposed one compared with the related works[1-4]?\n\nThere is a key difference between our work and other MARL methods using mutual information. In our work, we aim to **DECREASE** the mutual information between the agent's policy and team-related information to reduce the dependency between these two variables and avoid overfitting. In other works, they mainly try to **INCREASE** the mutual information between two variables to enhance the dependency between variables. Of course, there is no right or wrong here, we just try to use mutual information to solve different problems.\n\n[1] Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks, Papoudakis et al., NIPS 2021.\n"}, {"review_id": "R3JmVPuXd6", "reviewer": "Reviewer_KNmz", "summary": "The paper presents a MARL algorithm that utilizes an SAC-based auxiliary mutual information reward regularizer to generalize learned decentralized multi-agent policies to different team compositions. The method has been evaluated against related baselines and in two simple and complex domains, demonstrating a relatively superior performance. At the current state of the paper, I have a few concerns and questions:\n\n- There’s no clear definition of “team composition”, although understanding this terminology is pivotal for a reader. It’s not immediately clear what “composition” means in this context. I recommend adding a formal definition early on in the introduction.\n\n- Related to the comment above, lack of such formal definitions makes it hard to understand the general, real-world motivation behind the solution presented here. Note that this is different from the technical motivation which is already provided. I recommend trying to give a real-world running example of a multi-agent scenario where such solution would be necessary.\n\n- The approach and ideology presented here seems to be closely connected to InfoPG or MOA (Konan et al, 2022 and Jaques et al, 2019), and etc., where specifically in InfoPG (Konan et al, 2022) the notion of mutual information is leveraged through action-conditional policy distributions to help agents of a cooperative team ignore bad agents, i.e., bad information received from other agents for instance in a BGP problem. Yet I don’t see a mention of such works or discussion in this paper. For example, what problems can LIMI handle that InfoPG cannot? Therefore, the literature review under Section 2.3 seems weak and unconvincing. \n\n[1] Konan, Sachin G., Esmaeil Seraj, and Matthew Gombolay. \"Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming.\" International Conference on Learning Representations. 2021.\n\n[2] Jaques, Natasha, et al. \"Social influence as intrinsic motivation for multi-agent deep reinforcement learning.\" International conference on machine learning. PMLR, 2019. \n\n- Directly adding a mutual information regularizer to the reward to “avoid relying too much on team information” or to minimize (as stated in the paper) sounds counter-intuitive for “cooperative” MARL where we do want agents to have higher mutual information of each other’s policy. Intuitively, the more an agent knows about its teammates actions and policies, the better decision it can make regarding its own action and policy. Also, methods like InfoPG ([1] above) indirectly maximize or minimize the mutual information between the policies of agents in a team through relying on positive or negative environment feedback and cooperativity of agents rather than manually forcing agents to “know less about each other” as suggested here. How do you justify this counter intuition and how do you balance?\n\n- I might be wrong, but from Fig 1 and Eq 16-17, it seems like you would need access to s+ and s- (i.e., relevant and irrelevant information)? How would you identify/distinguish such information?\n\n- Given that a major part of the contribution here is the zero-shot generalization and that LIMI doesn’t improve training performance significantly over existing benchmark, I suggest adding more results from more complex environments to showcase the potential. \n\n- Please try to spell-out acronyms when you first define. It increases readability and understandability of your paper.\n\nAt current states I vote weak rejection, although the algorithm seems to be sound and working. I need to see more discussions and revisions as suggested above, as well as suggested by my fellow reviewers before changing my vote. I’d be happy to increase my score when authors satisfactorily addressed the comments.", "questions": "See above", "limitations": "One potential limitation is discussed. No potential societal impact are discussed. I think the provided discussions are relevant and satisfactory.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "See above", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> There’s no clear definition of “team composition”... \n\nAs we described in Line[136-138], we assume each state $s$ can be decomposed into the state of entities $s_e$, and entities include agent entities and non-agent entities. So the term \"team composition\" here simply means the composition of agent entities, which may vary in terms of numbers and types. Thanks for pointing out this potentially misleading part, we will add corresponding texts in the later version of our paper for this.\n\n> I recommend trying to give a real-world running example of a multi-agent scenario where such solution would be necessary.\n\nThanks for your comment. Including a running example surely eases the understanding of our paper. If we understand your question correctly, you are talking about the necessity of training agents that can generalize to dynamic team compositions. We can use the automated warehouse as an example, where it is very common for you to add more agents (more agents got purchased), or delete some agents (some agents got broken). Therefore, agents have to deal with different teams in the application scenario, which can not be fully covered during training. To solve this problem, training agents that can generalize to different teams becomes a necessary topic.\n\n> The approach and ideology presented here seems to be closely connected to InfoPG or MOA....\n\nThanks for your comment. In our previous version of related works, we mainly focus on research where mutual information (and other informational measurements) does not simply serve as additional loss terms, but alters the form of policy iteration. However, we agree that we should also cover previous works like InfoPG and MOA, as they also account for a large part of MARL research where informational measurements are used. Therefore, we will add corresponding citations and texts in our related work section to emphasize the difference between our work and previous research in this line. \n\nWe think the main difference between our work and InfoPG or MOA is that we focus on the generalization ability of agents over different teams, whereas both InfoPG and MOA focus on the performance of a fixed team. Also, InfoPG requires communication between agents to achieve action conditioned policy, whereas LIMI does not assume such a communication channel. In MOA, authors proposed to use agent modeling to avoid direct communication between agents. However, as the agent modeling module requires supervised training using the actions of other agents, therefore it may not work well in the generalization setting where the behavior of agents in unseen teams is not predictable.\n\n> Directly adding a mutual information regularizer... sounds counter-intuitive for “cooperative” MARL... Also, methods like InfoPG ([1] above) indirectly maximize or minimize the mutual information relying on environment feedback rather than manually forcing agents to “know less about each other”... How do you justify this counter intuition and how do you balance?\n\nThanks for your comment. It is true that by knowing much about others, agents can make better decision and achieves a higher level of coordination. However, this is only true if we focus on just a single team (or a fixed set of teams) where agents can overfit each other at an arbitrary level without damaging the performance. When it comes to generalization, overfitting becomes a problem and leads to degenerated performance, which is common in deep learning algorithms [1]. Therefore, as we focus on the generalization ability of the agent's policy, we use MI as a regularizer to prevent agents from overfitting to the training set, which is similar to using the L2 norm to avoid the neural network from overfitting to the training set. Except, in our case, the overfitting is in terms of the dependency among agents rather than the parameters of the neural network. From this perspective, our method can be viewed as an instantiation of the regularization method in MARL, which becomes less counter-intuition. \n\nAgain, same as those regularization methods, we have to balance the learning progress and the level of regularization. In our case, we use $\\alpha$ to balance the trade-off between learning progress (reward) and regularization (MI).\n\nAlso, as we focus on the generalization ability, where we may have to deal with unseen teams. It is hard to utilize the environmental feedback in the training set as it may vary a lot between the training team and the test team. However, extrapolating this environment feedback and using it for better regularization is indeed a promising direction, and we may work on this in our future work. But this is beyond the scope of this paper.\n\n> It seems like you would need access to s+ and s- (i.e., relevant and irrelevant information)? How would you identify/distinguish such information?\n\nAs we mentioned in our paper Line[151-154], we will decompose observed information into team-unrelated information $s_{i}^{+}$, which includes information of agent per se and other non-agent information, and team-related information $s_{i}^{-}$, which includes information of other agents. To achieve this kind of decomposition, agents only need to distinguish different entities in the environment, which is commonly assumed in previous research, see [2] and [3].\n\n> Given that a major part of the contribution here is the zero-shot generalization and that LIMI doesn’t improve training performance significantly over existing benchmark, I suggest adding more results from more complex environments to showcase the potential.\n\nPlease refer to the common response for more experiments.\n\nAdditionally, we will correct the problem of acronyms. \n\n[1] An overview of overfitting and its solutions, Ying Xue, JPCS 2019.\n\n[2] Randomized entity-wise factorization for multi-agent reinforcement learning, Iqbal et al., ICML 2021.\n\n[3] Self-Organized Group for Cooperative Multi-agent Reinforcement Learning, Shao et al., NIPS 2022.\n"}, {"review_id": "iJ4sRYueVC", "reviewer": "Reviewer_b1Qn", "summary": "This paper propose a new method: LIMI, which utilizes mutual information as a regularizer to help MARL algorithms better face to the variation of team decomposition and generalize on unseen tasks. Both detailed theory proof and practical details are provided.", "questions": "Will the code be released later?", "limitations": "No obvious negative societal impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, both detailed proof and practical details are provided.\n- Good empirical results on SMAC: better zero-shot generalization than other methods.", "weaknesses": "Main issue:\n- More environments needed: SMAC is very sensitive to hyper-parameters. Therefore more results on other environments are needed to make the claims in the paper persuasive (e.g. smacv2, MAMuJuCo).\n\nMinor issues:\n- Where is the 'Ignore' of 'Learning to Ignore' in this paper?\n- Two papers[1][2] should be cited as they used value decomposition in off-policy gradients method.\n\n[1]  Dop: Off-policy multi-agent decomposed policy gradients\\\n[2]  Facmac: Factored multi-agent centralised policy gradients", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> More environments needed: SMAC is very sensitive to hyper-parameters. Therefore more results on other environments are needed to make the claims in the paper persuasive (e.g. smacv2, MAMuJuCo).\n\nThanks for your valuable suggestion. We have carried out additional experiments on more scenarios to further verify our method. Please refer to the common response for more experiment results.\n\nHowever, as in the default setting of smacv2, team composition can only vary in terms of agent types, whereas in our smac experiments, following existing research [1][2][3], team composition can vary in both types and numbers, so we believe that our smac setting is more challenging than smacv2.\n\nAlso, the released version of MAMuJoCo does not support the setting of dynamic team composition. Since the number of joints is fixed for each robot, changing the team that controls the robot is not available. Although we can reformulate MAMuJoCo completely to allow a robot controlled by different teams (for example, a 12 joints robot can be controlled by one 6-joint agent and two 3-joint agents, and it can also be controlled by four 3-joint agents), it requires a lot of efforts and is not easy to build such an environment from scratch. Thus, MAMuJoCo is currently out of consideration.\n\n> Where is the 'Ignore' of 'Learning to Ignore' in this paper?\n\nIn our method, agents will learn to properly ignore team-related information to avoid overfitting. Such ignorance is achieved by reducing the mutual information between the agent's policy and team-related information.\n\n> Two papers[1] [2] should be cited as they used value decomposition in off-policy gradients method.\n\nThanks for your comment. We will add corresponding citations in our related works section.\n\n> Will the code be released later?\n\nWe will release our code once our paper is accepted.\n\n[1] Randomized entity-wise factorization for multi-agent reinforcement learning, Iqbal et al., ICML 2021.\n\n[2] Coach-player multi-agent reinforcement learning for dynamic team composition, Liu et al., ICML 2021\n\n[3] Self-Organized Group for Cooperative Multi-agent Reinforcement Learning, Shao et al., NIPS 2022."}]

Author rebuttal:
We would like to thank all reviewers for their time and efforts to review our paper and give insightful comments that could further improve our work. 

As several reviewers suggest including more experiments, we are now working on this. We plan to include experiments on Resource Collection (an MPE scenario used in COPA, which requires strong collaboration), and new baseline MAPPO in all scenarios, also ablation on alpha coefficient (including alpha=$\infty$ stands for the case where team-related information is completely removed). However, due to the limited computation resource, we can only release them one by one. We will keep working on them during the rebuttal and discussion to improve our work.

**1. Resource Collection**

We further evaluate LIMI on Resource Collection, which is a more challenging scenario in terms of the level of collaboration used by COPA [1]. During training, the map randomly initializes 3-5 agents, and during the evaluation, we will have 6-8 agents. For LIMI, $\alpha$ and $\alpha_i$ are fixed as 0.05. As suggested by previous research [1,2], random sub-group partitioning does not work well in Resource Collection, therefore we choose not to use it for LIMI in this experiment.

We plot the curve of training and evaluation performance in the attached pdf file. As we can see, LIMI outperforms the baselines by a large margin, which indicates that LIMI can also perform well in scenarios requiring strong collaboration.

**2. MAPPO**

As shown in the figure in the attached pdf file, MAPPO underperforms other methods in Resource Collection. We also include MAPPO as a new baseline in SMAC experiments, results are shown in the following table. In short, MAPPO does not perform very well in the setting of dynamic team composition.

| Tasks/#Agent | Algorithms |       Training         |                        |       Evaluation       |                        |
|:------------:|:----------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|
|              |            |          3-5           |           6            |           7            |           8            |
|      SZ      |    LIMI    |$0.659\pm0.02$          |$\textbf{0.453}\pm0.08$ |$\textbf{0.404}\pm0.062$|$\textbf{0.276}\pm0.076$|
|      SZ      |    REFIL   |$\textbf{0.674}\pm0.038$|$0.441\pm0.103$         |$0.352\pm0.078$         |$0.236\pm0.103$         |
|      SZ      |    AQMIX   |$0.528\pm0.044$         |$0.343\pm0.105$         |$0.291\pm0.084$         |$0.182\pm0.058$         |
|      SZ      |    CollaQ  |$0.588\pm0.03$          |$0.366\pm0.086$         |$0.314\pm0.076$         |$0.198\pm0.097$         |
|      SZ      |    MAPPO   |$0.256\pm0.01$          |$0.129\pm0.019$         |$0.148\pm0.031$         |$0.036\pm0.015$         |
|--------------|------------|------------------------|------------------------|------------------------|------------------------|
|              |            |          3-5           |           6            |           7            |           8            |
|      CSZ     |    LIMI    |$0.548\pm0.032$         |$\textbf{0.42}\pm0.102$ |$\textbf{0.297}\pm0.112$|$\textbf{0.261}\pm0.09$ |
|      CSZ     |    REFIL   |$\textbf{0.568}\pm0.027$|$0.348\pm0.057$         |$0.229\pm0.053$         |$0.164\pm0.06$          |
|      CSZ     |    AQMIX   |$0.509\pm0.054$         |$0.323\pm0.096$         |$0.216\pm0.101$         |$0.152\pm0.071$         |
|      CSZ     |    CollaQ  |$0.459\pm0.061$         |$0.362\pm0.13$          |$0.267\pm0.099$         |$0.231\pm0.095$         |
|      CSZ     |    MAPPO   |$0.248\pm0.037$         |$0.12\pm0.029$          |$0.06\pm0.028$          |$0.054\pm0.013$         |
|--------------|------------|------------------------|------------------------|------------------------|------------------------|
|              |            |          3-5           |           6            |           7            |           8            |
|      MMM     |    LIMI    |$0.548\pm0.023$         |$0.495\pm0.054$         |$\textbf{0.447}\pm0.041$|$\textbf{0.467}\pm0.067$|
|      MMM     |    REFIL   |$\textbf{0.605}\pm0.057$|$0.437\pm0.118$         |$0.329\pm0.171$         |$0.224\pm0.163$         |
|      MMM     |    AQMIX   |$0.501\pm0.036$         |$0.447\pm0.043$         |$0.344\pm0.071$         |$0.251\pm0.089$         |
|      MMM     |    CollaQ  |$0.589\pm0.027$         |$\textbf{0.513}\pm0.07$ |$0.423\pm0.026$         |$0.286\pm0.083$         |
|      MMM     |    MAPPO   |$0.289\pm0.097$         |$0.32\pm0.102$          |$0.25\pm0.063$          |$0.275\pm0.098$         |
|--------------|------------|------------------------|------------------------|------------------------|------------------------|

**3. Ablation on alpha**

To be updated during discussion once the results are available.

---

[1] Coach-player multi-agent reinforcement learning for dynamic team composition, Liu et al., ICML 2021

[2] Self-Organized Group for Cooperative Multi-agent Reinforcement Learning, Shao et al., NIPS 2022.

Meta-review: The paper is about the use of mutual information as an intrinsic reward to prevent the overfitting of specific team decompositions and generalize over unseen tasks.
Although the reviewers raised some concerns about some possible side effects of the proposed approach, the lack of some related works and relevant baselines in the empirical evaluation, and some presentation issues.
Nevertheless, thanks to the authors' clarifications, the reviewers had most of their issues solved, and they found a consensus on acceptance.
The authors need to modify their paper to incorporate into the paper all the clarifications they provided during the rebuttal period and follow all the reviewers' suggestions to improve the final version of the paper.
Decision: Accept (poster)