A Mutual Information Duality Algorithm for Multi-Agent Specialization
Authors: Stefan Juang
Qiyang Cao
Yuan Zhou
Ruochen Liu
Nevin Zhang
Elvis S. Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-agent
Reinforcement Learning
Mutual Information
Duality
Policy Gradient
Social Graph

Abstract: The social behavior change in a population has long been studied as an essential component of multi-agent learning. The learning of behavioral change not only involves reinforcement learning (RL), but also be measured against the general population with mutual information (MI). The combination of RL and MI led us to derive MI optimizations from policy gradient. With MI as multi-agent's optimization objective, we discover that the dual properties of MI can result in distinctly different population behaviors. From MI maximization that maximizes the stability of a population to MI minimization that enables specialization among the agents, the dual of MI creates a significant change in a population's behavioral properties. In this paper, we propose a minimax formulation of MI (M\&M) that enables agents specialization with stable regularization. Empirically we evaluated M\&M against the prior SOTA MARL framework, and analyze the social behavior change in performance, diversity, and the stability of their social graphs. 

Under review as a conference paper at ICLR 2023
A MUTUAL INFORMATION DUALITY ALGORITHM FOR MULTI-AGENT SPECIALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The social behavior change in a heterogeneous population is an essential study of multi-agent learning. The interactions between unique agents not only involves the optimization for single agent learning, agents' behavioral changes also depend on the mutual similarity and dissimilarity between the agents. Our study provides theoretical derivation of the policies interactions under the formulation of joint policy optimization. We discover that joint policy optimization in a heterogencous population can affect the population behaviors through mutual information (MI) maximization. In our study, we introduce a minimax formulation of MI (M&M) that optimizes the population behavior under MI minimization against the joint policy optimization. The main findings of our paper show that MI minimiza- tion can minimize the behavioral similarity between agents and enable agents develop individualized policy specialization. Empirically M&M demonstrates a substantial gain in average population performance, diversity, and narrows the performance gap among the agents.
INTRODUCTION
From the success of multi-agent game plays [(OeeAAl et al. 2019), (Vinyals et al. 1. the field of heterogeneous multi-agent learning is being actively studied in areas such Al game design (Contributors] 2022). embedded IoT' (Toyama et al. (02T). and the research for future Human- AI interaction. With the unique physique and specialized character attributes in a heterogeneous population, the ability to optimize policies for specific purposes is essential. The difficulty as well as the goal of the heterogeneous population learning research to search and define a general learning algorithm that optimizes an agent population such that the uniqueness of each agent can be fully utilized within the population. To approach the problem, our research aims to understand the symmetric and asymmetric social behavior changes that occur during the population learning through mutual information formulation.
To study the learning and behavioral change, multi-agent learning has formed two branches of stud- ies. In the simplest form of multi-agent RL, individualized learning is performed to learn separate behavioral policies for each agent. These prior works include Independent Q-Learning (IQL) (Tan 1993). Policy Space Response Oracle (PSRO) (Lanctot et al., 2017). and AlphaStar (Vinyals et al. 2019. However, the empirical success of these prior approaches have excluded knowledge sharing. Since the training is done independently, one agent's 's learned experiences do not transfer to another. The individualized behaviors result in high redundancy of re-exploration and low socialized behav- iors among the agents. On the contrary, joint policy optimization is proposed as a solution to the listed problems. Joint policy optimization utilizes single conditioned policy learn the diverse skillset, and the character attributes of the population via distributed RI. optimization. Through population experiences sharing and joint policy optimization, a single conditioned policy network can learn set of generalized policy skills that transferable across the different agents. Notable examples include OpenAlFive (OpenAl et al. 2019), HAPPO (Kuba et al., (202I) and NeuPL (Liu) et al. 2022 that optimize the multi-agent behaviors under the expected population accumulated rewards.
Through knowledge sharing and joint policy optimization, population learning in interactive games has benefited from the increased learning efficiency, and learning generalization of agents' social behaviors. Our research focuses the latter, where analyze the cause of social behavior change.

---

Under review as a conference paper at ICLR 2023
By analyzing heterogeneous population with pairwise interactions, we discover that the joint policy optimization of a two-player competitive game converges to MI maximization among the agents. Unfortunately, MI maximization optimizes the commonality in population. We found that in heterogeneous population where agents are individually unique, MI maximization benefits agents that have characteristies close to the population norm, but severely degrades the performance of unique agents with character attributes dissimilar to the average population. To address the drawback of MI maximization, our research's main contribution novel minimax MI (M&M) formulation of population learning that enables individual agents to learn specialization from the dual formation of MI maximization and minimization.
BACKGROUND AND RELATED WORK
Multi-agent learning is a broad field of study that not only covers the necessary intelligence to achieve individual agent reward maximization, but also the social behavior change when agents interact with other agents. Prior researches in population learning have studied the performance of the agent population in competitive and cooperative environments.
2.1 COMPETITIVE BEHAVIORS LEARNING
In competitive environment, the goal of the multi-agent research is to utilize competition to op- timize the performance of an agent population. One approach to perform the learning iteration is individualized learning. Each agent improves its policy through learning best-response (BR) against the prior iterations of agents. The iterated elimination dominated strategies optimizes a population of policies under Game Theory. Prior studies such as (Jaderberg et al. 2018). PSRO, PS-TRPO (Gupta et al., (017)) (Vinitsky et al. 2020) and Alphastar utilizes different variants of self-play (Heinrich et al. 2015) to learn competitive Nash Equilibrium behaviors for a population. The different variants addresses the stability (TRPO constraint), robustness (adversarial population) and diversity (leagues of policies pretrained on human data) of individualized learning.
In contrast, joint policy optimization framework proposes by (Foerster et al. 2016) and (Lowe al. 2017) optimizes the population with Centralized Learning Decentralized Execution (CLDE). The joint optimization enables common skill transfer across policies. The commonality among the agents can be learned once and the learned behavior can be utilized by different agents of population. This form of joint policy optimization optimizes the population as one-body problem with Mean Field Theory (Yang et al,, 2018). Under Mean Field Theory, the variations of individual agents can be averaged. The modeling of the population behaviors are reduced from many many-body problem to an one-body problem. Prior works include MADDPG, HAPPO (Kuba et al.) 2021
OpenAlFive (OpenAl et al., (209)) and NeuPL. (Liu et al., 2022), where OpenAlFive and NeuPl. have both further developed efficient graph solvers based on (Shoham & (ccyton-Brwwn 2008) to
optimize the social graphs of the population. The graph solver I' optimizes the match pairing of the agents (x.y) with weighted edges E(r.y), The objective of F commonly optimize the policy learning to be robust against adversarial exploitation or agents with the most similar performance strengths.
2.2 COOPERATIVE BEHAVIORS LEARNING
To develop social behaviors of cooperation, prior studies have proposed auxiliary rewards and reg- ularization of mutual information as part of the objective function. This includes OpenAlFive's 's team spirit reward, (Chenghao al. 202T)'s Q-value, (Cuervo Alzate 2020)'s PPO, and (Ma- hajan et al.) 2019 's latent variable regularization of MI maximization. While the above studies suggest that learning of cooperative social behaviors require auxiliary modifications with mutual information, (Dobbe al. (017)) shows an interesting analysis the distortion rate of the joint policy optimization versus the individualized learning with MI. The study shows that even without auxiliary modification, there is significant stability difference in distortion rate between the joint optimization and individualized learning. This issue has shown to negatively impact cooperative learning.
2

---

Under review as a conference paper at ICLR 2023
2.3 A NEW PERSPECTIVE ON EXISTING WORK: FROM POLICY GRADIENT MUTUAL INFORMATION
In this section first show that policy gradient (Sutton al. 1999) in multi-agent joint optimization can be reduced to MI maximization of an agent population. We derive the learning objective of multi-agent learning as the following.
To model the learning dynamic of a two-players competitive zero-sum game with Player x and Player y, we let [a)s) be a population joint policy. Given a game state s, players act according to the policy [a)s) and let a" and a denote the actions taken by Player x and y. We define Qss, a", a²), V(s), Adv(s, a", aa"a") as the centralized learning Q value, and advantage functions respectively, where Adv(s, a", a Q(,, a", a²) (s). The objective of the policy optimization to perform maximization of the expected return (() from episodes of self-play. The joint policy gradient of a single step optimization is then defined by:
Marimize ((0 E E E (duo(s, a", a²)) (1)5(")ls) (a)(a)))))
(I)
To analyze the behavioral change of the players, derive Eq(3)'s gradient to inspect the direction of policy change. We take the derivative with the product rule and apply the log trick (Williams) 1992) to derive the expanded form of the joint gradient:
(0) = E E El Adu(s, aa,aa"". ,aa)
s at at
* [ag((a)))) (s))) Volog (a")s))))) + (1 (s) (a")s) (olag(1)((a"ss)) [s)]]]
(2)
Here we expand the Advantage function to Q(s,a", aa,)) V(s). With (a*))) and IT" specific to each side of and y, we assume they are independent. By applying the probability of independence and the logarithmic product rule (A.6), the joint gradient can be defined by:
= """ [I x.y) (a* a² a>1s]]
= a²
Volog (x (a* a² [s)
-IV ( (a" a a*|s]] * Velog II (a*)s)) + Velog (a s))ll
(3)
The derivation of E3)3) shows components of mutual information formulation. By taking T steps of joint policy optimization, the integral of Eq(3) removes Vo from the equation. The convergence result of the joint policy optimization is weighted variant of mutual information maximization shown below Eq(4). This derivation suggests that the long-term behavioral change between two competitive players will converge to mutual behaviors maximization.
w( v) T(X;Y) =EE[[w(x,) (x, (xx
(x,x)))xx,)) (Pog(Xx(x)) + (og(((
(4)
In a heterogeneous population of agents { i, ii... N}, we extend the two-player game (X,Y) to all pairings of unique agents, the expected joint gradient is then denoted
N
R E (x * (X) + HYY) - [XI Y)]]
(5)
N
X
The key finding from Eq(3) Eq(5) that through the accumulated gradient update, the maxi- mization of J(Ø) results in MI maximization in the behavior of a heterogencous population. We find that the MI maximization formulation can impose a constraint on heterogeneous population. Particularly agents with character attributes dissimilar to the average population cannot realize their own competitive advantage when their behavioral policies are subject to MI maximization constraint with the average population. An iterative BR variant of joint policy optimization is more difficult to derive the population-leve behavior change during the joint policy optimization. For the BR variant, Eq(5) approximates only the population-leved behavioral change both sides of the policy response in a symmetric two-players game are near convergence. i.c. the joint policies utilized by the two players approximately the same.
3

---

Under review as a conference paper at ICLR 2023
3 M&M: MINIMAX MUTUAL INFORMATION SPECIALIZATION
We introduce M&M as a MI minimax population learning that aims to address the drawback of joint policy optimization. In particular, propose minimization of MI to optimize disjoint set of policies against the MI maximized joint policy optimization. This process enables individual agents specialize behavioral policies relative the average population. We denote the Generalists the joint policy that maximizes MI, and can be utilized on the general population. We define Specialists disjoint set of policies that minimize MI against the Generalists. Specialists' behavioral policies deviate from the population norm through learning behavioral policies that align with agent's 's innate attributes.
3.1 PROBLEM FORMULATION
We consider non-Markov two-player zero-sum Game (X,Y) with heterogeneous population of size N {i, ii... N J. The game is defined by (0, A, R) where O the observation space, S fully observable game. The discounted
O x O is the joint observation of the two players return for each player is defined as Rs = -r= Yrr, where y is the discount factor y E [0,1).
J. We denote the Generalists as {11-- 9-- as an e Nash Equilibrium (NE) policy solution reached by the joint policy optimization after I iterations of population self-play optimization. Here every player receives an expected payoff within e of NE as a numerical relaxation of NE. We define and initialize the Specialists as of disjoint policies from replication of the Generalists ( JN= < {1g-- g= where { J training iteration.
denotes a set of N disjoint policies parameterized by ib on the O-th
3.2 CONDITIONAL MUTUAL INFORMATION MINIMIZATION
For each of games, we formulate M&M one-vs-all BR policy optimization of Specialist against the population of the Generalists in a zero-sum two-players game. Given a game state s, players act according their own policy. Let a and a denote the actions taken by a Specialist k and a Generalist g in game. We define Q(s, a a9) and V(s) as the Q and value functions individually for each Specialist. The objective of the policy optimization is to maximize the expected return J (() while minimizing MI with the gradient. In each optimization step we fix the parameter of the Generalists and treat the joint policy as part of the fixed e-NE solution. We then define the effects of a part of the environment and allow the disjoint policies to condition the MI maximization joint policy, and perform MI minimization. Formally, we define the conditioned policy gradient for Specialist as:
N Σ Ell Ell (ss,a" a²) Ta
Marimize ((bb)
g=d (n.a0)
- The (a* (a*1s)]
(6)
By treating ITg. (a*|s) as the average population's 's fixed c-NE response, a Specialist k's policy gra- dient optimization can be simplified to learn BR against stationary environment. The derivative of the gradient w.r.t. di can be defined as:
N
V S J(
Σ
E
E
(Qssaa,a ad) TT (a*ls) V Vellog(T (a (a*ls))
N
at (e,00)
(> * Tipe (a*1s) Volon(T (a (a*ls))]
(7)
w(X,Y) ((x;)) - -
[[(X) H(X))
(8)
By taking T' steps of ((b maximization, the integral of E7)) maximizes the negative of a weighted conditional MI w.r.t. the Generalists denote in Eq(8). Alternatively, the optimization is equivalent to
4

---

Under review as a conference paper at ICLR 2023
optimize Tu to minimizes its mutual policy response with respect to IIg. by weighting the individ- ual agent's 's Q-value gain against the value-function estimation conditioned on the MI maximizing Generalists population.
N
N
N N
w (KxxXgg) * 1(Xx:
=
(x:: (H))) + HX)) (9)
N
N
g=
The overall expected social behavior change over time when averaging the disjoint policies' gradi- ents is shown in Eq(9). w(Xx; (Xxx;; Yg) denotes the weighting of Q(s, a .a)) V(s ( and the full formulation shows minimization effect of MI between every Specialist the Generalists' joint policy optimization. Utilizing the dual properties of mutual information, M&M optimizes the Spe- cialists population as individual agents that each specializes according to an agent's innate attributes. The benefit of specialization over joint policy optimization is that rather than relying common skill transfers for policy improvement, Specialist learns a unique policy that best aligns with its characteristics. This makes the population social interactions more diverse and enhances the performance of agents with out-of distiiiribttiin character attributes. We show an algorithmic imple- mentation of M&M policies optimization in AA.7
3.3 SOCIAL GRAPH OPTIMIZATION
In our population learning of two-players competitive game, we have implemented a graph solver both optimize and analyze the social interactions between vertices of agents' policies. Let F be defined as a graph solver, and each agent's 's policy denotes graph vertex. The directional weighted edge between two vertices defines the interaction between two vertices as a two-players game. The direction of the edge denotes the relationship between the pairs of vertices as dominance or domi- nated, and the weight of each edge denotes the probability of the competitive outcome for vertex as U The initialization of all the edges are set to 0.5 as 50% chance of winning losing for each pair of vertices. After each batch finished games, the weights and directions of the edges involved updated according to the game outcomes. Based on the weights between the vertices, F updates the match distribution of vertex by defining Σş as the probability distribution of vertex & forms an interactive game other vertices. We define the optimization objective of F to prioritize the sampling of matchmaking based on the exploitability of vertex k relative to other policies. Due to the probabilistic sampling of concurrent game interactions, the sampled adversary opponents form a set of mixed strategies we denote as a.
EXPERIMENTS
Empirically we evaluate the specialization of agents on the Mobile Fighting Game: Naruto Mobile. We compare and contrast the Generalists and Specialists population on advantage analysis, social graph connectedness, behavioral diversity, and agent performance. We discuss the advantage and social graph for each agent to demonstrate how the social behavior change can be largely influenced by the methods of policies optimization. Additionally, our analysis on behavioral diversity and agent performance show that M&M optimizes agents' policies for specialization. By reducing MI, agent population can benefit from behavioral diversity and performance gain.
4.1 INTRO TO NARUTO MOBILE GAME MECHANICS
Game mechanics: Naruto Mobile a real-time 2D mobile Fighting Game of vs gameplay. The game includes pool of over 300 unique characters. Each character is designed with unique set of character attributes such as attack speed, range and movement variations. Additionally, each char- acter has skills of different cooldown, status effects and duration. The large population of characters and real time competitive interactions provides a testbed for research on a heterogeneous population. Before the start of a match, both sides of players start by selecting their characters, auxiliary skill scroll and a pet summon. The key to winning a match is to reduce the opponent's health point (HP) to zero, or having higher HP at the end of 60 second match timer.
5

---

Under review as a conference paper at ICLR 2023
4.2 AGENT SPECIALIZATION
We record the advantage of the Generalists' and the Specialists' population against their prior it- eration of population policies. The recorded advantage vectors of agent k comprise the expected advantage of agent against its opponents over the duration of evaluation. The recorded advantage vectors of an agent are projected on the unit circles to visualize the vectors' magnitude and the vec- tors' direction components. The radial plots are measures in the population 's mean-deviation with the center of the circle representing zero advantage over the average population, and the radial edges represent the maximal deviation gain against specific radial opponent.
Generalizta
Spacialists
Apmi
Figure 1: Agent's Advantage And Specializations: From MI maximization optimization, both Generalists Agent I. and Agent K. show low deviations of advantage relative to the general popu- lation. This shows MI maximization optimizes the marginalized advantage of the population. In contrast, the Specialists Agent I. and Agent K. have demonstrated strong deviation from the pop- ulation norm. With MI minimization being the optimization objective, Specialists learn to gain an additional competitive edge by specializing agents' unique agent-attribtes.
From before to after agents' specialization, Figure I shows Agent I. has an increased magnitude of agents' advantage vectors. This implies that under M&M specialization. the MI between Agent I. and the general population is minimized. Similarly, Agent K. is also able to minimize its MI to achieve its own specialization. While both agents are able achieve specialization, the direction components are different for the two agents. Agent I. is specialized against prior iteration of itself and Agent K., and Agent K. is specialized against prior iteration of Agent I. and Agent O.. The unique directions of each agent's 's specialization shows that M&M specialization not only minimizes MI of the population, each agent is also optimized for its own aggnt-attributes.
Moreover, the Generalists and Specialists populations show different aggregated bounds. With the aggregate of Generalists as the dotted line region and the Specialists population as the solid line region, the radial plot shows that the Generalists dotted bound is restricted close to the center. With the center representing the average population, the closer the region to the center, the more generic the agent population behaves. In the extreme case of conformity behavior convergence, MI max- imization can reduce the region and eliminate the diversity in the population. On the other hand, Specialists show individual differences relative to the population norm. The individual specialization creates a collective diversity for the population.
4.3 SOCIAL GRAPHS AND INTERACTION STABILITY
With NeuPL (Liu et al. 2022 and (Shoham & Leyton-Brown 2008) graph solver, the different social graphs of the Generalists and Specialists are plotted on their corresponding radial charts. In a social graph, the labeled agent is the center node of the graph and is connected with edges E(x, y) outward to other agent nodes. E(, y) are normalized with population mean-deviation. The MI Maximized Social Graph is shown the left in blue, and M&M Social Graph on the right in red.

---

Under review as a conference paper at ICLR 2023
The social graphs display the relative frequency agents interact with each other and the deviation from the mean. The larger an edge value is, the more frequent the agent-nodes interact.
Maolmized Sedial Graph igentic U3.
MGM Sodel Graph
Agmali Dres 18u
Apent.
Agent
Agent Edp (KL
Figure 2: Social Graphs of Generalists and Specialists: In MI Maximization Social Graph, the uniformity of edge-connection strengths shows each agent approximately interacts with other agents a similar number of times. This result shows each agent receives approximately the same distribu- tion of experiences. In contrast, the social graphs of the Specialists gradually shift away from the symmetry. This allows each agent to interact with more relevant agents, and acquire more individ- ualized information distribution. The differences in social graphs directly influence the stability of population learning and agents' experiences distribution. The more disjoint the experience distribu- tions are, the sparser some interactions become.
MI Maximized Social Graph shown in Figure 2 illustrates that agents are connected with approx- imately equal connection strength in each edge. This indicates a densely connected social graph. This approximately translate to uniform distribution of {Px,y y) ))} with high stability in - HI(( (X, Y). contrast, M&M's Social Graph shows few strongly connected edges, such as Agent T. to Agent O., and sparse connections with other agents. With sparsity in the social graphs, each agent is to receive a different distribution experiences.
4.4 FROM MONOTONE: BEHAVIOR TO DIVERSE INTERACTIONS:
To analyze the agents' behavior, we introduced several metrics to track the timing of agents' skills. ForcingMoves are metrics that measure agent's 's attack initiation prior to the opponent's attacks. CounterMoves measure skills used to interrupt the opponent during their skill casting. CloneSubsti- tute measures the frequency that an agent avoids or escapes from the opponent's attack. The metrics displayed via mean-deviation frequency account for the different skill cooldown among the agents. We collect a batch of 100 games for each agent-type to look at how similar or different the learned strategies are.
Generatin Action Spuce Diveritty
Loall Music
saateeMont
Figure 3: Timing and Strategy: Comparing the behavioral difference between the Generalists and the Specialists, there is a clear difference in how Generalists and Specialists respond. For the Generalists, the mean-deviation shows similar change despite playing four uniquely different agents. Except for the cooldown constraint, the actions of the four Generalists are largely the same response. This shows the drawback of MI maximization optimization. On the contrary. the Specialists are incentivized to formulate creative plays based on their individual agent- attribast M&M creates a wide variety of strategy timings and competitive behaviors.
7

---

Under review as a conference paper at ICLR 2023
In Figure 3, the stacked Generalists and Specialists action timing reveals the similarity and differ- ences in different population's ss strategy timing. Specialists learn to time their skills differently when embodied in different agents. Each agent adapts its behavior such that its current strategy is optimal relative to the general population. In contrast, the Generalists population shows low variations in strategies. With the similar action timing, the Generalists have developed social behaviors that form with other agents. While under MI maximization, the optimized generic behavior works across the different agents, but the diverse Specialists behaviors better demonstrate the essence of general intelligence for adaptive learning and creativeness.
4.5 PERFORMANCE COMPARISON:
The e Nash of the Generalists agents iteratively optimized with rounds of joint policy opti- mization via NeuPL.'s graph solver (Heatmap shown in Appendix A.4). Here we display the top 8 performing Generalists relative to the M&M optimized Specialists. The displayed competitive per-
formance are the weighted win rates of the 8 agents versus the Nash of the Generalists population.
Eym
AydAA
A001
Eaperic 2017 Haar
Figure 4: Performance Re-Balance with M&M Specialization: In experiment 4.5 we look at how agents' performances be better balanced with M&M specialization. We first observe that with joint policy optimization, the top performing NeuPL. agents converge to win rates of 58.2 92.5% in deep blue with high performance inconsistency among the agents. The variation of performance shows that the joint policy may converge towards a set of behavioral strategies that benefits some agents more than the others. Particularly, MI maximized joint policy benefits the average population by adapting mutually similar joint policy, but for agents such as Agent A. and Agent K. their char- acter attributes may fall outside of the population norm. We see that with M&M, Agent A. and K. demonstrate a substantial performance gain and re-balanced the population performance gap (77.6 97.3%). Our empirical result shows that minimizing MI can enable agents to search for uncommon strategies that outperform the MI maximized counterpart.
Aside from the performance gain from agent specialization, Figure 4 also shows an interesting ob- servation that optimizing for population's 's commonality (MI maximization) does not translate to more balanced population. With joint policy optimization, the population converges with high per- formance differences among the agents. The standard deviation of MI maximized optimization is 9.95, while it is reduced to 5.99 after M&M's specialization. With approximately 30 % reduction in population performance deviation, it shows that equity of population (performance gap) can be improved when individual agents specialize.
4.6 SAMPLE EFFICIENCY:
In our last experiment we look at the sample efficiency the different optimizations. The joint policy optimization of the Generalists performed on set of 16 agents under MI maximization for 9 rounds. We then compare the relative performance gain to sample required with joint policy optimization and M&M for 2 additional rounds of optimization. The experiment measures the sam- ple complexity needed for the two methods to reach a similar level of relative performance prior to specialization.
8

---

Under review as a conference paper at ICLR 2023
Model Nance
Trinsin
Ganeraaa
Canarall::
Specialos
M4 Speciaie
*th Houzd
Cezarala:
720,00
33.0%
233%
64%
Gezeralt::
150,000
661%
1939
Agent Specill
Specialtt
Agent
850,000
950N
BRON
657%
Figure 5: Training Efficiency and Competitiveness: With 170k additional learning steps, the 11th round of Generalist can efficiently increase the average performance of the agents by 16.1 % (50% 66.1%) With M&M specialization, Agent K. and Agent M. can reach the same relative win rate against the general population as the 9th round of population with an additional 130k learning steps. The experiment indicates that when optimize for MI maximization, experience sharing allows joint policy optimization to efficiently optimize the expected behaviors of the whole population on objective,
commonality. Since MM optimization each experience sample is unique to each agent and would create separate learning distribution for each agent. To collect each separated learning experiences, additional computation is required.
longer optimize with MI maximization
In Figure 5, joint policy optimization demonstrates that it can use less learning steps per agent to optimize a population of agents. contrast, due to the diversified exploration M&M requires additional samples to explore the set of MI minimized agents interactions. The main takeaway from this experiment is that it would be more learning efficient to first learn the common skills under joint policy optimization. Sequentially, after Generalists' convergence, M&M specialization further benefit each agent with additional samples of training that diversify the learning experiences and enables specialization.
5 CONCLUSION
In our paper, we have derived two formulations of MI from multi-agent policy gradient. One brings new perspective on the existing work of joint policy optimization, the other explores the poten- tial of conditional MI minimization. The two learning representations under the dual formulations of MI measure the long-term effects of population's social behavior change. With the joint pol- icy optimization, the MI maximization of gradients significantly increased the sample efficiency of population learning by learning set of transferable skills across the agent policies. Through maxi- mizing MI as policy's gradient, the joint policy optimization optimizes the heterogeneous population toward set of mutually similar behaviors among the agents.
In contrast, our proposed M&M population learning aims to minimize the MI between the disjoint policies and the MI maximized joint policy agent population. Our conditional MI minimization population learning enables each agent to define an unique policy that best aligns with the agent's s own character attributes. Without the MI maximization constraint, the learned policy does need to be transferable to other agents. In a heterogeneous population the specialization of behavioral policies are especially beneficial agents that have dissimilar character attributes relative to the average population. As result, M&M can optimize diverse population of Specialists that better balance the performance inconsistency within a population. We believe that through studying the gradient formulation of population policy gradient help to bring better understanding the social behavior change that occurs within a population.
The current limitation of method is on the additional compute resource for learning agent's individualized specialization. Due to the process specialization being specific to each agent's character attributes relative to the joint policy, sample efficiency improvement on joint and disjoint policies optimization can both benefit the training needed for social behavior study. We thank the reviewers for suggestions and their valuable constructive feedback study.
Future Work:

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaogiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogencous systems, 2015. URL hitps: //wp://wwww.. www.erfissooo.ooooooomo org/ Software available from tensorflow.org.
Pierpaolo Battigalli and Marciano Siniscalchi. Strong belief and forward induction reasoning. Jour- nal of Economic Theory, 106()):356-39)) 2002.
Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Cel- ebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:3991-4002, 2021.
MuloCo Menagerie Contributors. MuJoCo Menagerie: A collection of high-quality simu- lation models for MuJoCo, 2022. URL http://github. com wttp:///////eece////c/ccccccccc joco
meragerie
Santiago Cuervo and Marco Alzate. Emergent cooperation through mutual information maximiza- tion. arXiv preprini wrXiv:2006.17700 2020.
Roel Dobbe, David Fridovich-Keil, and Claire Tomlin. Fully decentralized policies for multi-agent systems: An information theoretic approach. Advances in neural information processing systems, 30, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-leamer architectures. In International Conference on Machine Learning, pp. 1407-1416. PMLR, 2018
Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agen control using deep reinforcement learning. In International conference on autonomous agents and multiagent systems, pp. 66-83. Springer, 2017.
Johannes Heinrich, Marc Lanctot, and David Silver. Fictitious self-play in extensive-form games. In International conference on machine learning, pp. 805-813. PMLR, 2015.
M Jaderberg, WM Czarnecki, I Dunning, I. Marris, G Lever, AG Castaneda, et al. Human-level per- formance in first-person multiplayer games with population-basee deep reinforcement learning. arxiv. arXiv preprint arXiv:1807.01288 2018.
Woojun Kim, Whiyoung Jung, Myungsik Cho. and Youngchul Sung. maximum mutual informa- tion framework for multi-agent reinforcement learning. arXiv preprint arXiv:2006.027. 2020.
Jakub Grudzien Kuba, Ruiqing Chen. Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint ar 1099 /251, 2021.
Mare Lanctot, Vinieius Zambaldi. Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent rein- forcement learning. Advances neural information processing systems, 30, 2017.
Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. Neupl: Neural population learning. arXiv preprint 1rXiv:2202.07415, 2022.
10

---

Under review as a conference paper at ICLR 2023
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed coperative-compettivy environments. Advances in neural informa- fion processing systems, 30, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, 32, 2019.
mickep (https://math.ttackkk www.ssscccccccee.coooo skcceeeccangg.ommmsss////////nn Prove log(xy) log x log y. com wttp///omcc/////// 1/t/ (version: 2015-01-22).
Mathematics Stack Exchange. /Bl./httsss/maach.ssacckkeee http:://mmm..ctccccccccaa...ccc/
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
OpenAl, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw De- biak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. 2019. URL /atttps:/arxiv..ee org/ http:/abs/111112 06680
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in Ten- sorFlow. arXiv preprint arXiv:1802.05799 2018
Yoav Shoham and Kevin Leyton-Brow Brown. Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. AndroidEnv: A reinforcement learn- ing platform for android. abs/2105.13231, 2021. URL http::/aaaaiv.oor/bb//22222
13231
user65203. Prove log(xy) log x log y. Mathematics Stack Exchange. https://: stackexchange com/ wttpc//oo///////ecccc ////g///11111 wtg.ht/w//maaahasaakocccggee///p (ver- sion: 2015-01-22).
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Picter Abbecl, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint 27X1::2008.01825, 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforeement learning, Nature, 575(7782)3500-554 2019.
Ronald J Williams. Simple statistical gradient-followiin algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-256 1992.
Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi- agent reinforcement learning. In International conference on machine learning, pp. 5571-5580. PMLR, 2018
11

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MOBILE DEEP LEARNING ARCHITECTURE
But
Danert
Network
Modul
Meaking
Moduls
Citen
Figure 6: DRL architecture for mobile devices: We build our neural network deployable to mobile devices. We first pass inputs embedding layer reduce the dimension and also ConviD as opposed to the Dense layer to reduce compute load, Our input to output is separated into four modules.
1. Basic Module for game env stat & characters specific information.
2. Buff Module buff and debuff info during gameplay.
3. Element Module characters equipment of weapon IDs, armor, etc.
4. Action Masking Module gives the model the information on whether each action currently available.
The separation modules in Figure 6 allows us to have sparse connectivity in the model. The compute load is reduced compared to fully connected layers. This allowed us to inference our models locally on mobile devices.
MDP: The model receives from the Basic Module identify the characters and equipped Summons and Scrolls. Additionally, characters position, movement and skills' buff' element information are received from Buff and Element Module. The above the input states s to model. Our model predicts an action output a, to control the 2D movement of the agent, and the available attack and skills. The reward can be customized, but in standard mechanies is based on the (weighting) of an agent's 's own HP(10), opponent's 's HP(10), the result of the battle(10) combo(5), and mana(5). With the transition of the action, new state 8141 given the model for the next iteration of MDP.
A.1.1 HYPERPARAMETERS AND HARDWARE USED
Hyperparameters
Hardware Used
PPO: 0.1
CPUS: 5,300
GPUS: 0
n-step: 100 frames
max Batch size: 5120
Reward discount factor: 0.995
Average Compute Time:
Learning rate: le-4
180 Hrs (Generalist Training)
12

---

Under review as a conference paper at ICLR 2023
A.2 HETEROGENEOUS ACTION SPACE REPRESENTATION LEARNING:
With large collection of distinct agents, the learning problem face is their unique action sets and their relative similarity. Inspired by word embedding (Mikolov et al. 2013). we approach the problem with structure of information bottleneck to learn the representation of the action sets as a dense representation of a vector. We define our information bottleneck via the structure of a neural net [[() where S is the learn-able state space that holds the contextual information of the involved agents (x, y), action sets (a",a") and the joint observations. The goal of the model structure to compress the large contextual information learn a joint policy that captures the actions' similarity in specific interaction context.
A.3 INDEPENDENT CAUSALITY BETWEEN AGENTS' INTERACTIONS:
VM3-AC Kim et al. 2020 has pointed out the possible causality dependency of multi-agent inter- actions. When player decides its action oj it may depend on player j's action a². The obvious problem of reasoning is the circular causality of the two players, aj depends a and a depends on ar To resolve this issue, (Kim et al. 2020) has introduced a random variable to disassociate the loop dependency via variational lower bound.
We define our agents' causality graph without the Markov Assumption on agent's action. By remov- ing the Markov Assumption, can opponent's 's past actions 0u:-1 to predict opponent's 's current move a, This breaks the loop and assume the casual graph of players' actions are independent at time L. This gives the agents temporal consistency of Forward Induction (Battigalli & Siniscalchi. 2002] of Game Theory for self-play. The reached Nash Equilibrium under Forward Induction is self-consistent that the rational opponents in the past will continue to choose rationally in the future.
A.4 GENERALISTS SELF-PLAY ALGORITHM CONSTRUCTION:
Our neural population learning for joint policy optimization is performed under the multi-agent so- cial graph of Neuu et al. 2022). The nodes are the different generations of agents, and are connected by the weighted edges of (E(a))) NaN NeuPL provides a population self-play frame- work that not only competes the current population HT with combinations distinct agents, but the weighted edge prioritization also extend to different generations of e-Nash population IT - Each population is represented as a conditional joint policy that learns a set of BR strategies against all previous generations of multi-agents streeestrategies.
Tuaknng Banoratrr Toa
Reael1
Restt
Rearli
Raae?
Real 11
Rosw117
Ream
Reand
80.0%%
1.26
2304
184
248%
1.4%
Reanc
20.7%
37.45
4475
485%
5256
730%
624%
244
55%
215%
47%
115
40%
Reanc
9039%
F7485
66.9%
64085
14.95
10505
21.0%
2.5%%
3.8%
17
18.5%
15.6%
9077
17645
6428%
21.02
X082%
21.0%L
ML
717%
1.7%
11.0%
Rearc?
99.9%%
17.22%
53.7%%
58,8%
154Fh
75.0%
02.0%
25,24
21%
2890%
2488%
Reaut
90.0%
17.9%
56.9%
09.2%
73.9%
73 10%%
1200
N.50%
35 10L
89%
31.0%
31.05
3151
33.0%
Raadd
169%
17.4%
63.8%
6822%
18.25
75.4%
84.4%
41%
1/5%%
30.8%
1aand
90.0%
$7.5%
93.7%
98.0%
62/4h
7044%
79.0%
75.9%%
87.9%
98 1.199
50%
2 20105
3671%
17.9%
5460%
9:71h
80.2%
7.32%
4330%
Figure 7: The heatmap shows the evaluation matches across the different Generalist iterations TT The ablation evaluation shows joint policy optimization roughly diminished as the training ap- proaches the 11th 13th iteration. In particular, the 13th iteration only has a 57.6 % winrate against the 12th iteration of Generalist, which close to the Nash equilibrium of 50 %.
13

---

Under review as a conference paper at ICLR 2023
In Figure 7, evaluate all rounds of the Generalist policies against each other. The heatmap shows monotonic convergences of the Generalist population. At the 13th iteration, the Generalists population has converged as -- Nash Equilibrium where e = 7.6%. Further training may minimally increase the performance, but worsen the strategy diversity.
A.5 CORRELATION OF DIVERSITY AND PERFORMANCE
Since Agent T. is the only agent without performance gain after M&M specialization, we compare its Advantage vector change, performance chart with Agent H.'s.
Generalists
Specialists
Apent 10 MrnMte Sound
Agent Specialet:
Ageet
Agent
Agent
Agent
Agett
Specialer:
M/Madmaazter Bound
Advertage
Macimaation Sound
Advertage
Agent.
MI Mil Sound
Agent
M. MraMMax Sound
Agent
Agent
witrote
winate
Specka Generaist
084
07766
Tuinng Time
hr
15
Figure 8: The performance metric of Agent H. and T. Specialization: Agent H. shows initial performance of 72.7 % performance relative to the other 15 Generalists population. With M&M's specialization, Agent H. shows a substantial increase in competitive performance of 87.2 %. On the other hand, Agent T. shows an initial performance of 87 % performance relative to the other 15 Generalists population. With M&M's specialization, Agent T.' performance does not deviate from the initialization..
From Figure 8, shows the positive correlation strategy diversity and character performance still holds. With Agent H., the radial plot shows a large deviation when the strategy is being converted from Generalist to Specialist. This correlates with the notable increase in agent performance on the left chart of Figure 8. On the other hand, Agent T. shows more muted deviation within the Generalist bound. The lower strategy deviation is also reflected in Agent T'' flat performance on the right chart of Figure 8. The lack of specialization for Agent T. may be due to the Generalists' converged strategy is already close to the optimal strategy for Agent T. This may imply that Agent T. has the most centered agent attributes in relation to the population. As the Generalist policy
14

---

Under review as a conference paper at ICLR 2023
converges towards the population mean, the converged strategy become the optimal strategy for Agent T.
A.6 INTEGRATION:
Let (a* z, (a) y, and x, y > then the change variable approach (mickep https://math.staakeee sta.deecccage..om///ss//////// (user65203)
Let log(x) 1" δt
(10)
1=1
log(zy) 100
22 Øt
= 1-
ayy Øt
par
x Ōt
zry Òt
+
1=1 t
1=1 t
1=1
1=1 t
t= t
t
Ot
du
Let such that
u
Ot
1
Ou
(1I)
u
This gives us log(Pi (s) (a"ls)) = III (a (a"ls)) With the probability of in-
dependence assumption, Q(a", a², 8) function has the joint probability of Vglog(11! a())) in Eq(3).
A.7 ALGORITHM PSEUDOCODE
In this section we breakdown M&M population learning into pseudocode, where the main compo- nents the Specialists optimization are the neural population learning (NPL), c-NE of Generalists policy (Ip Specialists policies ((Twg J and graph solver F. Let NPL be the concurrent opti- mization of two-players game between two agents' policies. For every episode NPI. simulates the match (æ, a, IT), where π is matched against an agent member of TT and the population of col- lectively plays mixed strategy a. We collect the policies interaction trajectories, T, into replay buffer for policy gradient optimization.
Algorithm Neural Population Learning By RL NPL(T, o, 1))
replayBuffer e {
// Initialize trajectory replay buffer
for Episode E I..... do
Start game engine with policies (æ, II). Store trajectories from policies interactions (æ, a, II) into BeplayBu/ fer.
replayBufl +- replayBuf fer U T
end for
return replay Bu fer
For a given heterogeneous population (i, and an e-NE Generalists population HY, M&M optimizes [T.g ) to minimize MI of the individual agent policy against the Generalists population HN, Each one-vs-all matchmaking is sampled based the priority given by the graph solver F. After a batch of episodes of NPL, we optimize THs with PPO optimization. Additionally, we define Eval() compute the aggregate outcomes of NPi..
15

---

Under review as a conference paper at ICLR 2023
Algorithm 2 M&M Multi-Agent Specialization Pseudocode
Input:
Population (i.
Heterogeneous population of N distinct agents // N disjoint policies
( Ji=
Hg. (E* = ma (Tg.) 1
E e-NE Generalists
Agent k's social graph
F: RIXN + RIIN
Graph solver NeuPL (Chen et al. 2022)
Parameter: H6 f IN
Ik IIN, {E*)
k==
Output:
Algorithm Starts:
1: for E Population do
2: Tep <-
initialize N disjoint policies
3: end for
4: while (true do) do
5:
for k E Population do
6:
IT (II9 (a9
// Specialist k's social graph with the Generalists
7:
N JR, _1=))
// One-vs-all population learning with T& )// Optimize policy with PPO optimization Eval() computes the aggregate values of // vertex k's game outcomes.
8:
Te PPOO (gradienn.Steplrite
9:
t t (val(Twe: {II
10:
11:
ER <- F(UU))
Define k's social graph
12:
end for
IT TT U {oTWyy 1
Adding Specialists to opponent pool Iteratively specialize Specialists {Teg JN
Jk=i
15:
end while
return {TT J 11 {E
With Section 3.3's defined social graph, U denote the probabilistic outcome distribution of all pairwise game matches with Specialist k. We use F(U*) to update k's social graph weighted edges prioritize sampling of adversarial opponents. After all Specialists have converged, add the population into the opponent pool set. M&M iteratively optimizes the set of Specialized population until population performance convergence.
A.8 SOFTWARE AND LICENSING
The models implemented via Tensorflow, TensorflowLite (Abadi et al. 2015). IM- PALA(Espeholt al., 2018), and (Segeedee & Balso 2018). These softwares are all licensed under Apache License 2.0.
16


Reviews:
[{"review_id": "dSAwhjBF3ra", "reviewer": "Reviewer_Eoc4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies agent specialization for multi-agent reinforcement learning. The authors proposed a min-max formulation of mutual information. The authors claim that the proposed method enables agents specialization with stable regularization. The proposed approach is evaluated on the newly introduced ‘Naruto Mobile’ which  is a real-time 2D mobile game with two competitive players.", "summary_of_the_review": "In summary, the reviewer found the paper not ready for publication due to the writing quality. \n", "strength_and_weaknesses": "### Strength:\nThe paper studied social behavior change and agent specialization, which are important problems for multi-agent reinforcement learning. \n\n### Weakness:\n\nThe writing of the paper is casual and lacking the rigor needed by a research paper. Many terms and symbols are undefined. Many sentences and paragraphs are unclear. Please see the following for more details:\n1. In Eq 1, symbols such as $\\theta, a^x, a^y, s, \\pi$ and the term ADV are all undefined. Without defining the symbol, the authors left the readers guessing what each symbol meant.   \n2. In P3, Q, V are undefined.   \n3. In P.3, what are Generalists and Specialists? Please provide a formal definition.  \n4. In P.4, what is a graph solver F? Please provide definition and context.   \n5. In P.5, what is a ‘recorded vector’? Please define and explain.   \n6. The term social graph is used throughout the paper without definition.     \n\nThe above list is not exhaustive. There are still a lot of undefined terms.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "_OUwJnkpvX", "reviewer": "Reviewer_khP1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As implied by the author, multi-agent systems can be studied in the sense of social behavior. Where each agent needs to be an expert in a particular task, they should share a common understanding of the whole space with other agents. To this end, a wide variety of methods are proposed to fulfill both of these criteria. One can think of that common understanding/knowledge as the ability to generalize from bottom to top. That means when an agent is an expert in one task, it can easily be extended to contribute to the solution of a larger task. Since an intelligent agent can be showcased by using a single policy function, many of the existing methods in this matter focus on relations between different agents(a.k.a policies). This paper generally centers around proposing an optimization framework in multi-agent systems by using Reinforcement Learning notions and classic game theory. Specifically, by using the Mutual Information concept, which conveys the level of shared understanding of the same random variable, a gap will rise between generalization and expertise. Having grounds of knowledge about almost everything will, obviously, be counterproductive in the case of getting narrow in a specific task. Therefore, the author tries to employ classic game theory to balance this dilemma.", "summary_of_the_review": "* As stated before, the contribution is not perfectly annotated in the paper. The author pretty much employs the related techniques in the proposed algorithm and does not clearly claim the novelty.\n* The proposed method closes an essential gap in multi-agent systems, but still the generalist seems to take so much time to converge\n* The overall quality of paper sounds acceptable, but slight improvements on the presentation, specifically, on the contribution will make it more understandable", "strength_and_weaknesses": "The most important merit of this paper is the contribution toward an equilibrium between generalization and expertise in multi-agent systems without having any assumptions about cooperation or competitive setup. Once the initialization happens, the proposed algorithm will basically follow the min-max strategy to reach an equilibrium satisfying both generalization and expertise. \n\n**Strengths**\n* No assumption on the cooperation/competition.\n* Practical on policy optimization that provides better performance compared to value-based methods.\n* Intuitive representation of results using radial charts.\n\n**Weaknesses**\n* The contributions are not highlighted enough in the paper.\n* Although it cannot be a direct weakness, we should try to avoid using KL in policy optimization due to complexity.\n* Background and Results are limited to two player games.\n* The results are not compared to any state-of-the-art method.\n* The algorithm is not discussed in details.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YgXQoCh65zh", "reviewer": "Reviewer_fF3t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deal with the problem of mutii-agent specialization: which is the issue related to the developing of a single general learning algorithms for a relatively diverse population. Typically each agent in a multi-agent paradigm is trained using separate behavioral policies. But this has now led way to jointly optimized learning solutions, \n\nThe authors here report that the joint policy is related to a simple information theoretic measure: mutual information (MI) among agents. This solution however leads to a bottleneck, which is that individual agents do not specialize. To circumvent this issue, the authors derive a minimax MI solution (M&M) and show that the agent trained using this method not are better specialisists but do better on competitive play metrics. \n", "summary_of_the_review": "The paper makes an interesting observation regarding the training of multi agent models. The proposed solution however seems rather extreme and the context under which this would be relevant in the training of other multi-agent models remains to be seen. The current results on a single benchmark are well taken. ", "strength_and_weaknesses": "Pros:\n1. This study is a valiant attempt to demystify multi-agent learning in current RL models\n2. The proposed solution is intuitively easy to understand.\n\nCons:\n1. The paper is really hard to follow and there are several typos.\n2. It is hard to know if the experiments truly generalize. Besides the theoretical argument, the experiments were performed on a single game: Naruto Mobile. While the authors report considerable improvement there is considerable work to be done. \n3. There is an obvious link between stability (specilization) and plasticity (generalization to new tests). The authors have not discussed this theme. It is not always desirable for agents in a multi-agent game to specialize and I wonder how the authors think about this in context of their work. I worry that M&M may NOT in fact the most desirable in most case of AI based training methods. \n4. The conclusion and discussion sections were notably sparse on details. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8bzpAu-zKn-", "reviewer": "Reviewer_NL8L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper offers a new perspective to solve the behavior homogeneity issue in current joint policy optimization approaches. The paper points out the connections between policy gradient in multi-agent joint policy optimization and mutual information (MI) maximization followed by the drawbacks of MI maximization. The paper proposes a minimax formulation of MI, M&M, to improve agents’ diversity and specialization while maintaining the population stability.", "summary_of_the_review": "This paper purposes a minimax MI approach to mitigate the behavior homogeneity issue among agents in joint policy optimization. The idea proposed in the paper could be inspiring to the community, but I would expect a better presentation of the proposed method and a more comprehensive evaluation and discussion.", "strength_and_weaknesses": "Strengths: The key idea is straightforward and easy to understand. The paper is backed by a succinct theoretical analysis when it talks about the underlying relationship between joint policy gradient and MI maximization. \n\nWeaknesses: Even though the method is well-motivated, the proposed method is not easy to follow, with terms and symbols left undefined. Moreover, the experiment section needs more work. I would like to see a more thorough evaluation and clearer explanation of the M&M results compared to NeuPL, such as the root cause of Agent A’s significant improvement. In addition, it would be good to show how M&M works in games with more cooperation among agents like MuJoCo Football mentioned in the NeuPL paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Q0PDdXrU7", "reviewer": "Reviewer_ryJg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper connects policy gradient method in joint policy optimization of multi-agent RL to MI maximization framework and shows that joint policy optimization essentially leads to MI maximization. Thus joint policy optimization produces skills that are transferable across agents but it costs in diversity and performance of agents. To address this issue, the authors propose a novel minimax algorithm called M&M that is able to learn agents that specialize at different skill via conditioning on the stable generalist policies obtained through MI regularization. Experimental results demonstrate the similarity between the agents learned through joint policy optimization and how specialist agents learned by their algorithm shift away from Generalist’s behavior to specialize in their own specific behaviors. They also presented a qualitative comparison of their algorithm with prior works like NeuPL.", "summary_of_the_review": "The paper is well motivated and the proposed M&M algorithm appears quite novel. However, technical section 2.3 and section 3 lacks clarity and should be improved further.", "strength_and_weaknesses": "- The paper is well motivated and seeks to learn agents with diversified behavior pattern in a multi-agent policy optimization framework.\n    \n- The experimental section demonstrated the advantage of their algorithm over prior works.\n    \n- However, the technical sections 2.3 and 3 lacks clarity and has some issues(see comments in next section for more details).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2I0TclgSvh", "reviewer": "Reviewer_3HQh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a multi-agent reinforcement learning setup and takes an information-theoretic approach to the joint policy gradient method. The first observation is that the joint policy gradient maximizes the mutual information (MI) between the agents -- an observation that follows by definition of MI. The authors extract an important insight from the observation, namely, that the joint policy gradient is in fact maximizing the stability of social behavior, and may compromise the individual performances of the agents. In order to address this challenge, they propose Minimax MI algorithm, which specializes for individual goals by maximizing individual performance conditioned on the socially stable behaviors. The authors provide experimental evaluations of the proposed method, showcasing its ability to specialize. ", "summary_of_the_review": "In the current state, I would not recommend the paper for publication, due to the concerns I have raised in the Strengths and Weaknesses section. I believe that the observations made by the authors and their proposed algorithm are interesting and the topic is relevant to the community, but the paper needs a lot of improvement (see Strengths and Weaknesses section). With that being said, I am willing to reconsider my score, conditioned on the authors addressing the raised concerns.", "strength_and_weaknesses": "### Strengths\n\n1. The paper considers an interesting problem which is relevant to the MARL community. \n2. The key observation that the authors make provides an interesting insight into the intuition behind joint policy gradient, albeit it being a straightforward derivation.\n3. The proposed algorithm seems to address the specialization issue of joint policy gradient. \n4. Experimental evaluation of the method is present, and seems to prove the effectiveness of the algorithm with respect to vanilla joint policy gradient. \n\n### Weaknesses\n\nThe most major weakness of the paper is that it is poorly written:\n 1.  There are typos everywhere, from abstract to conclusion.\n\n 2.  Definitions of mathematical objects are practically non-existent:\n     - x and y, or $\\Pi$, in the beginning of Section 2.3; \n     - $\\psi^k_i$ and $\\epsilon$-Nash in Section 3.2. \n     - $S: O \\times O$ is said to be a function, but I do not understand what the range of the function is.\n\nI understand that most things are left implicit, but this practice makes the paper prone to errors, since the derivations are not based on well-defined objects, and makes the paper very hard to read. \n\n 3. The paper could benefit from additional remarks or further elaborations:\n    - There should be an explicit formulation of the graph solver from (Chen et al. 2022) and its relevance to the problem considered, not just a reference. \n    - What is the meaning of $Self-play$ function in the pseudocode?\n    - What is the meaning of $D_{kl}(gradient_{step}(\\pi_{\\psi^k_\\tau}))$?\n    - What is the meaning of $Eval(\\pi_{\\psi^k_\\tau}, \\{ \\Pi_g\\}^N_{g=I})$, and why is the first index $I$ in these definitions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1TfYGOpph8", "reviewer": "Reviewer_BFMo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a minimax formulation of MI (M&M) that enables agents specialization with stable regularization. In this paper, authors have derived two formulations of MI from multi-agent policy gradients. One brings a new perspective on the existing work of joint policy optimization, the other explores the potential of conditional MI minimization. Then authors empirically evaluated M&M against the prior SOTA MARL framework, and analyzed the social behavior change in performance, diversity, and the stability of their social graphs.\n", "summary_of_the_review": "The paper provided a new perspective on joint policy optimization to enable agent specialization. The considered problem is interesting. But the reviewer found the present methodology is hard to follow and the experiments can be enriched by testing on more scenarios.\n", "strength_and_weaknesses": "Strength:\n\nThe paper provided a new perspective on the existing work of joint policy optimization, The experimental environment selected a novel Naruto game. Naruto Mobile is a real-time 2D mobile Fighting Game of 1 vs 1 gameplay. The game has a pool of over 300 unique characters, and each has different character attributes in attack speed, range and movement variations. Additionally, each character has skills of different cooldown, status effects and duration. The large population of characters and real time competitive interactions create a testbed environment for research on heterogeneous specializations, embedded agents and Game Theory of Nash Equilibria.\n\nWeaknesses:\n\n* The writing of this paper is confusing, and there are many basic mistakes. For example, PSRO is Policy Space Response Oracle, not Policy Search Response Oracle. \n\n* The objective function in Eq.(1) is confusing. As this is a competitive game, how is the advantage function $Adv(s,a^x,a^y)$ computed?\n\n* In eq(2), how could the policy gradient of two competitive players be computed together?\n\n* It’s not clear to me how mutual information helps in multi-agent specialization.\n\n* The definition of $\\epsilon$-nash is not given.\n\n* This work would benefit from testing on more tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0VxWG-AEJi8", "reviewer": "Reviewer_LEQx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Given Joint policy optimization encourages knowledge sharing and social behaviors, this paper connects the dots between policy gradient formulation of joint policy optimization and MI maximization. Authors have proposed a minimax formulation of MI (M&M) that enables agents specialization with stable regularization. It’s pointed as a good sweet spot to balance the tradeoffs with MI maximization/minimization alone - M&M optimizes agent specialization while regularizing the learning instability with conditional MI minimization. Authors have included a theoretical and experimental analysis on the social behavioral change of agent population and demonstrate that M&M benefits in population social diversity and specialists competitive play.", "summary_of_the_review": "The motivation of this paper is technically sound and empirical evaluations shown significant performance gains. This paper features minimax formulation of MI (M&M) that enables agents specialization with stable regularization. experimental results have shown significant benefits in population social diversity and specialists competitive play. However, some sub-sections of the experimental section aren't clear/seek more details. It would have been further helpful if authors included qualitative analysis. For more details, please refer to the strengths and weaknesses part of the review.", "strength_and_weaknesses": "Authors have well drafted the related work/background section, clearly explaining past work in competitive & cooperative behavioral learning. Solid motivation pointing out the trade offs with MI maximization/minimization alone. Gains in empirical evaluation are significant - performance gain from agent specification is significant(few agents beating sota by >20 points interms of winrate%). Also, 30 % reduction in population performance deviation, as authors pointed out, shows that equity of a population can be improved when individual agents specialize.\n\nSome sub-sections under experiment aren’t clear/seek more details. In section 4.5, Winrate % is not consistent across all agents, specifically seems to be more beneficial for weaker agents. For example, agent T, agent D see no/little performance gain with M&M specialization. Also, training efficiency and competitiveness is not clear and require more details - for ex, why are benchmarks limited/specific to 2 agents (K and M)? Lastly, what would the impact of number of unique agents on joint policy optimization benefits (in these experiments, 16 unique agents of Naruto Mobile characters were chosen).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject