The Emergence of Individuality in Multi-Agent Reinforcement Learning
Authors: Jiechuan Jiang
Zongqing Lu
Venue:  (2021)

Abstract: Individuality is essential in human society, which induces the division of labor and thus improves the efficiency and productivity. Similarly, it should also be a key to multi-agent cooperation. Inspired by that individuality is of being an individual separate from others, we propose a simple yet efficient method for the emergence of individuality (EOI) in multi-agent reinforcement learning (MARL). EOI learns a probabilistic classifier that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classifier. The intrinsic reward encourages the agents to visit their own familiar observations, and learning the classifier by such observations makes the intrinsic reward signals stronger and in turn makes the agents more identifiable. To further enhance the intrinsic reward and promote the emergence of individuality, two regularizers are proposed to increase the discriminability of the classifier. We implement EOI on top of popular MARL algorithms. Empirically, we show that EOI outperforms existing methods in a variety of multi-agent cooperative scenarios.

Under review as a conference paper at ICLR 2021
THE EMERGENCE OF INDIVIDUALITY IN MULTI- AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Individuality is essential in human society. It induces the division of labor and thus improves the efficiency and productivity. Similarly, it should also be key multi-agent cooperation. Inspired by that individuality is of being an individual separate from others, we propose a simple yet efficient method for the emergence of individuality (EOI) in multi-agent reinforcement learning (MARL). EOI learns a probabilistic classifier that predicts a probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classifier. The intrinsic reward encourages the agents to visit their own familiar observations, and learning the classifier by such observations makes the intrinsic reward signals stronger and in turn makes the agents more identifiable. To further enhance the intrinsic reward and promote the emergence of individuality, two regularizers are proposed to increase the discriminability of the classifier. We implement EOI on top of popular MARL algorithms. Empirically, we show that EOI outperforms existing methods in a variety of multi-agent cooperative scenarios.
INTRODUCTION
Humans develop into distinct individuals due to both genes and environments (Freund et al., 2013). Individuality induces the division of labor (Gordon, 1996), which improves the productivity and efficiency of human society. Analogically, the emergence of individuality should also be essential for multi-agent cooperation.
Although multi-agent reinforcement learning (MARL) has been applied to multi-agent cooperation, widely observed that agents usually learn similar behaviors, especially when the agents are homogeneous with shared global reward and co-trained (McKee et al., 2020). For example, in multi-camera multi-object tracking (Liu et al., 2017), where camera agents learn cooperatively track multiple objects, the camera agents all tend track the easy object. However, such similar behaviors can easily make the learned policies fall into local optimum. the agents can respectively track different objects, they are more likely solve the task optimally. Many studies formulate such a problem as task allocation or role assignment (Sander et al., 2002; Dastani et al., 2003; Sims al., 2008). However, they require that the agent roles are rule- and the tasks pre-defined, and thus are not general methods. Some studies intentionally pursue difference agent policies by diversity (Lee et al.. 2020; Yang et al., 2020) or by emergent roles (Wang et al., 2020a), however, the induced difference is not appropriately linked to the success of task. On the contrary, the emergence of individuality along with learning cooperation can automatically drive agents behave differently and take a variety of roles, if needed, to successfully complete tasks.
Biologically, the emergence of individuality is attributed innate characteristics and experiences. However, as in practice RL agents are mostly homogeneous, we mainly focus on enabling agents to develop individuality through interactions with the environment during policy learning. Intuitively, in multi-agent environments where agents respectively explore and interact with the environment, individuality should emerge from what they experience. In this paper, we propose a novel method for the emergence of individuality (EOI) in MARL. EOI learns probabilistic classifier that predicts probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted probability by the classifier. Encouraged by the intrinsic reward, agents tend to visit their own familiar observations. Learning the probabilistic classifier by such observations makes the intrinsic reward signals stronger and in turn makes the agents more identifiable. In this

---

Under review as a conference paper at ICLR 2021
closed loop with positive feedback, agent individuality emerges gradually. However, at early learning stage, the observations visited by different agents cannot be easily distinguished by the classifier, meaning the intrinsic reward signals are not strong enough to induce agent characteristics. Therefore, we propose two regularizers for learning the classifier to increase the discriminability, enhance the feedback, and thus promote the emergence of individuality.
FOI is compatible with centralized training and decentralized execution (CTDE). We realize EOI on top of two popular MARL methods, MAAC (Iqbal & Sha, 2019) and QMIX (Rashid al., 2018). For MAAC, as each agent has its own critic, it is convenient to shape the reward for each agent. For QMIX, we introduce an auxiliary gradient and update the individual value function by both minimizing the TD eπrоr of the joint action-vait value function and maximizing its cumulative intrinsic rewards. We evaluate EOI in three scenarios where agents are preferred to take different roles, ie., Pac-Men, Windy Maze, and Firefighters, and we empirically demonstrate that EOI significantly outperforms existing methods. Additionally, in a micro-task of StarCraft II (Samvelyan al., 2019) where the need for the division of labor is unknown, EOI also learns faster than existing methods. By ablation studies, we confirm that the proposed regularizers indeed improve the emergence of individuality even if agents have the same innate characteristics.
2 RELATED WORK
MARL, We consider the formulation of Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where each timestep l each agent receives a local observation of. takes an action at. and gets a shared global reward r-. Agents together aim to maximize the expected return E E r'r', where 7 is a discount factor and T' is the time horizon. Many methods have been proposed for Dec-POMDP, most of which adopt CTDE. Some methods (Lowe et al., 2017; Foerster et al., 2018; Iqbal & Sha, 2019) extend policy gradient into multi-agent cases. Value function factorization methods (Sunehag et al., 2018; Rashid et al., 2018; Son et al., 2019) decompose the joint value function into individual value functions. Communication methods (Das et al., 2019; Jiang et al., 2020) share information between agents for better cooperation.
Behavior Diversification. Many cooperative multi-agent applications require agents to take different behaviors to complete the task successfully. Behavior diversification can be handcrafted or emerge through agents' learning. Handcrafted diversification is widely studied as task allocation or role assignment. Heuristics (Sander et al., 2002; Dastani et al., 2003; Sims et al., 2008; Macarthur et al., 2011) assign specific tasks or pre-defined roles to each agent based on goal, capability, visibility, or by search. M (Shu & Tian, 2019) learns manager to assign suitable sub-tasks to uule-based workers with different preferences and skills. These methods require that the sub-tasks and roles are pre-defined, and the worker agents are rule-based. However, in general, the task cannot be easily decomposed even with domain knowledge and workers are learning agents.
The emergent diversification for single agent has been studied in DIAYN (Eysenbach et al., 2019), which learns reusable diverse skills in complex and transferable tasks without any reward signal by maximizing the mutual information between states and skill embeddings as well as entropy. In multi-agent learning. SVO (McKee et al., 2020) introduces diversity into heterogeneous agents for more generalized and performiff policies in social dilemmas. Some methods are proposed for hehavior diversification in multi-agent cooperation. ROMA (Wang et al., 2020a) learns a role encoder to generate the role embedding, and learns role decoder to generate the neural network parameters. However, there is no mechanism that guarantees the role decoder can generate different parameters, taking as input different role embeddings. Learning lowelevel skills for each agent using DIAYN is considered in Lee et al. (2020); Yang et al. (2020), where agents' diverse low-level skills are coordinated by the high-level policy. However, the independently trained skills limit the cooperation, and diversity is not considered in the high-level policy.
3 METHOD
Individuality is of being an individual separate from others. Motivated by this, we propose EOI, where agents are intrinsically rewarded in terms of being correctly predicted by a probabilistic classifier that is learned based on agents' observations. If the classifier learns to accurately distinguish agents, agents should behave differently and thus individuality emerges. Two regularizers introduced for
2

---

Under review as a conference paper at ICLR 2021
learning the classifier to enhance intrinsic reward signal and promote individuality. Unlike existing work on behavior diversification, EOI directly correlates individuality with the task by intrinsic reward, and thus individuality emerges naturally during agents' learning. EOI can be applied to Dec-POMDP tasks and trained along with CTDE algorithms. We design practical techniques to implement HOI top of two popular MARL methods, MAAC and QMIX. We also propose an unbiased gradient estimation for learning from other agents' experiences to accelerate learning, which appears in Appendix A.I.
3.1 INTRINSIC REWARD
As illustrated in Figure 1, two agents learn to collect two items, where item b is the casier one. Collecting each item, they will get a global reward of +1. Sequentially collecting the two items by one agent or together is sub-optimal, sometimes impossible when the task has limited horizon. The optimal solution is that the two agents go collecting different items simultaneously. It is easy for both agents to learn to collect the easier item b. However, after that, item becomes even harder to be explored, and the learned policies easily fall at local optimum. Nevertheless, the emergence of individuality can address the problem, e.g., agents prefer to collect different items.
To enable agents to develop individuality, EOI learns a prob- abilistic classifier P([[O) to predict a probability distribution over agents given on their observation, and each agent takes the correctly predicted probability as the intrinsic reward at each timestep. Thus, the reward function for agent i is modified as
by TOEEE
r + ap(ilo),
(1)
PaTTOs
where r is the global environmental reward, p(ilo:) is the pre- dicted probability of agent i given its observation og, and a is a tuning parameter to weight the intrinsic reward. With the reward shaping, EOI works as follows. If there is initial differ- ence between agent policies in terms of visited observations, the difference captured by P(Z]O) it is fitted using agents' experiences. The difference is then fed back to each agent as an intrinsic reward. As agents maximize the expected return, the difference in agents' policies is exacerbated together with optimizing the environmental return. Therefore, the learning process is closed loop with positive feedback. As agents progressively behave more identifiably, the classifier can distinguish agents more accurately, and thus individuality emerges gradually.
Bdag wit PORME
Figure 1: HOI
The classifier Po P& (IOO) is parameterized by neural network é and learned in a supervised way. At each timestep, we take each agent i's observation o, input and the agent index i as the label and store the pair < o,, i > into a buffer B. Φ is updated by minimizing the cross-entropy loss (CE), which is computed based on the uniformly sampled batches from B.
3.2 REGULARIZERS OF (T(O)
In the previous section, we assume that there is some difference between agents' policies. However, in general, the difference between initial policies is small (even differences if agents' policies are initially by the same network weights), and the policies will quickly learn similar behaviors as in the example in Figure 1. Therefore, the intrinsic rewards are nearly the same for each agent, which means no feedback in the closed loop. To generate the feedback in the closed loop, the observation needs to be identifiable and thus the agent can be distinguished in terms of observations by (1)).. To address this, we propose two regularizers: positive distance (PD) and mutual information (MI) for learning (I)OO
The positive distance is inspired from the triplet loss (Schroff et al., 2015) in face recognition, which is proposed to learn identifiable embeddings. Since of and its previous observations (o of --At+1
Positive Distance.
of 1) are distributed on the trajectory generated by agent i, the previous observations in the åt-length window could be seen as the positives of of. To make the probability distribution the anchor of close that on the positives, we sample an observation of
3

---

Under review as a conference paper at ICLR 2021
from fot-Ar, o- .... o " and minimize the cross-entropy loss
CE (po(-lot).p(-lo)))
The positive distance minimizes the intra-distance between the observations with the same "identity", which hence enlarges the margin between different "identities". As a result, the observations become more identifiable. Since the positives are naturally defined on the trajectory, the identifiability generated by the positive distance is actually induced by the agent policy. Defining the negatives is hard but we find that just using the positive distance works well in practice.
Mutual Information. If the observations are more identifiable, it is easier to infer the agent that visits the given observation most, which indicates the higher mutual information between the agent index and observation. Therefore, to further increase the discriminability of the classifier, maximize their mutual information,
MI(I:O) = H((I)= - H(TTO)
(3)
-[(TT= - Eoop(o)
Σ p( log p(ilo)
Since store 0,, of every agent in B, the number of samples for each agent is equal. Fitting (o(1)) using batches from B ensures H(T) is a constant. To maximize MT(7; O) to minimize O() Therefore, equivalently, we sample batches from B and minimize
CE
(4)
Therefore, the optimization objective of Po(1(O) is to minimize
CE (po(-lo)), one hot(i)) + S,CE
CE (po(-10)). p(-lo)) + &,CE (po(+lo))
(5)
where B1 and B2 are hyperparameters. The regularizers increase the discriminability of Po (())), make the intrinsic reward signals stronger to stimulate the agents to be more distinguishable, and eventually promote the emergence of individuality. In this sense, (2(IO) not only is the posterior statistics, but also serves as the inductive bias of agents' learning. The learning process of EOI is illustrated in Figure 1.
3.3 IMPLEMENTATION WITH MAAC AND QMIX
Existing methods for reward shaping in MARL focus independent learning agents e.g., McKee et al. (2020); Wang et al. (2020b); Du et al. (2019). How to shape the reward in Dec-POMDP for centralized training has not been deeply studied. Since the intrinsic reward can be exactly assigned to the specific agent, individually maximizing the intrinsic reward is more efficient than jointly maximizing the sum of all agents' intrinsic rewards (Hughes et al., 2018). Adopting this idea, respectively present the implementation with MAAC and QMIX for realizing EOL. MAAC is an off-policy actor-critic algorithm, where each agent learns its own critic, thus it is convenient to directly give the shaped reward + app (ilo.) ) to the critic of each agent i, without modifying other components, as illustrated Figure 2(a). The TD error of the critic and the policy gradient are the same as in MAAC (Iqbal & Sha, 2019).
TD ..ok
apol((e)
Crite
ATT
messages
Pa
Mising Natwork
IVF
di,
V. (Ag -
Actor /
Agent
Agent
(a) EOI+MAAC
(b) EOI+QMIX
Figure 2: Illustration of EOI with MAAC and QMIX.
4

---

Under review as a conference paper at ICLR 2021
Treacun
Figure 3: Illustration scenarios: Pac-men (leff), Windy Maze (mid), Firefighters (right).
In QMIX, each agent has individual action-value function Qf. All the individual action-value functions are monotonically mixed into joint action-value Qio by a mixing network. Each agent selects the action with the highest individual value, but the individual value has neither actual meaning nor constraints (Rashid et al., 2018). Therefore, we can safely introduce an auxiliary gradient of the intrinsic reward the individual action-value function Q Each agent learns an intrinsic value function (IVF) O which takes as input the observation and the individual action-value vector Qi(o.) (o.) and approximates EE Er 2'p(i of) by minimizing the TD error,
(6) Of and O are the target value functions and D is the replay buffer. In order to improve both the global reward and intrinsic reward, we update Q parameterized by 0,, towards maximizing I22 (o,, Q7 (o.: 0s)) ] along with minimizing the TD error of Qoo (denoted as 5e as illustrated in Figure 2(b). Since the intrinsic value function is differentiable with respect to the individual action-valle vector O (os; 0z), we can establish the connection between O and Of by the chain rule, and the gradient of 0g is.
E Ec (o)).))) [[Q (o., Q (o.)) y)²] where y po(ilo.) + (o Q7
Ootot aQ. (i (Q7 0.)
Ve ((.) = 00,
(o;;0;; 0.)
00,
MAAC and QMIX are off-policy algorithms, and the environmental rewards are stored in the replay buffer. However, the intrinsic rewards are recomputed in the sampled batches before each update, since (2)) is co-evolving with the learning agents, the previous intrinsic reward is outdated. The joint learning process the classifier and agent policies can be mathematically formulated as a bi-level optimization, which is detailed in Appendix A.2.
4 EXPERIMENTS
To clearly interpret the mechanism of EOI, we design three scenarios: Pac-Men, Windy Maze, and Firefighters, which are illustrated in Figure 3, and adopt a micro-task of StarCraft (Samvelyan et al., 2019). In the evaluation, we first verify the effectiveness of EOI and the two regularizers on both MAAC and QMIX by ablation studies. Then, we compare EOI against EDTI (Wang et al., 2020b) to verify the advantages of individuality with sparse reward, and against ROMA (Wang et al., 2020a) and Lee et al. (2020); Yang et al. (2020) (denoted as HC and HC-simul respectively) to investigate the advantages of individuality over emergent roles and diversity. The details about the experimental settings and the hyperparameters are available in Appendix A.3.
4.1 PAC-MEN
There are four pac-men (agents) initialized at the maze center, and some randomly initialized dots. illustrated in Figure 3 (left). Each agent has a local observation that contains a square view with 5 x 5
Muxt oxternccaal
Maxx coverntt
COI MAAC
ECt
1OL MANE only FD
FOL SANEE e MI
FOi MAVE win
FDr GMIS Pask 0M00
MAAC
Figure 4: Learning curves in Pac-Men: EOI+MACC (left) and EOL+QMIX (right).
5

---

Under review as a conference paper at ICLR 2021
Figure 5: Distributions of agents' positions of QMIX (left) and EOI+QMIX (mid), and kernel PCA of agents' observations of EOI+QMIX (right) in Pac-Men. The darker color means the higher value.
grids centered at the agent itself. At each timestep, each agent can move to one four neighboring grids or eat dot. The agents only get global reward, i.e., the total eaten dots, at the final timestep.
As depicted Figure 4, MAAC and QMIX get the lowest mean environmental reward respectively, since some agents learn go to the same room and compete for the dots. This can be verified by the position distribution of QMIX agents in Figure 5 (left) where three agents move to the same room. MAAC agents behave similarly. At the early stage of training, it is easy for the agents to explore the bottom room and eat dots there to improve the reward. Once the agents learn such policies, it is hard explore other rooms, the agents learn similar behaviors and fall the local optimum.
Driven by the intrinsic reward without both regularizers, EOI obtains better performance than MAAC and QMIX. But the improvement is not significant since the observations of different agents cannot be easily distinguished when there is little difference between initial policies. The regularizers of PD and MI can inerease the discriminability of (T(O), providing stronger intrinsic signals. Guided by Po(TIO) with PD or MI, the agents go to different rooms and eat more dots. MI theoretically increases the discriminability even the initial policies have no differences, while PD makes the observations distinguishable according to policies. Combining the advantages of the two regularizers leads to higher and steadier performance, as shown in Figure 4. With both two regularizers, the agents respectively go the four rooms and achieve the highest reward, which is indicated in Figure 5 (mid). We also visualize the observations of different agents by kernel PCA, as illustrated in Figure 5 (right), where darker color means higher correctly predicted probability of (T(T(O). We can (T can easily distinguish agents given their observations.
To further investigate the effect of the regularizers, show the learning curves of intrinsic reward and environmental reward of EOI with or without regularizers in Figure 6. EOI with the regularizers converges to higher intrinsic reward than that without regularizers, meaning agents behave more distinctly. With regularizers. the rapid increase of intrinsic reward occurs before that of environmental reward, which indicates the regularizers make (I)O) also serve as the inductive bias for the emergence of individuality.
We also investigate the influence of the difference between initial policies. The action distributions (over visited states) of the four initial valuee functions in QMIX are illustrated in Figure 7, where we find there is a large difference between them. The inherent observation makes agents distinguishable initially, which is the main reason EOI without the regularizers works well. We then initiate the four action-value functions with the same network weights and re-run the experiments. The performance of EOI without the regularizers drops considerably, while EOI with the regularizers has almost no difference, as illustrated in Figure 8. Therefore, we can conclude that PD and MI also make the learning more robust to the initial policies. In other words, even with the same innate characteristics, agent individuality still emerges.
Maat Mastionnteee cnstoneeen PORMER
Moan eneeic mar--- sidee
Mast MAAAMETEE POUME
plt
Frotade
Figure 6: Learning curves of intrinsic reward and envirommental reward of EOI+MAAC (left) and EOI+ (right) in Pac-Men.

---

Under review as a conference paper at ICLR 2021
Meae iniorment rooddd
24
DO OMIX
DO CMIX
Agont
201- -O0--C0XXX wi
Agent
Agert
QUT
10" 6:
10
Expodas
Figure Action distributions of the initial action-value functions QMIX.
Figure 8: Learning curves with EMI+ The dotted lines are the version with the same initial action-value functions.
4.2 WINDY MAZR
In the scenario, there are two pirates (agents) initialized at the bottom of the T-shaped maze, and two treasure boxes initialized at the two ends, as illustrated in Figure 3 (mid). Each agent has a local observation that contains a square view with 5 x grids and could move to one of four neighboring grids or open the box. There is wind running right from the dotted line. Shifted by the wind, forty percent of the time, the agent will move to the right grid whichever action takes. The agents could only get global reward, i.e., the number of the opened boxes, at the final timestep.
Figure shows the learning curves of EOI+MAAC and EOI+QMIX. Under the effect of the wind, it is easy to go to the right end and open the box. even if the agent acts randomly. Limited by the time horizon, it is impossible that the agent first opens the right box and then goes to the left end for the other box. MAAC and QMIX only achieve the mean reward of 1. Due the wind and the small state space, the trajectories of the two agents are similar, thus EOI without regularizers provides little help, where the intrinsic reward is not strong enough induce individuality. With regularizers, the observations on the right path and the left path can be discriminated gradually. In the learning process, we find that the agents will first both learn open the right box. Then one of them will change its policy and go left for higher intrinsic reward, and eventually the agents develop the distinct policies and get a higher mean reward.
The left box is hard to explore, thus we also compare EOI with EDTI (Wang et al., 2020b), multi- agent exploration method. EDTI gives the agent an intrinsic motivation, considering both curiosity and influence. It rewards the agent to explore the rarely visited states and to maximize the influence on the expected returns of other agents. The optimization of the intrinsic motivation is integrated with PPO (Schulman et al., 2017). However, EDTI does not perform well in this scenario. When the agent policies are similar, the two agents will be curious about similar states. When they both explore novel states, the environmental reward decreases, which draws the agents to go back to the right box again. So the curiosity is not stable, and we can see its fluctuation during training in Figure
4.3 FIREFIGHTERS
There are some burning grids two areas and two rivers other areas. Four firefighters (agents) are initiated at the same burning area, illustrated in Figure 3 (right). The agent has local observation that contains a square view with 5 x 5 grids and can move to one of four neighboring grids, spray water, pump water. They share water tank, which is initiated with four units water. Once the agent sprays water on the burning grid, puts out the fire and consumes one unit water. Once the agent pumps water in the river, the water in the tank increases by unit. The agents only get global reward, i.e., the number of the extinguished burning grids, at the final timestep.
1,00
1.55
L0E-1AMC
1,00
EDSHONT
L01-MAAC x/ on PO
6.7
L01-GND 7D
D06-MAAC xi Ki
TOI-ONIR 10
1ON-MAMC w/s Butt
EOI-ONMD Bich
MAAC
218X
Figure 9: Learning curves in Windy Maze: BOI+MAAC (left) and EOI+QMIX (rightt).
7

---

Under review as a conference paper at ICLR 2021
LOMK QUTE
QMIX EDI ROMA
22.3
HC
Madi
25:37
EX3P
3.5 11:32" 1x30
Eppoda
Tincstopo
Figure 10: Learning curves in Firefighters (left) and 8m (right).
As illustrated by Figure 10 (left), MAAC and QMIX fall into local optimum. The agents learn to put out the fire around the initial position until the water is exhausted, and get the mean reward 4. They do not take different roles of pumping water or putting out the fire farther. Benefited from the emergent individuality, EOI+MAAC achieves the best performance with clear division of labor, where two agents go to the river for pumping water, and two agents go to different areas for fighting with fire. EDTI could escape from the local optimum, but it converges to a lower mean reward than FOL. This because encouraging curiosity and influence cannot help division of labor, and the intrinsic motivation of EDTI is too complex to suit the cases with more than two agents.
ROMA (Wang et al., 2020a), based on QMIX, uses a role encoder taking the local observation to generate role embedding, and decodes the role to generate the parameters of individual action-value function in QMI.. However, ROMA converges to a lower mean reward than EOI+QMIX and learns much slower. Due to the large parameter space, generating various parameters for the emergent roles is less efficient than encouraging diverse behaviors by reward shaping. Moveover, since the agents initiated with the same observations, the role encoder will generate the similar role embeddings, which means similar agent behaviors at the beginning, bringing difficulty to the emergent roles.
HC (Lee al., 2020) first provides p each agent of diverse skills, which are learned independently by DIAYN (before 6 x 10x episodes), then learns high-level policy to select the skill for each agent. Since the agents are trained independently without coordinating with others' skills, the skills could be diverse but might not be useful, e.g., ignoring the pumping action. So it converges to a lower mean reward. We also train the primitive skills and high-level policy simultaneously, i.e., HC-simul (Yang et al., 2020), and find it falls into local optimum. Before the diversity emerges in the skills, the high-level policy learns to choose the skill of putting out the nearby fire for the cheap reward, however other skills would not be chosen to learn diversity. That is to say, HC-simul only considers diversity in the low-level policies but ignores that in the high-level policy. EOI encourages individuality with the coordination provided by centralized training, considering both of the global reward and other agents' policies. Therefore, BOI is more effective than HC and HC-simul. Moreover, EOI also outperforms ROMA and HC in Pac-Men, available in Appendix A.4.
4.4 STARCRAFT II
We additionally evaluate EOI on a micromanagement scenario of StarCraft II (Samvelyan et al., 2019) 8m, where Marines face enemy Marines and the need for the division of labor unknown. As illustrated in Figure 10 (right), EOI learns faster than ROMA and QMIX, showing the classifier Po (T(O)) could be effectively trained and provide useful intrinsic signals in the highly dynamic environment with many agents. Note that we represent the individual action-value function as MIP, not GRU, which means that the action value is generated only based on of, and the agents do not share the neural network parameters that the result of QMIX is different from that in the original paper (Rashid et al., 2018).
CONCLUSION
We have proposed EOI, a novel method for the emergence of individuality in MARL. EOI learns a probabilistic classifier that predicts probability distribution over agents given their observation and gives each agent an intrinsic reward of being correctly predicted by the classifier. Two regularizers introduced to increase the discriminability of the classifier. We realized EOI on top of two popular MARL methods and empirically demonstrated that EOI outperforms existing methods in a variety of multi-agent cooperative tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference Machine Learning (ICML), 2019.
Mehdi Dastani, Virginia Dignum, and Frank Dignum. Role-assignment in open agent societies, In International Conference Aulonomous Agents and Multiagent Systems (AAMAS), 2003.
Yali Du, Lei Han, Meng Fang, Ji Liu, Tianhong Dai, and Dacheng Tao. Lirr: Learning individ- ual intrinsic reward multi-agent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (ICLR), 2019.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Julia Freund, Andreas M Brandmaier, Lars Lewejohann, Imke Kirste, Mareike Kritzler, Antonio Krüger, Norbert Sachser, Ulman Lindenberger, and Gerd Kempermann. Emergence individuality in genetically identical mice. Science,
2013.
Deborah M Gordon. The organization of work in social insect colonies. Nature, 38066570 1996.
Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar DueñezzGuzzman, Antonio Gareía Castañeda, lain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. Advances in Neural Information Processing Systems (NeurIPS), 2018.
Shariq Iqbal and Fei Sha. Actor-attention-critie for multi-agent reinforcement learning. In Interna- tional Conference on Machine Learning (ICMI., 2019.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.
Youngwoon Lee, Jingyun Yang, and Joseph J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference on Learning Representations (ICLR), 2020.
Wengian Liu, Octavia Camps, and Mario Sznaier. Multi-camera multi-object tracking. arXiv preprint arXiv:1709.0700 07065, 2017.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed nooperaive-comppttiiee environments. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Kathryn Sarah Macarthur, Ruben Stranders, Sarvapali Ramchum, and Nicholas Jennings. A dis- tributed anytime algorithm for dynamic task allocation in multi-agent systems. In AAAI Conference on Artificial Intelligence (AAAI), 2011.
Kevin McKee, Ian Gemp, Brian McWilliams, Edgar A DuéñezzGuzzán. Edward Hughes, and Joel Z Leibo. Social diversity and social preferences mixed-motive reinforcement learning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning (ICML), 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farguhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1 9902.044433, 2019.

---

Under review as a conference paper at ICLR 2021
Pedro V Sander, Denis Peleshchuk, and Barbara J Grosz. A scalable, distributed algorithm for efficient task allocation. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2002.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In IEEE Conference Computer Vision and Paltern Recognition (CVPR), 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Tianmin Shu and Yuandong Tian. M Mind-aware multi-agent management reinforcement learning. In International Conference on Learning Representations (ICLR), 2019.
Mark Sims, Daniel Corkill, and Victor Lesser. Automated organization design for multi-agent systems. Autonomous agents and multi-agent systems, 16(2):151-18) 2008.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtram: Learning to factorize with transformation for cooperative multi-agen reinforcement learning, In International Conference on Machine Learning (ICML), 2019.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Valoe-decompositton networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and Mulliagent Systems (AAMAS), 2018.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjic Zhang. Multi-agent reinforcement learning with emergent roles. arXiv preprint 2rXiv:2003308033 2020a.
Tonghan Wang, Jianhao Wang, Yi Wu, and Chongjie Zhang. Influence- based multi-agent exploration. In International Conference on Learning Representations (ICLR), 2020b.
Jiachen Yang. Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2020.
A APPENDIX
A.1 LEARNING FROM EXPERIENCES OF OTHER AGENTS
When agents are exploring the environment, agents have various exploration patterns. The experiences of one agent can be helpful for the learning of other agents, i.e., improving data efficiency and accel- erating learning. Therefore, we propose an unbiased gradient estimate for learning from other agents' experiences. centralized training, all agents' experience tuples O', A", r 0++1, 8 g are stored in the replay buffer, and sampled as batch Ω to compute the gradients jointly. For each agent i, the probability of sampled observation is p(o)t). To use others' experiences, we randomly rearrange the agent index in subset of Ω2 denoted w. For example, in two-agent case, two agents exchange o', a., and +11 to build a new training experience. In the experiences with shuffled index w, the training experience of each agent is sampled based on p(o) rather than p(ol)). By Bayes rule,
p(o p(ilo)p(o) p(i)
Since the number of observations for each agent is equal in the training set, p(i) is constant. Then we have
p(oli) x p(i)o)p(o).
We use Pa (ilo) as importance weight (IW) for the gradient of individual action-valus function/policy to make sure that the estimate of the expected gradients based on p(o) is unbiased from that based on p(oli).
F Eggp(oos) ((.) x (oo
10

---

Under review as a conference paper at ICLR 2021
034-00 004-MAMC
0.35
EOH-MARE
FOlI MANS LED
MII-MAKHHLOO WS
600
Epiakoo
Epooda
Epeodo
(a) BOI+MAAC
(b) EOI+QMIX
(c) importance weight
Figure 1I: Learning curves of BOI with LFO on top of MAAC (a) and QMIX (b), and curve importance weight during the training (c) in Pac-Men.
Figure (a) and 11(b) show the learning curves of FOI (with both PD and MI) with unbiased learning from others' experiences (LFO) and without importance weight respectively in Pac-Men. LFO greatly accelerates the learning of EOI since the agents have access more experiences, and converges to the similar performance with EOI since the gradient estimation is unbiased. Applying LFO MAAC and QMIX also accelerates the convergence. Without important weight, LFO leads to lower mean reward, because the gradient estimate using others' experience is biased, which would harm the learning. To evaluate how much knowledge learned from others, in Figure (1(c) we plot the curves of mean important weight on other agents' experiences Er,, [ Eise P (ilo;)], where and j are the agent indexes before rearranging. At the beginning, as the agents are exploring the environment, the mean importance weight high, which indicates that the agent learns a lot from others' experiences. In later phase, the mean importance weight becomes very low, because individuality has emerged and the experiences agents are very different, the agent cannnot benefit from others' experiences. In the experiment, |w]/||[| is set as a constant 0.2. However, enlightened by the trend of the mean importance weight, we could also adjust | [ according to a pre-defined schedule, e.g., time-decaying or adaptive methods related to the mean importance weight.
A.2 MATUEMATICAL FORMULATION
Let and & denote the parameters the joint policies and the probabilistic classifier, respectively. Then, the whole learning process corresponds to the following bi-level optimization:
max BEE
J(0, o0 (())
s.t.
o" (0) arg min C('',0) E0
where J is the RL objective with intrinsic reward, is the loss function the probabilistic classifier,
and @ is an implicit function of 0. Therefore, to solve this optimization, we can iteratively update 0 by
aJ(O (0))))
0J(0, x))
(0) aJ(0, Φ)
d0
00
d0
aΦ
(= (0)
(0)
where
d (0)
(8 ((. 0)
d0
DOasT
doo0T
(d=)((0)
which is obtained by the implicit function theorem. In practice, the second-order term is neglected due to high computational complexity, without incurring significant performance drop, such as in meta-learning and GANs. Therefore, we can solve the bi-level optimization by the first-order approximation with iterative updates:
Φx+1 RJ arg min L(o,
0 0 + VoJ 0++11)
A.3 HYPERPARAMETERS
The hyperparameters of EOI and the baselines in each scenario are summarized in Table 1. Since QMIX and MAAC are off-policy algorithms with replay buffer, we do not need maintain the buffer
11

---

Under review as a conference paper at ICLR 2021
Table Hyperparameters
Hyperparameter
Pac-man
Windy Maze
Firefighters
8m
runs with different seeds
10
5
3 120
horizon (7)
30
15
20
discount (7)
0.98
batch size
128
64
replay buffer size
2 x 10
2 x 10
# MLP units
(128, 128)
(64, 64)
actor learning rate
1 x 10
critic learning rate QMIX learning rate
x 10-4
1 x 10 ReLU
MLP activation optimizer
Adam
ó learning rate a QMIX a in MAAC B1 82
x 10
0.05
0.04
0.01
0.1
0.01
A
I but build the training data from the replay buffer D. For EDTI, ROMA, HC, and HC-simul, we use their default settings. the experiments, the agents do not share the weights of neural network since parameter sharing causes similar agent behaviors. All the curves are plotted using mean and standard deviation.
A.4 ADDITIONAL RESULTS PAC-MEN
In Figure 12, we compare EOI with HC and ROMA in Pac-Men. EOI converges to a higher reward than these two methods for behavior diversification, showing that EOI more conducive to the performance improvement.
To investigate the performance of FOI in the scenarios where individuality is not necessary, we perform the experiments on easy version of Pac-Men, where the dots are only distributed in the bottom room, illustrated in Figure 13 (left). To complete this task, the agents do not need to go to different rooms. As depicted in Figure 13 (right), EOI+QMIX converges the same mean reward with the QMIX, which verifies that EOI does not conflict with the success of task where agent individuality is necessary.
In Figure 14(a), we test EOI+MAAC with different a to investigate the effect of a on the emergent individuality. When is too large, the agents will pay much attention to learn the individualized be- haviors, which harms the optimization of the cumulative reward. However, the results of EOI+MAAC with different a are all higher than that of the vanilla MAAC without EOI (a 0), which shows that the proposed intrinsic reward could encourage individualized policies for better cooperation. In Figure 14(b), we apply EOI to PPO, where each agent controlled by an independent PPO. EOI+PPO outperforms the version without EOI, which verifies the effectiveness of EOI on independent learning algorithms. Intuitively, maximizing the -l.-divereence between agents Σj K LLp (a)o) (a)o)) for agent as a regularization could also encourages the individuality. In Figure 14(c), we compare EOI against the KL-divergence and find that KL-divergence does not outperform the vanilla MAAC,
20--CM
Peecean
Herr anaroreeen eveers woedd
7410" 4-10 92.0 421P wea
Ephoode:
Figure 12: Comparison with ROMA and HC Pac-Men.
Figure 13: The easy version of Pac-Men (leff) and learning curves (right).
12

---

Under review as a conference paper at ICLR 2021
Ltits indorratnn 1P1
Inforder I910
Mau rewurd
<LAMAAC
Epiado
Epoods
Ephodes
(a) Different
(b) EOI+PPO
(c) KL divergence
Figure Learning curves of EOI+MAAC with different a (a), learning curves of BOI+PPO (b), and learning curves of KL divergence+MAAAC (c) in Pac-Men.
and the learning of KL-divergence is stable. We think that the KL-divergence term is hard optimize with more than two agents.
A.5 SIMILARITY AND DIFFERENCE BETWEEN EOI AND DIAYN
DIAYN proposed to learn diverse skills with single-agent RL the absence of any rewards. trains the agent by maximizing the mutual information between skills (Z) and states (S), maximizing the policy entropy, and minimizing the mutual information between skills and actions (A) given the state. The optimization objective
MI(S; 2) M(ASS) MI(A; ZIS)
H((2) H(2[S) H(A[S) (H(AIS) H((A
=H(Z) - H(Z|S) H(A[Z,S).
To maximize this objective, in practice DIAYN gives the learning agent an intrinsic reward log q{=[s) log p(z), where (= approximates p(=ls) and p(z) is a prior distribution. EOI gives each agent an intrinsic reward of p()lo). Let agents correspond to skills and observations correspond states, then EOI has similar intrinsic reward with DIAYN. However, simply adopting the intrinsic reward in MARL, which can be seen EOI without the regularizers, cannot provide strong intrinsic reward signals for the emergence of individuality. The two regularizers, which capture the unique characteristics of MARI,, strength the reward signals and promote the emergence of individuality in MARL.
13


Reviews:
[{"review_id": "zVe0OW2o3ma", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper contributes a method based on reward-shaping to encourage the emergence of distinct agent behaviors in fully-cooperative multi-agent reinforcement learning (MARL), within the paradigm of centralized training with decentralized execution. They propose to learn a classifier that predicts agent identity given the agent's observation, and use the classifier probability as an intrinsic reward that is added to the environment reward. As such, there is a positive feedback loop whereby agents are rewarded for being distinguishable and hence generate data that is easier for the classifier to predict correctly. They provide two regularizers to help kick-start the positive feedback loop when agents' policies are similar at initialization. The paper implements the method, called EOI, on top of two base MARL algorithms. Using three illustrative environments where optimum team performance requires distinct behaviors among agents, the paper shows that 1) EOI outperforms the two base algorithms and other baselines that address exploration and diversity; 2) the two regularizers improve performance and help the intrinsic reward to serve as an inductive bias; 3) agents do perform distinct behaviors. They also show comparable performance on a benchmark StarCraft II micromanagement task.\n\nI give a rating of 5 for the following reasons. Classification-based intrinsic reward have been used for diversification in MARL in previous work (Lee et al., 2020, Yang et al. 2020), but this paper provides a different formulation along with effective heuristics to speed up the initial training of the classifier. The experiments are clear in showing the benefit of EOI and EOI + regularizers when built on top of the two base methods.\nHowever, there are main points of concern. I am willing to raise my score if the main points below are sufficiently addressed.\n\nThis paper incorrectly positions itself against the previous works Lee et al. (2020) and Yang et al. (2020). In section 2, the authors say \"none of these studies appropriately link behavior diversification to the success of the cooperative task\". However, in section 3, the authors say that \"EOI directly correlates individuality with the task by intrinsic reward\", making it seem that EOI differs from Lee and Yang in that regard. This is not true. The intrinsic reward in EOI is the probability of correct classification of an agent identity. The intrinsic reward in Lee and Yang are based on correct classification of a latent skill variable. Neither agent identity nor a latent skill variable are directly linked to task success. The intrinsic rewards in EOI and in these previous work encourage the emergence of distinguishable behavior; none of them have any direct link to task success. The authors either need to explain their claim in more detail, or amend the paper.\n\nAnother main concern is about the case where EOI is built on top of QMIX. The authors define $Q^p_i( o_i, Q^a_i(o_i))$, which takes in the vector of individual action-values $Q^a_i(o_i)$, and define a TD error loss with the intrinsic reward $p(i|o)$. In the first equation in Section 3.3, the TD error serves purely to _estimate_ the expected cumulative intrinsic reward under the current policy. It does not find some Q-function that _optimizes_ the expected cumulative intrinsic reward. However, the initial formulation on page 3 (where the total reward is the environment reward plus the intrinsic reward) implies that the agents should maximize the expected cumulative intrinsic reward as well. Therefore, it is not clear at all how the intrinsic reward is being maximized in the QMIX case.\n\nPoints that did not affect the decision, but where the paper can be improved:\n1) All equations should be numbered for easier reference.\n2) First sentence in abstract needs a language check.\n3) The introduction says \"...agents tend to visit their own familiar observations\". It seems that this has a negative impact on the exploration by an arbitrary agent. Authors should include more discussion.\n4) Section 3.3, a more succinct way is to say that assigning the intrinsic reward to the specific agent bypasses the multi-agent credit assignment problem.\n5) In Section 3.3, it is not clear what the authors mean by saying that the individual value (in QMIX) has no \"actual meaning\". Please be more precise.\n6) At the end of Section 3.3, the authors say that due to off-policy learning, $p_{\\phi}(i|o)$ is recomputed for each sampled batch. Is it recomputed purely because the current classifier is presumably more accurate? Is there any potential problem with doing so, in the context of off-policy learning?\n7) All plots need thicker line width, including the legend. Colors need to be more distinguishable from one another. Currently, it is hard to see which curves correspond to which algorithms.\n\n---\nEdit: score increased from 5 to 6", "extra_scores": {}, "extra_text": {}}, {"review_id": "ktO1Lf7KirH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper discusses an analysis of the emergence of individuality in a multi-agent system based on reinforcement learning. The emergence of individuality is based on intrinsic reward. The intrinsic reward is assigned to each agent by forcing them to be \"different\" according to a given distribution. The reviewer struggles to see this mechanism particularly insightful. This does not appear to the reviewer as emergence of individuality; his looks more like the emergence of different behavior (which can be seen as individuality, but it is more like forcing different behavior, not individual behavior, as interpreted normally when we study societies in my opinion).\n\nStrength:\n\n- According to the reviewer, any study that sheds a light on human phenomena using reinforcement learning is fascinating. \n\nWeakness: \n\n- The model used for studying the emergence of individuality is not convincing in my opinion. In fact the authors essentially impose the emergence of individuality from outside by giving a reward for being different. \n\n- The reviewer wonders if the emergence of individuality at the end of the day might actually emerge just by considering individual learning models. I would say that this is what you observe in many multi-agent systems with individual learning models. What is the main different in this case? This is not completely clear to the reviewer.\n\n- The choice of the mechanisms (MAAC, QMIX, etc.) is not sufficiently discussed by the authors.\n\n- The goal of the evaluation is unclear: the goal of this evaluation is not the emergence of individuality in my opinion.\n\nIn general, I would say that the question that the authors try to address is indeed a fascinating one. However, the reviewer really struggles to understand the choice of the mechanism selected by the authors. The reference to biology are not convinced. The authors say: \"Biologically, the emergence of individuality is attributed to innate characteristics and experiences. However, as in practice RL agents are mostly homogeneous, we mainly focus on enabling agents to develop individuality through interactions with the environment during policy learning\". Is it not something that happens if you have individual models in any case? It seems to me that the individuality/differences are artificially imposed on the system itself.\n\nOverall, the contribution of the paper is not significant in my opinion, since unfortunately the work does not really provide insights about emergence of individuality, which is the main stated goal of the paper.  \n\nQuestions:\n\n1. How do you map the mechanism used for the emergence of individuality to real situations? This appears in a sense as your goal, but it is difficult to see how it is possible to interpret the results of the simulation considering it as a simulation of a real-world society.\n\n2. In the Introduction the authors say \"Analogically, the emergence of individuality should also be essential for multi-agent cooperation\". I would claim that this is difficult to prove by analogy as stated by the authors.\n\n3. The goal of the evaluation is not completely clear. In fact you simulate a variety of games, etc. However, it is unclear how this  is related to the fundamental problem of the paper, which is the emergence of individuality.\n\n4. Why do you use \"positive distance\"? What is the motivation of this choice?\n\n5. The discussion about QMIX is not completely clear. Why is it necessary? How do you link this with the goal of the paper? Is there any clear mapping with real-world situations?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jM2Q0KpXWqt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of exploration in multi-agent RL,\nformulated as a Dec-POMDP. The authors propose to shape the reward\nusing the output of a classifier that tries to determine which agent\nsaw a particular observation. Moreover, the authors propose some\nregularization schemes to \"break ties\" early on in the training. Then,\nit is shown that the proposed reward shaping term can be integrated\ninto two popular MARL algorithms that use centralized training: MAAC\nand QMIX. The integration into MAAC is relatively straightforward\nbecause MAAC uses independent critics, whereas the integration into\nQMIX is more involved due to the Q-function mixing step.\n\nI believe this is a very important problem being tackled by the\nauthors. Even in single-agent RL, exploration is a major issue; this\nonly gets even more apparent in the multi-agent setting. I believe\nthat the paper was well-written, and the Introduction made it very\nclear what exactly the paper's contributions were, which I greatly\nappreciate.\n\nUnfortunately, I'm not sure that I believe that the proposed bias is\ntruly useful. I understand the intuition: we want agents to\n\"specialize\" their observations, such that it is easy to predict which\nagent is receiving any particular observation. However, won't this be\ncounterproductive in most practically interesting domains? For\nexample, consider a team of robots working together in a factory or a\nhousehold; they will constantly be changing their environment as they\ntake actions toward operating the factory equipment or cleaning the\nkitchen, which means the observations they receive will always be\nchanging. But the proposed reward shaping mechanism in this paper\nwould \"fight\" against this progress, because it would encourage the\nagents to engage in trivial behavior just to be able to see the same\nobservations over and over. In my mind, it would be important to\nconsider the dynamics of how the classifier p(i | o_i) is changing\nover time.\n\nAnother option could be to simply encourage the agents to learn\ndifferent policies, maybe measured via KL-divergence of P_{agent 1}(a\n| s) and P_{agent 2}(a | s). I believe that this bias would be\nsufficient to solve the example presented in Figure 1? Does this bias\nseem reasonable, and if so, how does it compare against EOI?\nBasically, more broadly, I would have liked to see experimental\ncomparisons that convince me that EOI is the *correct* bias to use\nversus other natural biases, whereas the current experiments only seem\nto compare against non-shaping methods like ROMA and HL.\n\nSome other questions:\n\n1. How significant is the fact that you are ignoring second-order\neffects in solving the bi-level optimization in Appendix A.2? Have the\nauthors conducted preliminary experiments to prove that it doesn't\nmake much difference? I understand that this practice is standard in\nmethods like MAML, but it would be nice to verify that it doesn't\nmatter in this setting either.\n\n2. Looking at the shaped reward computation, r + alpha * p(i | o_i),\nit seems like if the classifier were naive and simply outputted a\nuniform distribution, you would still be giving positive intrinsic\nreward in that case. Might it make more sense instead to consider the\ndivergence between p(i | o_i) and the uniform distribution?", "extra_scores": {}, "extra_text": {}}, {"review_id": "aEgFjI22Qh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This article solves the Dec-POMDP problem, based on MAAC and QMIX algorithms. When the agents are homogeneous and optimized only through team rewards, it is easy to learn similar policies for each agent. This makes the multi-agent algorithm finally converge to a locally optimal joint policy. If each agent can complete different parts of the overall goal, the joint policy that converges to is obviously better. Many current works model the above problems as task assignment or role assignment problems. However, the agents with different roles in these works are basically rule-based, and the tasks are all manually defined. There are also some works that unsupervisedly generate different strategies and roles by introducing diversity constraints, but the generation process has nothing t o do with the task. In order to solve the shortcomings of the above methods, this paper proposes a MARL method based on reward shaping to encourage the division of labor between agents, and at the same time introduces two regularize terms to solve the problem of too similar agent policies at the initial training stage. At the same time, reward shaping and reinforcement learning are optimized simultaneously, forming a bi-level optimization problem. The paper also designed three tasks that emphasize division of labor to verify the effectiveness of the algorithm.\n1. Neither the MAAC nor the QMIX algorithm on which the paper is based has good scalability. Although the independent learning algorithm is simple, it can achieve better performance on many tasks and has good scalability. This paper should additionally use independent learning algorithms as baselines, and apply the intrinsic rewards proposed in this paper to independent learning algorithms.\n2. The three tasks used to verify the algorithm in this paper are all specially designed, with a strong emphasis on division of labor. I think the paper should additionally explain the limitations of the algorithm, such as which scenarios will be more effective and which scenarios will limit the learning ability of the agent.\n3. The optimization process of bi-level problems is very unstable. The algorithm proposed in this paper contains many hyperparameters, and the sensitivity of the algorithm to hyperparameters should be shown in the experimental part.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a method to increase diversity/individuality of agents in a MARL setup, based on intrinsic rewards coming from a classifier over behaviours.

Reviewers tend to agree that this is an important/interesting problem, which is related to exploration, a central problem in reinforcement learning. Several reviewers point out that the paper is well written. I appreciate that the authors have been responsive to reviews and have answered and/or addressed several points of concern of the reviewers. The proposed method performs well on the experiments carried out.

Reviews still point out several things that could be improved. The experiments mostly report reward curved, and only few results are actually clearly pointing out the individuality between agents. The fact that this method outperforms the baselines is good, but does not prove individuality and may simply be due to the authors spending more time on the tasks, or other undiscovered phenomenon.
A reviewer is concerned that this extra reward could encourage trivial behaviours, and it seems clear that it will if the relative weight of the intrinsic reward is too high. This should be discussed more.
Finally, a reviewer points out that classifier-based intrinsic reward for diversity already exists in published works and that this paper is incremental work.

The average score for this paper is very close to the acceptance threshold, but based on the reviews I recommend to reject this paper for ICLR 2021. I am confident that when the authors address further the reviewers concerns and improve the experimental results, this paper will be published in a future venue.
Decision: Reject