ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward
Authors: Zixian Ma
Rose E Wang
Li Fei-Fei
Michael S. Bernstein
Ranjay Krishna
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-agent intrinsic reward
multi-agent reinforcement learning

Abstract: Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well. However, centralized training and dense rewards are not readily available in the real world. Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards. To address these issues, we propose a self-supervised intrinsic reward  \textit{ELIGN - expectation alignment - } inspired by the self-organization principle in Zoology. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with expectation alignment learn behaviors that match their neighbors' expectations. This allows the agents to learn collaborative behaviors without any external reward or centralized training. We demonstrate the efficacy of our approach across 6 tasks in the multi-agent particle and the complex Google Research football environments, comparing ELIGN to sparse and curiosity-based intrinsic rewards. When the number of agents increases, ELIGN scales well in all multi-agent tasks except for one where agents have different capabilities. We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves, break coordination symmetries, and confuse adversaries. These results identify tasks where expectation alignment is a more useful strategy than curiosity-driven exploration for multi-agent coordination, enabling agents to do zero-shot coordination.

ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Reward
Zixian Ma', Rose E. Wang', Li Fei-Fei', Michael Bernstein' , Ranjay Krishna 1.2
Stanford University", University of Washington"
riixiama,rovaangeefffeeiii msb, ranjaykrjjaaiiiin @cs stanford, edu
Abstract
Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well. However, centralized training and dense rewards are not readily available in the real world. Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards. To address these issues, we propose self self- eeffssuprviiee intrinsic reward ELIGN
expectation alignment inspired by the self-organization principle in Zoology. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with expectation alignment learn behaviors that match their neighbors' expectations. This allows the agents to learn collaborative behaviors without any external reward centralized training. We demonstrate the efficacy our approach across tasks in the multi-agent particle and the complex Google Research football environments, comparing ELIGN to sparse and curiosity- based intrinsic rewards. When the number agents increases, ELIGN scales well all multi-agent tasks except for where agents have different capabilities. We show that agent coordination improves through expectation alignment because agents learn to divide tasks amongst themselves, break coordination symmetries, and confuse adversaries. These results identify tasks where expectation alignment a useful strategy than curiosity-drive exploration for multi-agent coordination, enabling agents to do zero-shot coordination.
1 Introduction
Many real world Al applications can be formulated as multi-agent systems, including autonomous vehicles (Cao et al. 2012), resource management (Ying & Dayong 2005). traffic control (Sunehag et al. 2017). robot swarms (Swamy et al. 2020), and multi-player video games (Berner et 20,12 Agents must adapt their behaviors to each other order coordinate successfully in these systems. However, adaptive coordination algorithms are challenging to develop because each agent is not privy other agents' intentions and their future behaviors (Foerster al. 2017).
These challenges are more acute in decentralized training under partial observability than centralized training full observability. In the real world, agents act under partial observability and learn in decentralized manner: they do not learn collaborative behaviors with a single centralized algorithm with complete knowledge of the environment (Iqbal & Sha 2019 Liu et al. 2020). Unfortunately, the most successful multi-agent algorithms train agents with a centralized critic, assuming access to all agents' observations and actions (Foerster al., 2018 Rashid al. 2018 (Sunehag al. 2017] Lowe et al. 2017). The most successful multi-agent algorithms for decentralized training and partial observability assume task-specific reward shaping (Jain al. 2020 Iqbal & Sha (019). which is expensive to generate. These algorithms struggle to learn with sparse reward structure.
Consider cooperative navigation task, where N agents aim to simultaneously occupy N goal locations. A centralized algorithm with full observability is capable of optimally assigning the nearest
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Observation at
Fredicted next ocservation: o/
Oceervation at t+1:o
Fiuloy ng)
Jo (0
M
JeX
Agontt
fe
Agent]
Goal
Raceppio feild
Agent dynamics prediction of
(s) Aligned - high
[o] Miseligned - low Fim
Figure I: We introduce ELIGN, i.e, expectation alignment, task-agnosti intrinsic reward to improve multi-agent systems. Intuitively, ELIGN encourages agents to become more predictable to their neighbors. An agent (e.g., agent here) learns to behave in ways that match its neighbors' (e.g.. agent j's) predictions of its next observation. Here, agent j expects agent i to move up instead of down, moving closer point of interest above it. Agent i attains (a) higher reward when its action (e.g., upward) aligns with this expectation or (b) a lower reward when its action (e.g., downward) is misaligned.
goal location to each respective agent. However, with partial observability, agents can see only a handful of goal locations and other agents. They unaware of others' observations, actions, and intentions with decentralized training. We observe that agents simultaneously occupy the same goal; they fail to collaborate because they do not predict which goal each agent is expected to occupy. To overcome instances of miscoordination, decentralized algorithms have adapted single-agent curiosity- based intrinsic rewards (Pathak et al., 2017, (tadie Stadie al.) 2015). Multi-agent curiosity-based rewards incentivize agents to explore novel states (Igbal & Sha, 2020). Although curiosity helps agents discover new goal locations. it doesn't solve solve the challenge of coordination, such assigning goals to each agent. Only a few attempts explore other forms of multi-agent intrinsic rewards (Igbal & Sha 2020 Böhmer et al.) 2019; Schafer 2019).
In this work, we propose ELIGN as a novel multi-agent self-supervised intrinsic reward, enabling decentralized training under partial observability. Intuitively, expectation alignment encourages agents to elicit behaviors that decrease future uncertainty for their team: it encourages each agent to choose actions that match their teammates' expectations. Going back to the cooperative navigation task, expectation alignment encourages each agent to move to goals others expect it to occupy. like goals that are either closest the agent or goals that other agents aren't moving towards (Figure I We take inspiration from the self- oilf-organizttion principle in Zoology (Couzin 2007). This principle hypothesizes that collective animal intelligence emerges because groups synchronize their behaviors using only their local environment; they do not rely on complete information about other agents and can coordinate successfully by predicting the dynamics of agents within their field-of- ofl-owf-viww (Collett et al.) 1998 Theraulaz & Bonabeau 1995] Ben-Jacoh et al. 1994 Buhl et al., 2006). Similarly. expectation alignment as an intrinsic reward is calculated based on the agent's local observations and its approximation of neighboring agents' expectations. It does not require a centralized controller nor full observability. ELIGN task-agnostic and we apply it to both collaborative and competitive multi-agent tasks.
We demonstrate the efficacy of our approach in the multi-agent particle and Google Research football environments, two popular benchmarks for multi-agent reinforcement learning (Lowe et al.) 2017 Kurach al. 2019). We evaluate ELIGN under partial and full observability, with decentralized and centralized training, and in terms of scalability. We observe that expectation alignment outperforms sparse and -uriosity-based intrinsic rewards (Ndousse et al.] 2021] Stadie et al.) [0155] Iqbal & Sha, (020), especially under partial observability with decentralized training. We additionally test expectation alignment as way to perform zero-shot coordination with new agent partners, and investigate why ELIGN improves coordination. We show that agent coordination improves through expectation alignment because agents learn divide tasks amongst themselves and break coordination symmetries (Hu et al. 2020).

---

2
Related Work
Our formulation of expectation alignment, a task-agnostic intrinsic reward for multi-agent training, draws inspiration from the self-organization principle in Zoology, which posits that synchronized group behavior is mediated by local behavioral rules (Couzin 2007) and not by a centralized controller (Camazine et al.] 2020). Group cohesion emerges by predicting and adjusting one's behavior that of near neighbors (Buhl et al. 2006). This principle underlies the coordination found in multi-cellular organisms (Camazine et al., 2020). the migration of wingless locusts (Collett et al., 1998), the collective swarms of bacteria (Ben-Jacob al. 1994), the construction of bridge structures by ants Theraulaz & Bonabeau 1995). and some human navigation behaviors (Couzin 2007).
Intrinsic motivation for single agents. Although draw inspiration from Zoology for formaliz- ing expectation alignment as an intrinsic reward, there is a rich body of work on intrinsic rewards within the single-agent reinforcement learning community. To incentivize exploration, even when non-optimal successful trajectories are uncovered first, scholars have argued for the use of intrinsic motivation (Schmidhuber) 199T). Single-agent intrinsic motivation has focused exploring previ- ously unencountered states (Pathak et al. 2017 [Burda et al. 2018a), which works particularly well in discrete domains. In continuous domains, identifying unseen states requires keeping track of an intractable number of visited states; instead, literature has recommended learning a forward dynamics model predict future states and identify novel states using the uncertainty of this model (Achiam & Sastry] 2017). Other formulations encourage re- re-viiiting states where the dynamics model's prediction of future states errs (Stadic al., 2015: Pathak et al.) (017). Follow up papers have improved how uncertainty (Kim et 2020) and model errors (Burda et al. 2018b Sekar et al. 2020) are calculated.
Intrinsic motivation for multiple agents. Most multi-agent intrinsic rewards have been adapted from single-agent curiosity- based incentives (ūBöhmer et al.; 2019 Schafer| (019)) and have pri- marily focused on cooperative tasks. They propose intrinsic rewards to improve either coordination, collaboration, or deception: These rewards either maximize information conveyed by an agent's actions (3Chitnis et al. 2020; Wang et al., 2019), shape the influence of an agent (Jaques al.. 2019 Foerster et al. (017). incentivize agents to hide intentions (Strouse et al. 2018). build accurate models of other agents' policies (Hernandez-
al. 2019 Jaques al. 2019 or break extrinsic
rewards for better credit assignment (Du et al.) 2019).
Several multi-agent intrinsic rewards (Hernandez-Leal et al. 2019 Jaques et al.) 2019). including ours, rely on the ability to model others' dynamics in a shared environment. This ability is a key component to coordination, closely related to Theory of Mind (Tomasello et al.. 2005). Our work can be interpreted as using a Theory of Mind model of others' behaviors to calculate an intrinsic motivation loss. Unlike existing Theory of Mind methods that learn a model per collaborator (Roy eet al., 2020), we learn a single dynamics model, allowing our method to scale as the number of agents increase. Our proposal is related model-based reinforcement learning (Jaderberg et al., 2016; Wang et al. 2020a); however, instead of learning a dynamics model for control, we learn a dynamics model as a source of reward. Our work is closely related to recently proposed auxiliary loss on predicting an agent's 's own future states (Ndousse et al., 202I). However, there are three key differences. First, their work predicts ego-agent observations, whereas our work additionally predicts future observations from the other agents' point of view. Second, their loss optimizes state embeddings while ours optimizes agents' policies. Third, their work focuses on cooperative tasks whereas ours applies to both cooperative and competitive domains.
Multi-agent reinforcement learning algorithms. Today, the predominant deep multi-agent frame- work uses actor-critic methods with a centralized critic and decentralized execution (Lowe et al.) 2017: Foerster et al. 2018; Iqbal & Sha 2019: Liu et al. 2020 Rashid al. 2018). This framework allows a critic to access the observations and actions of all agents to ease training. However, there are several situations where centralized training may not be desirable or possible. Examples include low bandwidth communication restrictions or human-robor tasks where observations cannot be easily shared between agents (Ying & Dayong/ 20055 Cao et al., (2012; Huang al.) (015). Decentralized training is therefore the most practical training paradigm but it suffers from unstable training: the environment is nonstationary from a single-agent's perspective (Lowe et al. (017)). Our work uses a decentralized training framework and tackles the nonstationarity challenge with an intrinsic reward designed to improve an agent's ability to model others. We also apply expectation alignment to centralized training and observe that it still aids cooperative and some competitive tasks.
3

---

3 Background
We formulate our setting as a partially observable Markov game (S, A, Tex. N) LLttmann 1994).
A Markov game for N agents is defined by a state space S describing the possible configurations of the environment. The observation space for agents is (O....., ON) and the action space is A == (A,..... AN). Each agent i observes O; E O,, a private partial view of the state, and performs actions a, E A., Using the observation, each agent uses stochastic policy тo, O, x A, - [0, 1), where 0; parameterizes the policy. The environment changes according to the state transition function which transitions to the next state using the current state and each agent's actions, : S
A S. S x A I. The team's goal is maximize the total expected return: R = Ei where 0 05 S is the discount factor, / is the time step, and 7' is the time horizon. The environment may also contain adversarial agents who have their own reward structure.
The team of agents obtains shared extrinsic reward
function of the environment state, Tex
4 Expectation Alignment
To understand expectation alignment intuitively, let's revisit the cooperative navigation task, where
N agents are rewarded for simultaneously occupying as many goal locations as possible. In Figure] [ agent i has a dynamics model trained on its past experiences. It predicts how future states will cvolve from the point of view of agent j. who is within i's view. In this example, j will expect i to move towards the goal since is closer it. ELIGN encourages i to pursue the action that expects (Figure Ta).. In turn, j can now assume that the observed goal location will eventually be occupied by i and should therefore explore to find another goal. By aligning shared expectations, agent behaviors become more predictable. Conversely, when neighbors behave opposite to an agent's predictions, the agent 2002]. For example, in Figure (b), if agent observes running away from a goal, this surprising behavior might indicate the existence of adversary outside j's receptive field.
infer about the environment outside of its own receptive field (Krause et al.)
Our training algorithm consists of three interwoven phases of learning a dynamics model, calculating the ELIGN reward, and optimizing the agent's policy (Algorithm I).
4.1 Training the dynamics model
Similar to prior work (Wang et al. 2018 Kidambi et al., 2020), each agent learns a dynamics model fo, to predict the next observation 0g given its current observation and action o;, 0z i.e,
of = fe. (o,, oz).
We use a three-layer Multi-Layer Perceptron with ReLU non- linearitiies as the dynamics model. We minimize the mean squared error between its prediction and ground truth next observation of.
4.2 Calculating intrinsic reward
The intrinsic reward captures how well agent i aligns to its neighbors' (e.g., agent j's) expectations on its next state. Calculating this reward requires j to accurately predict i's behavior, simulating Theory of Mind (Tomasello et al. 2005). As suggested by the erff-organzzation principle, must learn to align j's predictions. Ideally, the ELIGN intrinsic reward is calculated as:
Pin 0.)
Σ JEN())
1 - fo, fo,(o,,)))
WOO
where (1)) is the set of neighbors within i's receptive field, including itself. The ELIGN reward high when the average L2 loss is small, i.c. when i's actual next observation is close to agent j's predicted observation of for all in its neighbors. In that case, has chosen an action that aligns with j's expectations of how should act.
In a decentralized training setup, however, doesn't have access to j's dynamics model fa,, so i approximates j's dynamic model with proxy: its own dynamics model fo, and the knowledge of agent j's observation radius. Such an approximation ecologically valid since we often approximate others' behaviors using second-order cognitive Theory of Mind (Morin) 2006). Additionally, i

---

doesn't have access to j's entire observation; so, we restrict the future prediction from j's point of view by using the portion of j's observation can see: Oinj = o, @ oj. Agent i's decentralized intrinsic reward then becomes:
(in a4) = [W(1)
Io - So, (oans az)l JEN(()
Σ
We found that the approximation of fo, using fo, works well empirically. Dynamics model losses for all agents quickly decrease within 5-10 training epochs. we validate its applicability in scale heterogeneous multi-agent tasks where agents have variable capabilities, although we find the methods perform similarly when more heterogeneous agents are added.
small-
4.3 Policy learning
Algorithm ELIGN: Expectation Alignment
Once the ELIGN rewards calculated, the to-
1: Initialize replay buffer D and D'
tal rewards at each step for each agent i is: T;
2: Initialize N agents with random 8,: i []
Tex + +r (og, az) where Tex is the extrinsic reward
3: while not converged do
provided by the environment and / is a hyper- parameter for weighing the intrinsic reward in the agent's overall reward calculation. In prac- tice, we set /3 be |O:] where [O:] is the obser-
4: for = do
5: Populate buffer D with episode using policies (To, (ToN)
6:
end for
vation dimension; we find this scale generalizes well across tasks. Since contribution is ag- nostic to any particular multi-agent training algo- rithm, the team of agents now be trained using any multi-agent training algorithm to maximize returns R = Σt= y'r.
7:
TRAIN DYNAMICS MODEL
8:
for agent i 1. N do
9:
Sample transitions: {(oi, ai, Tex, 0f))
D;
10:
Predict o; (o; a;)
11:
Update dynamics 0, using of.
Both centralized and decentralized training al- gorithms can make use of these rewards. We primarily the multi-agent decentralized vari-
12:
end for
13:
CALCULATE ELIGN REWARD
14:
for agent i 1. N do
ant of the soft-actor critic algorithm in our ex- periments (Haarnoja al., 2018 Iqbal & Sha 2019). Compared to centralized joint-action train- ing, whose action space grows exponentially in N agents, our decentralized method has linear space complexity. Further, decentralized training
15:
Sample
B
transitions:
{(oi, az Tex: o,)} D;
16:
Compute intrinsic rewards rin (o, a;) Add {(ii, di Tex + Brin, of)) to D2. end for
17:
18:
19:
// POLICY LEARNING
20: Update all 0,s using transitions from
can parallelize training time to be less than linear
with respect to N. Although we present results
21:
end while
with one centralized training framework, studying the impact of expectation alignment with all the centralized critic frameworks is out of scope for this paper.
4.4 Extending expectation alignment to competitive tasks
We extend the ELIGN formulation to competitive tasks where a team of agents compete against adversaries. In this case, agents are encouraged misalign with their adversaries' expectations, i.c, agents are incentivized to be unpredictable to their adversaries within its receptive field (Nadv
Tin
(N)dv(1)
Σ
I!o5nR ffe (oanke a;))
5 Experiments
Our experiments explore the utility of using expectation alignment as an intrinsic reward compared to sparse and curiosity-based intrinsic rewards. We primarily focus on decentralized training under partial observability. However, we also demonstrate that ELIGN can easily augment centralized methods and assist in fully observable tasks. We vary the number of agents in the multi-agent particle tasks to test scalability. We end by investigating how and why ELJGN improves coordination by designing three evaluation conditions. First, does expectation alignment improve coordination by
5

---

Table 1: We report the mean test episode extrinsic rewards and standard errors of decentralized training under partial observability in multi-agent particle and Google Research foot- ball environments. setheam outperform SPARSE and both curiosity- suriosity-based intrinsic rewards. ELIGNady achieves the best performance among all competitive tasks except for Physical deception, where eem is the best. These results demonstrate the benefit of using alignment intrinsic reward to train better decentralized policies under partial observability.
Conperative
Competilive
Task (Agth Advi)
Coop may (3v0)
Hetero nuw. (4v0)
Phy decep. (2+1)
Pred-prey (2x2)
Keep-away (2-2)
3v1 w/ keeper (3v2)
SPARSE!
139.07 13.63
264.42 + 12.83
93.00 8.61
4.72 2.4
4.58 3.27
0,020 + 0.001
LUIRU
123.93 = 7.00
226.22 + 9.57
68.80 + 7.55
6.50 = 2.18
11.88 =: 2.88
0,124 + 0,004
CURIO wen
11.95
202
85.31 11:55
-3.57 1.75
5.114
(U12)
ELIGNee
9:
69.91
-7.58 2.55
141.04 8.00
311.67 10.88
7.00 2.60
2:96
0J0z2 0.001
RL RLIGN
02:20 4.23
-2.51 1.70
19.46 5.05
0 0225 0.001
HAND-CRAR
76.66
18.90
22x 16 15.58
91.26 14
-0.77
0.17
62:14 - 3.11
Lowe 2017
Kumcc 2019
al.
2015
2020
Tex: Occusoncy (11/Callson (1) Court Per Seap
Coop [3/0]
Hatero (4:0)
Pty
Prodaton
Kex-
A22S of
Figure 2: We plot the average test occupancy/collision count per step of decentralized algorithms under partially observable multi-agent particle tasks. On these metrics, ELIGN and ELIGN perform the best on cooperative and competitive tasks respectively.
breaking symmetries (Hu et al. 2020: generalization to new partners? Lastly, is the dynamics model critical in aligning agent behaviors?
Wang et al., 2020b) Second, does ELIGN enable zero-shot
5.1 Environments
We evaluate ELIGN across both cooperative and competitive tasks in the multi aagett particle envi- ronment (Mordatch & Abbeel 2017 Lowe et al. 2017) and the Google Research football environ- ment (Kurach et al. 2019).
State and action space The multi-agent particle environment is a two-dimensional world. The Google Research football environment is three-dimensional world. Both environments have continuous state spaces and discrete action spaces. Particle agents observe all agents' positions and velocities. They can "stay" change their velocity in one of the four cardinal directions. Each football agent controls one player. Players observe the ball, other players' positions and directions. They can apply one of ten actions from reft"" "top", "top_right", "right", "bottom_righ right", "bottom". "bottom_leff" "sprint", and "dribble".
Observability The original environments assume full observability, where each agent can observe the position p = (xr y) and velocity v = (Az, Ay) of all agents; each agent's observation vector is thus 0z [p1. (P1....,PP PN, 01...., UN]]. We extend these environments to be partially observable, where agent observes only the portion within its receptive field; like prior work with partial observability (Corder et al.) 2019),
hide the position and velocity information of any agent outside of agent i's receptive field; i.e, if the Euclidean distance between agent and j surpasses vicinity threshold т, then Pj and by are in Oi.partill. We set T 0.5 for partially observable and oo in the original fully observable case, where the world's width and height are 2.0 in the multi-agent particle environment and 0.84 2.00 in the Google Research football environment. Both environments also contain objects such as obstacles, goals and ball; they are similarly hidden if of sight. Partial observability is a more ecologically valid training condition since most agents in real-world tasks can only observe a small portion of their environment at given time.
5.2 Tasks
Multi-agent particle environment We use the following tasks from the multi-agent partiele environ- ment (Lowe et al. 2017 Liu et al. 2020). We choose N based prior work.

---

Table 2: We report the test episode extrinsic rewards and standard errors as the number of agents increased and trained using of decentralized algorithms. When the number of agents increases, one of ELIGN still performs the best in all tasks except for Heterogenous navigation.
Cooperative
Competitive
Task (Agt # vs. Adv #) Coop nav, (5v0)
Hetero nav, (6v0)
Phy decep. (4v2)
Pred-prey (4v4)
Keep-away (4v4)
SPARSE!
459. 92 22.44
616.62 25.30
160.89 + 27.72
-28
0.75
CURIO
158. 15 19.79
702.73 = 18.57
146.35 + 29.05
-5533 6.16
10.52 : 5.48
CURIO
497.15 11.47
695.38 12.22
84.65 1: 16.94
-17.21 8:2
1.40 4: 2.06
ELIGNat
198 21 = 9.77
646. 70 23,
137.38 30.000
-9.14 5.57
9.83 = 11.22
ELIGN
488 83 20.82
638.74 74 28.93
186,83 1: 21.92
-21.4 5.98
2.07 1: 4.35
ELIGNat
182.61 + 17.63
-21.37 7.02
11.29 9.02
Lowe al. 2017
Stadie et al. 2015
lighl & Sha 2020
Cooperative navigation: N agents must cooperate to reach a of N goal locations. Agents are collectively rewarded based on the occupancy of any agent on any goal location.
Heterogeneous navigation: N agents must reach N goals but they differ in speeds and sizes. N agents are slow and big, and the other 2/2 agents are fast and small.
Physical deception: N agents cooperate to reach a single goal location and are rewarded any one occupies the goal. However, they are penalized when any of N adversaries occupies the goal and 2 gets rewarded. The adversaries do not know which landmark is the goal and must infer it from the agents' behavior. The agents should learn to deceive the adversaries by covering all the landmarks.
Keep-away: There are N landmarks, one of which the goal and known to N agents. Agents are rewarded for occupying it and preventing M adversaries from reaching it. Adversaries are rewarded for pushing the agents away from the goal, but they can only infer which landmark is the goal.
Predator-prey: N slow adversaries chase and capture N fast cooperating agents around randomly generated obstacle-filled environment. Each time an adversary catches an agent, the agent penalized and the adversary is rewarded.
We the Academy 3vsl with Keeper competitive task from the Google Research football environment (Kurach et al. 2019)). In this task, three agents try to score from the edge of the penalty box, one on each side, and the other the center. This task is initialized with the centered agent having the ball and facing the defender. There is an adversary who plays the keeper.
Google Research football
5.3 Training and evaluation
We train all algorithms with 5 random seeds. All the hyperparameters used in the training can be found in the Appendix. For the Multi-agent particle environment, each experiment uses one Tesla K40 GPU to train until convergence, i.c. the best evaluation episode reward hasn't 't changed for 100 epochs. Each epoch equates to 4K episodes of 25 timesteps. We evaluate the algorithms by running 1K test episodes of 25 timesteps and mainly report the mean average test episode reward and standard error across the random seeds. We also evaluate on task -secifpec metrics, including agent-goal pccupancy/agenntaadvessss
distance. For Google Research football, each experiment trains for 5M timesteps. We evaluate on and report the mean average episode rewards and the standard errors across the seeds.
collision count, and ggent-goallaggentaddee
5.4 Baselines
All algorithms are trained using the same agent architecture and optimization algorithm. They vary in task- mask-specfiic extrinsic rewards and intrinsic rewards. We use two versions of the soft actor-critic HHarnoja et al. (018)): a decentralized one that trains each agent individually without access to other agents' observations and actions (ic. the original soft-actor critic algorithm) and a centralized one with a critic that has access to other agents' observations and actions (Iqbal & Sha 2019). Note, our intrinsic reward can also be added to non-actor-critic methods, such as COMA (Foerster et al. 2018) and VDN (Sunehag et al. 2017). We leave this to future work to avoid conflating the effects of expectation alignment with COMA'S counterfactual reasoning and VDN's value decomposition.
algorithm/Haarnjja
We use SPARSE (Lowe et al. 2017; Kurach et al.] 2019), CURTO (Stadie et al., 2015). CURIOggam IIgal & Sha 2020). and variations of our ELIGN rewards. SPARSE rewards agents
7

---

only when they reach a goal state. CURTO is curiosity- multi-agent intrinsic reward which maximizes the average L3 loss (instead of minimizing it in ElIgn). It rewards agents for exploring novel states (Iqbal & Sha 2020). CURIO,elf also maximizes the T2 loss but only using agent i's own observation (Stadie et al., 2015). We experiment with three variants of ELIGN: ELIGN incentivizing alignment to one's s own expectation ELIGN incentivizing agents to align to their team, and ELIGN,dv incentivizing misalignment to adversaries' expectations. Note that EeI is similar to the auxiliary loss in Ndousse et al. (2021) ) but we use it for policy optimization, rather than for training a state encoder. We also add hand-crafted dense rewards to provide oracle performance for all tasks.
5.5 Results in partially observable environments with decentralized training
ELIGN outperforms baselines across cooperative and competitive tasks in the multi-agent particle environment. Table 1 demonstrates that both ELIGN and ELIGN outperform all SPARSE and CURIOselficam baseline rewards in cooperative tasks. While not all ELIGN variants surpass the baselines in competitive tasks, we find that ELIGNeam achieves the highest reward in Physical deception, and ELIGN performs the best in Predator-prey and Keep-away. Similarly, Figure 2 shows that ELIGN achieves the highest per-step occupancy count in both cooperative tasks, and ELIGNadv does the best in all competitive tasks.
ELIGN outperforms baselines in the complex Google Research football environment. As shown in Table I ELIGN achieves the best mean average episode reward in the competitive Academy 3vs/ with keeper task. Collectively, these results provide empirical evidence that the self-organizing principle improves coordination under partial information, a setting that is most realistic to real world multi-agent systems.
In competitive tasks, agents benefit more from being misaligned to adversaries than being aligned to their team members. Among the four competitive tasks in the multi-agent particle and football environments, we find that ELIGN outperforms all SPARSE and cuRhe baselines and other variants of ELIGN in Predator-prey, Keep-away and Academy 3vs/ with keeper. This suggests that being misaligned to adversaries, ie taking surprising actions that conflict with the adversary's expectations, might be a more useful strategy in multi-agent competitive tasks.
When the number of agents increases, FLIGN scales well in all multi-agent particle tasks except for Heterogenous navigation. Table Sishows that our ELIGN intrinsic reward still largely achieves the best performance when more agents are added to cooperative and competitive tasks. The only exception is the Heterogenous naviga-
Scalabiity in the Cooper Navigation Task
1500
Method
SPARRE
ELIGMooa
Efjooda roward
tion task, where both ELIGNsett and ELIGN outperform SPARSE but not CURIO agents predict their neighbors' future states accurately when there are more agents with different sizes and speeds, and errors in dynam- ics prediction could lead to misleading alignment signals. Further, we see a consistent increase in ELIGN 's performance compared against SPARSE even when the number of agents scales to ten in Cooperative navigation (Figure 3
We hypothesize that it is more difficult for
250
10
Numbor of agents
Figure 3: ELIGN achieves consistent gains compared against SPARSE when the num- ber of agents increases in the Cooperative navigation task.
5.6 Results with full observability and centralized ELIGN
We further test the utility of decentralized ELIGN in fully observable environments and centralized ELIGN under partial observability. We find that decentralized expectation alignment helps in fully observable Cooperative navigation, Heterogenous navigation, and Predator-prey tasks where expectation alignment has been observed in nature. Similarly, centralized ELIGN also improves agents' performance compared against SPARSE and CURIO rewards in the same tasks with partial observability. These results can be found in Tables 3 and 4 in the Appendix. As full observability and centralized training are our main focus, we leave it to future work to investigate why expectation alignment benefits these tasks but not others.
5.7 Investigating how the ELIGN reward helps
8

---

Text Occuprecy (11/Colison (Il Court Per Step in Symmetry Brosking Settings
Coop (500
(600)
[M]
Prodesr
Hospaway (2xd)
Oapan Care
cout
SPARO
oran madt
SPARSE
SPHRLE
como. oa
SPARSE
rond
Figure 5: We plot the test occupancy/collisson count per step of decentralized algorithms in symmetry- breaking settings under partial observability. EII performs the best in both cooperative tasks.
Eam and ad are the best strategies in Physical deception and Predator-prey.
0.4
0.2
delta test reward (%)
0.0
-0.
-0.4
-0.6
-0.8
-1.0
2
6
8
10
12
14
16
dynamics model loss
Figure 6: Test performance decreases with dynamics model loss (R² = 0. 0.242), implying that ELIGN requires an accurate dynamics model.
We further investigate how expecta- tion alignment improves coordination through three evaluation setups.
Agent
Advensrry
Lanomark
ELIGN helps agents divide sub-
tasks. A core challenge in multi- agent collaboration is efficient task di- vision [Wang et al., 2020b). Here, we test whether expectation alignment im- proves sub-task allocation. We initial- ize agents in states without optimal sub-task allocation, necessitating symmetry-breaking setups: in cooperative navigation, when agents are initialized equidistant to all the goal locations, there isn't an optimal allocation of agents to goals. We find that ELIGN achieves the best performance in both cooperative tasks, while ELIGN and ELIGN ady the best strategies in Physical deception and Predator-pre respectively (Figure 5)). Upon a qualitative evaluation of Cooperative navigation, we observe that agents with expectation alignment able to predict which goals will be covered by their collaborators and move towards their allocated one. Without expectation alignment, agents move towards the same goal.
Coop navigation (5v0)
Predaton prey (4v4)
Keep-away (4v4)
Figure 4: We visualize the commetry--beeainng setups in three example tasks. More details can be found in the Appendix.
commetry--rrakkinn (Hu et al. 2020). Figure illustrates the
ELIGN helps agents generalize to new partners. Another core challenge in multi-agent collabora- tion is zero-shot coordination, where agents tested to collaborate with new partners they haven't 't been trained with. We study whether expectation alignment enables better zero-shot coordination. New partners are sampled from other training runs with different seeds and the team is evaluated using the same metrics as before. We observe that ELIGN strategies enable better performance than SPARSE on average, and one of ELIGN performs the best in Heterogenous navigation, Physical deception and Keep away. (Figure 77). These results suggest that ELIGN results in better zero-shot coordination with new partners sampled from separate training runs.
Accuracy of the dynamics model affects ELIGN. We investigate the accuracy of the dynamics model in calculating useful intrinsic rewards. Since ELIGN uses dynamics model to calculate

---

Text Occapincy (')/Collionn [J] Count Per Step in Zare-hot Conndiaation under Partil Oburrabilt
Coop 15.0)
Rztew (60)
(Ac)
Prodetereey (4ed)
Kosp-awyy
SPAIOE
SPARSE
SPAILE
SPARSE
Figure 7: We plot the test occupancylcollision count per step of decentralized algorithms in zero-shot evaluation with new partners. We find that one of ELIGN red achieves the best performance in Heterogenous navigation, Physical deception and Keep away.
rewards, we test whether an inaccurate model misleads agents towards unaligned behaviors. We train agents on noisy dynamics models by adding Gaussian noise ~ (0, )) to the predictions made by the dynamics model. We rum experiments with multiple o values to study how performance changes as the dynamics model becomes more noisy: E [0.5. 1.0, 2.0). Our experiments cover one cooperative and one competitive task. Figure plots the final dynamics loss against the reward change from the noiseless run. As the dynamics model degrades, we observe that the task performance also drops. This study identifies the importance an accurate dynamics model, suggesting that expectation alignment should used in environments where an accurate dynamics model can be learned.
O
6 Discussion
Limitations and future work. While curiosity has proven useful for exploration in single-agen tasks, we find that expectation alignment-whien which mathematically encourages agents to be more predictable instead of finding outtefo-s curiosity numerous multi-agent tasks. We hypothesize that our results arise because today's s multi-agent task state space requires significantly less exploration than those used for single-agent (e.g. Atari games). Our findings limited the multi-agent particle and Google Research football environments, which have a smaller action space than most ecologically valid scenarios.
Language, motion, and human gesture are all combinatorially vast; in such action spaces, expectation alignment might develop social dynamics that hinder non-optimal multi-agent behaviors. Similarly, photorealistic environments have a larger state space, where teams perform common household activities (e.g., cooking, cleaning, etc.) drive together in crowded cities (Srivastava et al. ) Future work should develop new multi-agent environments that demand exploration complexity and where both curiosity and expectation alignment would be necessary for collaboration. For example, in a search and rescue task where single agent is unable to carry the injured, curiosity would encourage "search" while ELIGG would speed up "rescue". In the end, envision that both these forms of rewards would be necessary for successful collaboration. However, choosing when to encourage curiosity versus expectation alignment is an open research problem.
Additionally, enabling stable multi-agent training without centralized training could open up future opportunities for legible (Dragan et al., 2013) agents in human environments. Agents with inter- pretable actions can induce more faithful human mental models, improving human-Al interaction; however, predictability does not imply legibility. Future work could explore the role of legibility in designing intrinsic rewards.
Future work should also explore the use of expectation alignment in massive collaboration settings with hundreds of agents. Drawing on Zoology research Couzin (2077) expectation alignment should scale in such settings if agents align their behaviors only to their nearest neighbors and not to the entire cohort.
Inspired by the self- self-organizing principle in Zoology, we introduce ELIGN, i.e, expecta- tion alignment, simple, task- agnostic, and self-suue intrinsic reward for multi-agent systems. ELIGN rewards agents when they act predictably to their teammates and unpredictably to their adversaries. ELIGN improves multi-agent performance across six cooperative and competitive tasks in the multi-agent particle and Google Research football environments, especially for decentralized training under partial observability. It also scales well, helps agents break symmetries, and generalize to new partners.
Conclusion.
10

---

References
Achiam, J. and Sastry, S. Surprise-based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732, 2017.
Ben-Jacob, E., Schochet, O., Tenenbaum, A., Cohen, I., Czirok, A., and Vicsek, T. Generic modelling of cooperative growth patterns in bacterial colonies. Nature, 36666466):46-49, 1994.
Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 2019.
Böhmer, W., Rashid, T., and Whiteson, S. Exploration with unreliable intrinsic reward in multi-agent reinforcement learning. arXiv preprint vrXiiv/906.000 2019.
Buhl, J., Sumpter, D. J., Couzin, I. D., Hale, J. J., Despland, E.. Miller, E. R., and Simpson, J. From disorder to order in marching locusts. Science, 312 1406, 2006.
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Large-scale study of curiosity-driven learning. arXiv preprint /808. 04355, 2018a.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. arXiv preprint arxiv:1810. 12894, 2018b.
Camazine, S., Deneubourg, J.-L., Franks, N. R., Sneyd, J., Theraula, G., and Bonabeau, E. Self- organization in biological systems. Princeton university press, 2020.
Cao, Y., Yu, W., Ren, W. and Chen, G. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial informatics, (1):4277433, 2012.
Chitnis, R., Tulsiani, S., Gupta, S., and Gupta, A. Intrinsic motivation for encouraging synergistic behavior. arXiv preprint arXiv:2002.051 2020.
Collett, M.. Despland, E., Simpson, J., and Krakauer, D. C. Spatial scales of desert locust gregarization. Proceedings of the National Academy of Sciences, 95(22):10522113055 1998.
Corder, K., Vindiola, M. M., and Decker, K. Decentralized multi- actor-critic with generative inference, arXiv preprint 1xXiv:1910.03056 2019.
Couzin, I. Collective minds. Nature, 445(7129):715-7155 2007.
Dragan, A. D., Lee, K. C., and Srinivasa, S. S. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference Human--Rbbl Interaction (HRI), pp. 301-308. IEEE, 2013.
Du, Y., Han, L., Fang, M., Dai, T., Liu. J., and Tao, D. Liir: learning individual intrinsic reward in multi-agent reinforcement learning. Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 4403-4414, 2019.
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with opponent-leaming awareness. arXiv preprint arXiv:/709.04322 2017.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, 2018.
Hernandez-Leal. P., Kartal, B., and Taylor, M. E. Agent modeling as auxiliary task for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 15, pp. 31-37, 2019.
Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. "other-play" for zero-shot coordination. In International Conference Machine Learning, pp. 4399-4410, PMLR, 2020.

---

Huang, C.-M., Cakmak, M., and Mutlu, B. Adaptive coordination strategies for human-rokee robot handovers. In Robotics: science and systems, volume 11. Rome, Italy, 2015.
Iqbal, S. and Sha, F. Actor-attention-critte for multi-agent reinforcement learning. In International Conference Machine Learning, pp. 29222299977 PMLR, 2019.
lqbal. S. and Sha, F. Coordinated exploration via intrinsic rewards for multi-agent reinforcement learning, 2020. URL
Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1.05397. 2016.
Jain, U., Weihs, L., Kolve, E., Farhadi, A., Lazebnik, S., Kembhavi, A., and Schwing, A. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In European Conference on Computer Vision, pp. 471 1-490. Springer, 2020.
Jaques, N., Lazaridou, A., Hughes, E., Gulcehre, C., Ortega, P., Strouse, D., Leibo, J. Z., and De Freitas, N. Social influence as intrinsic motivation for multi-agent deep reinforcement Icarning. In International Conference on Machine Learning, pp. 3040-3049. PMLR, 2019.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, Morel: Model-based offline reinforce-
ment learning. arXiv preprint arXiv:2005.0595/ 2020.
Kim, K., Sano, M., De Freitas, J., Haber, N., and Yamins, D. Active world model learning with progress curiosity. In International conference on machine learning, pp. 5306-5315. PMLR, 2020.
Krause, J., Ruxton, G. D., Ruxton, G., Ruxton, I. G., et al. Living in groups. Oxford University Press, 2002.
Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., and Gelly, S. Google research football: A novel reinforcement learning environment. CoRR, abs/1907. 1180, 2019. URL hattp://arxa /arxivvvvv... org axg/abs/1907 11180
Littman, M. L. Markov games as framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Liu, I.J.,, Yeh, R. A., and Schwing, A. G. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Conference on Robot Learning, pp. 590-602. PMLR, 2020.
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. Multi-agent actor-critic for mixed cooperafive-comppttiie environments. Advances in Neural Information Processing Systems, 2017.
Mordatch, and Abbeel, P. Emergence of grounded compositional language in multi-agent popula- tions. arXiv preprint arXiv: 044008 2017.
Morin, A. Levels of consciousness and self-awareness: A comparison and integration of various neurocognitive views. Consciousness and cognition, 15(2):358-37) 2006.
Ndousse, K., Eck, D., Levine, S., and Jaques, N. Emergent social learning via multi-agent reinforce- ment learning, 2021.
Pathak. D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning, 2018.
Roy, J., Barde, P., Harvey, F., Nowrouzezahrai, D., and Pal, C. Promoting coordination through policy regularization in multi-agent deep reinforcement learning. Advances in Neural Information Processing Systems, 33:15774-15788 2020.
Schafer, I.. Curiosity multi-agent reinforcement learning. PhD thesis, Master's thesis, The
University of Edinburgh, 2019.
12

---

Schmidhuber, J. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-22221111111 1991.
Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models. International Conference on Machine Learning, pp. 8583-8592.
PMLR. 2020.
Srivastava, S., Li, C., Lingelbach, M., Martin-Martin, R., Xia, F., Vainio, K., Lian, Z., Gokmen, C., Buch, S., Liu, C. K., et al. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. arXiv preprint mxiiv21088033322 2021.
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1.X.:00507.00814 2015.
Strouse, D., Kleiman-Weiin M., Tenenbaum, J., Botvinick, M., and Schwab, D. J. Learning to share and hide intentions using information regularization. Advances Neural Information Processing Systems, 11:10249-10259 2018.
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et coloppsccomooiitionn networks for cooperative multi- agent learning. arXiv preprint arXiv: 7706.05266 2017.
Swamy, G., Reddy, S., Levine, S., and Dragan, A. D. Scaled autonomy: Enabling human operators to control robot fleets. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 5942-5948.. IEEE, 2020.
Theraulaz, G. and Bonabeau, E. Coordination in distributed building. Science, 26965224),886-6888, 1995.
Tomasello, M., Carpenter, M., Call, J., Behne, T., and Moll, H. Understanding and sharing intentions:
The origins of cultural cognition. Behavioral and brain sciences, 28(5):675-691, 2005.
Wang. R. E., Kew, C., Lee, D., Lee, T.-W. E., Zhang. T., Ichter, B., Tan, and Faust, A. Model- reinforcement learning for decentralized multiagent rendezvous, 2020a.
Wang, R. E., Wu, S. A., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman-Weinee M. Too many cooks: Bayesian inference for coordinating multi-agent collaboration, 2020b.
Wang, T., Wang, J.. Wu, Y., and Zhang, C. Influence-based multi-agent exploration. arXiv preprint arXiv:1910.05512. 2019.
Wang, X., Xiong, W., Wang, H., and Wang, W. Y. Look before you leap: Bridging model-free and model-based reinforcement learning for planned-ahead dision-and-language navigation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 37-53, 2018.
Ying, W. and Dayong, S. Multi-agent framework for third party logistics in e-commerce. Expert Systems with Applications, 29(2):431-436, 2005.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 5 the results mentioned in the abstract.
(b) Did you deseribe the limitations of your work? [Yes] See Section for a discussion on the limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines.
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material
URL.)? [Yes] See the
Appendix
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Experiments and Appendix sections.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See the S
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See the Experiments 5 and Appendix sections.
4. If you using cxisting assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See the Alignment and Experiments 3
sections.
(b) Did you mention the license of the assets? [Yes] See the Appendix
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] See the Appendix
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We did not collect any data.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We did not collect any data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A] We did not crowdsource any data.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] There were no human subjects experiments in our work.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] No human subjects were hired.
14


Reviews:
[{"review_id": "dZAl1AHA5z7", "reviewer": "Reviewer_TEM3", "summary": "This paper studies a new self-supervised MARL intrinsic reward, called Alignment, to facilitate learning in a decentralized training paradigm under partial observability. Inspired by the self-organization principle in Zoology, the intrinsic reward of Alignment aims to guide agents to match their teammates' expectations in order to emerge cooperative strategies. The experiments are carried out on the multi-agent particle and Google Research football, and the results show that Alignment outperforms sparse and curiosity-based intrinsic rewards.\n", "questions": "The following questions correspond to three points listed in the WEAKNESSES.\n\n1) How do the agents share their expectations with teammates during decentralized training?\n\n2) When the teammates have entirely different expectations for agent $i$, how does agent $i$ match their expectations?\n\n3) Why use tables instead of learning curves to visualize learning performance?", "limitations": "The main concerns are listed in the WEAKNESSES. The authors should justify the soundness of the proposed method in Section 4.2.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reviewer will list the main strengths and weaknesses as follows:\n\nSTRENGTHS\n\n1) This paper introduces an interesting intrinsic reward, called Alignment, in decentralized training under partial observability. This idea is inspired by Zoology and brings a new heuristic to emerge coordination during decentralized training.\n\n2) This paper conducts several experiments on popular MARL benchmarks to show that the Alignment intrinsic reward is a more useful method than curiosity-driven exploration.\n\nWEAKNESSES\n\n1) The major concern is that the method of Alignment (see Section 4) theoretically conflicts with the decentralized training paradigm. In decentralized training, agents cannot share their expectations. Thus, each agent theoretically cannot match their teammates' expectations because they do not know them. For example, the dynamics model of agent $i$ in line 150 is trained on the $o_i$ and $a_i$ but the alignment intrinsic reward $r_{in}$ defined in line 155 apply the dynamics model of agent $j$ on the $o_i$ and $a_i$. Note that the dynamics model of agent $j$ is trained on the set of $o_j, a_j$ rather than $o_i,a_i$. $r_{in}$ defined in line 155 may induce huge penalty due to this distributional shift from $o_j,a_j$ to $o_i,a_i$.\n\n2) When the agent aims to match their teammates' expectations, what if their teammates do not reach a consensus? For example, in the definition of $r_{in}$ on line 155, if agent $i$'s neighbors have completely different expectations, which expectation does agent $i$ match?\n\n3) Sample efficiency is an important measure for MARL evaluation. Could you provide the learning curve during training to better visualize the sample efficiency in comparison? Tables may not be a natural way to demonstrate the advantage of exploration methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T2CTb47Yrk3", "reviewer": "Reviewer_D8qp", "summary": "This paper tackles the problem of coordination in multi agent reinforcement learning, particularly in the decentralized training / partial observability setup. The paper proposes to derive a self-supervised reward from alignment. Agents are encouraged to take actions that match other agents' expectations when they cooperate and they are encouraged to do the opposite with competing agents.\n\nFor each agent the reward is computed using a dynamics model. This model is trained by predicting the next observation given the current action and observation. This model is then used to derive a reward that estimates the neighbors expectations of the agent's next state. The reward is decentralized and does not require the centralized training / decentralized execution which is common in the multi agent reinforcement learning literature.\n\nThe proposed reward is evaluated across many problems using the multi agent particules environment and Google Research football. The authors compare their algorithm with several existing baselines and the alignment based reward is shown to perform better than existing techniques for cooperative and competitive problems. \n", "questions": "L282 \"When the number of agents increases, alignment scales well in all multi-agent particle tasks except for heterogenous navigation.\"\nDo you expect this statement to remain true when the number of agents increases to hundreds or even thousands? In that case I would imagine that the summation L159 would make the alignment reward pretty noisy and coordination between agents would be more difficult.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I found this paper to well be written and easy to follow. The authors made an effort to provide many helpful illustrative examples of the behaviour of the proposed technique. The method is simple and appears easy to implement.\n\nThe empirical evaluation of the paper is solid, it includes cooperative and competitive problems, with decentralized and centralized training and the proposed method is compared with several existing baselines. I found section 5.7 to be useful to understand how alignment reward helps and what are the limitations, particularly that the overall performance of the system decreases when the accuracy of the dynamics model decreases or that the method is less is competitive with heterogenous agents are used.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Qkwi_AwwObV", "reviewer": "Reviewer_whMR", "summary": "The authors proposed an intrinsic reward that learns each agent to align with the predictions of neighbors in a fully decentralized way for improving multi-agent coordination. They compared the proposed method with the intrinsic reward-based baselines, e.g. curiosity-based approach in several environments including Google Research football, and provides some investigations to show the performance improvement of the proposed method.", "questions": "1. The proposed intrinsic reward is based on how much the neighbors predict the agent well. For the fully decentralized training, the authors replace the neighbors’ dynamic models with their own dynamic models. Eventually, the proposed intrinsic reward is the opposite of the novelty-based intrinsic reward approach, and thus the multiple agents will be trained to visit the familiar states that each agent can predict well. Although the approximation of the neighbors’ dynamic models with a proxy is empirically good, it is not exactly the same as the neighbors’ models. Each neighbor may predict the agent differently. Can we say that the proposed method enhances the alignment? the performance improvement can come from other reasons.\n\n2. Since the proposed intrinsic reward is the opposite of the novelty-based intrinsic reward, can the proposed intrinsic reward degrade the exploration?  \n\n3. In general, the curiosity-based approaches are not good at the dense-reward environment. For a fair comparison, the authors should provide experiments on both dense/sparse environments. \n4. Furthermore, it would be great if the authors provide the performances as the learning processes.\n5. The authors should provide the experiments on the SMAC environment. Also, the intrinsic reward-based baselines such as [1] are missing. The authors should compare the proposed method with [1]. \n6. In Section 4.2, the authors restricted the future prediction from j’s point of view by using the portion of j’s observation i can see. The reviewer thinks that for the fully decentralized training, A should be unaware of how much B can observe. Is it okay that Agent i knows o_{i \\and j} in the fully decentralized training?\n\n[1] Zheng, Lulu, et al. \"Episodic multi-agent reinforcement learning with curiosity-driven exploration.\" Advances in Neural Information Processing Systems 34 (2021): 3757-3769. ", "limitations": "As mentioned in the “Questions”, the author should provide the experiments on the SMAC environment and compare the proposed method with the state-of-the-art baseline of the intrinsic reward-based MARL algorithm [1].", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n-\tThis paper is well written and easy to follow. \n-\tThis paper deals with enhancing predictability of other agents in a fully decentralized way, which is a significant problem in MARL.\n-\tThis paper provides some experiments to show why the proposed method works.\n\nWeaknesses\n\n-\tThe novelty of this paper is not enough for accepting the venue of NeurIPS.\n-\tThere are some concerns of the proposed method (Please see “Questions”)\n-\tThe lack of experiments on the popular benchmarks such as SMAC and the sparse environments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M95nZAxAM4M", "reviewer": "Reviewer_DoA9", "summary": "The authors propose an auxiliary training reward called ALIGNment. Agents are rewarded when either they (ALIGNself) or their teammates (ALIGNteam) can predict their actions, or when their adversaries (ALIGNadv) cannot predict their actions. The authors use experimental evidence to support the intuition that alignment helps agents coordinate which subtasks to complete and helps agents coordinate zero-shot with new partners. Compared with 3 prior approaches in 6 Google Research football environments and 5 multi-agent particle environments, at least one ALIGN variant achieves state-of-the-art results in every task except for the heterogeneous navigation task.", "questions": "Did you train the Sparse, Curio-self, and Curio-team methods yourself? Or did you take numbers from prior work? It seems like you trained them yourself, but it would be helpful to make this more clear.  (I think it’s stronger if you can report numbers directly from prior work where applicable, because the authors of prior work have an incentive to tune their algorithms as well as possible.)\n\nIf you trained prior methods yourself: Did you spend equal effort tuning the prior approaches as you did tuning your own method, and can you quantify this in the paper (eg by having tested the same number of hyperparameter settings)?", "limitations": "The authors note that their method, ALIGN, performs worse than prior methods in Heterogenous navigation. They hypothesize that this is because the differing size and speed of agents in Heterogenous navigation makes it difficult for agents to predict each other. I appreciate that the authors analyze this limitation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nOverall, I found the method easy to understand. Algorithm 1 makes the method clear.\n\nThe experimental evaluation is thorough, comparing against 3 prior approaches and 11 total tasks. At least one ALIGN variant performs best across all but one task.\n\nThe authors include additional experiments to help study why ALIGNment improves performance.\n\nAs this is an intuitive approach that is explained clearly and supported by thorough experimental evaluation, I recommend acceptance.\n\n## Weaknesses\nThe word “alignment” already has a meaning in artificial intelligence. Specifically, it is used to mean how well an artificial intelligence is aligned to human preferences. See, for example, “The Alignment Problem” by Brian Christian. I would recommend the authors use a different word for their method, such as “Predictability.” This renaming would respect the existing use of the word “alignment” as well as be more specific about what the auxiliary loss is rewarding.\n\nI didn’t find Figure 1 very helpful for understanding the method. The idea behind alignment – a predictive loss – is very simple, yet Figure 1 is quite complex. I’d recommend trying to redesign Figure 1 to make it as simple as possible.\n\nThe overall results are presented in two dense tables, Table 1 and Table 2. It would be helpful if the authors could present aggregated results in a single figure. If possible, I would recommend normalizing results using the performance of an expert policy and a random policy as is done in the D4RL benchmark [1], i.e., computing (performance - random) / (expert - random). This gives a common scale over which the authors can aggregate.\n\nOnly the maximum performance is bolded in the results tables. I recommend bolding all results within 5% or 10% of maximum performance, to account for noise, as is done in other work such as [2].\n\n[1] D4RL: Datasets for Deep Data-Driven Reinforcement Learning\n\n[2] Offline Reinforcement Learning with Implicit Q-Learning", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4GtxXGplEJ", "reviewer": "Reviewer_oB8u", "summary": "- This paper proposes to tackle the problem of multi-agent learning under decentralised training and partial observability through using a novel self-supervised intrinsic reward that encourages agents to behave predictably to their neighbouring teammates and unpredictably to their adversaries.\n- A dynamics model (trained on the agent’s own observations as a proxy given training is decentralised), is used to predict future states from other agents’ viewpoints under partial observability. The negative error in this prediction is then used as an intrinsic reward to supplement training in environments with sparse extrinsic rewards.\n- The effectiveness of this approach is compared to curiosity driven intrinsic rewards on 6 cooperative and competitive multi-agent tasks. Further experiments are conducted in regimes of full observability, centralised training, and zero shot coordination.", "questions": "1. Did you compare this intrinsic reward based approach to other (non intrinsic reward based) approaches to the sparse learning problem (e.g. Ndousse et al 2021)?\n    - Some discussion around why the framing as an auxiliary reward instead of an auxiliary loss would be helpful.\n2. Under what conditions do we expect alignment/misalignment to work well? A discussion around why we see that ALIGN self, team and adv have varying quality on different tasks would be helpful. \n    - For example, why does modelling from the other agent’s point of view, instead of directly predicting your own future state (which incorporates the other agents action) more helpful? When and why is ALIGN_team better than ALIGN_self? It seems that the team set up is just a partially observed view of the self set up, because the agent’s own dynamics model is used.\n    - In the adversarial case, it seems that the agent will learn to be unpredictable to itself (as it uses its own dynamics model as a proxy) so it is unclear to me why this would help. Conversely, ALIGN_team and ALIGN_self seem to work better than ALIGN_adv on some competitive tasks indicating that it is sometimes better to be predictable to an adversary which seems counterintuitive.\n    - In the first paragraph of Section 4 there is discussion about gaining information outside of the agent’s receptive field through encountering surprising actions from other agents. How does the alignment reward help convey this information? Does this help at execution time?\n3. In the centralised regime experiments, did you use the actual dynamics model learnt by each agent to calculate the intrinsic reward or did you still use the individual agents dynamics model as a proxy as in the decentralised case?\n4. Do you have any suggestions for how to improve performance using alignment for heterogenous agents?\n5. Did you evaluate zero shot coordination with CURIO models as well?", "limitations": "- The authors state limitations of their approach, namely:\n    - the small action/state spaces of the environments considered.\n    - difficulty generalising when there are heterogenous agents due to agents learning only one dynamics model (of themselves). They argue that this enables more scalability, but demonstrate that this leads to decreased performance.\n    - The method is reliant on the ability to learn a good dynamics model which may be more difficult in noisy, real-world environments.\n- I have addressed the other limitations in the strengths and weaknesses section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## \n\n- *Originality:*\n    - The authors frame their work well in the context of the literature on intrinsic motivation in reinforcement learning, and motivate their approach from the natural world. I appreciate that they explicitly state the differences of their approach to closely related work, e.g. Ndousse et al 2021.\n    - Some references need to be corrected:\n        - Line 45: “*Only a few attempts explore other forms of multi-agent intrinsic rewards”.*   A more specific, referenced statement would be better here.\n        - Line 33: To my knowledge, there is no reference to decentralised training and partial observability in Lowe 2017. MADDPG is a centralised training, decentralised execution algorithm. The authors should reference more appropriate work for that statement on task-specific reward shaping to motivate their work.\n- *Quality*:\n    - Two standard coordination and competition benchmark environments are used to evaluate their framework. The same architectures and optimisation algorithms are chosen to provide a fair comparison against other intrinsic rewards. I particularly liked the symmetry breaking experiments to demonstrate learned coordination.\n    - Table 1 shows promising results in small scale decentralised, partially observable settings. However results are not consistently *significantly* better than the baselines across tasks and settings. There are claims of superior performance in some tasks in the text but the the baselines have similar performance to the proposed method within standard error. For example, Figure 3 is selective to show results against SPARSE while CURIO performs similarly to ALIGN within error. Table 1 3v1 w/ keeper and Table 2 keep-away - CURIO baseline has similar performance within error. Symmetry-breaking performance is also not significant when scaled (appendix). In general the authors should be careful not to overstate claims unless there is statistical significance to back it up.\n    - To improve the clarity of results I would suggest incorporating some learning dynamics curves. Only converged test rewards and task-specific converged metrics are currently given. Does coordination help with sample efficiency of training?\n    - Comparison to Ndousse et al 2021 would also be useful to see whether this reward framing has a benefit over their proposed approach.\n    - On line 161 there is a claim that the proxy dynamics model is validated in heterogeneous tasks but in the experiments it is shown that this is not the case in the experiments.\n- *Clarity*:\n    - The paper is well written in many parts. Paragraph headings and subsections are used effectively and the paper on the whole reads quite intuitively. Figure 1 is clear and informative to the reader.\n    - To improve clarity:\n        - the authors mention a gaussian noise experiment in the main text and refers to appendix but I cannot find this section in the results.\n        - Figure 1: I would suggest increasing the font width for readability (particularly the equation)\n        - The tables and figures are quite far away from their relevant sections in the text which hinders readability.\n        - Line 293-294 : “analyse this in future experiments” - it is unclear if this means later in the paper or in future work.\n    - A careful proofread is also required. Some issues I spotted:\n        - Line 256: “a” not “an” goal state\n        - Table 2 caption: misspelled “trained”\n        - Iqbal & Sha 2019a and 2019b seem to refer to the same paper.\n        - Line 160 misspelled “empirically”\n        - Line 178: misspelled “team”\n        - State and action space paragraph starting on Line 194 has a number of grammatical errors.\n- *Significance:*\n    - The paper addresses the problem of decentralised training which is less explored in the literature compared to centralised training but important for the scalability of MARL.\n    - The alignment idea to use predictability of actions as an intrinsic reward is novel to my knowledge and well motivated. However, as the authors note in the related work section, there is a body of similar work that utilise model-based predictability in different ways e.g. as an auxiliary loss. It is not clear to me yet why using this inductive bias as a reward instead is significant, particularly without an experimental comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept