Online Ad Hoc Teamwork under Partial Observability
Authors: Pengjie Gu
Mengchen Zhao
Jianye Hao
Bo An
Venue: ICLR 2022 Poster (2022)
Keywords: coordination
reinforcement learning

Abstract: Autonomous agents often need to work together as a team to accomplish complex cooperative tasks. Due to privacy and other realistic constraints, agents might need to collaborate with previously unknown teammates on the fly. This problem is known as ad hoc teamwork, which remains a core research challenge. Prior works usually rely heavily on strong assumptions like full observability, fixed and predefined teammates' types. This paper relaxes these assumptions with a novel reinforcement learning framework called ODITS, which allows the autonomous agent to adapt to arbitrary teammates in an online fashion. Instead of limiting teammates into a finite set of predefined types, ODITS automatically learns latent variables of teammates' behaviors to infer how to cooperate with new teammates effectively. To overcome partial observability, we introduce an information-based regularizer to derive proxy representations of the learned variables from local observations. Extensive experimental results show that ODITS significantly outperforms various baselines in widely used ad hoc teamwork tasks.

Published as a conference paper at ICLR 2022
ONLINE AD HOC TEAMWORK UNDER PARTIAL OBSERVABILITY
Pengjie Gu', Mengchen Zhao 2,, Jianye Hao Bo An
School of Computer Science and Engineering, Nanyang Technological University, Singapore' Noah's Ark Lab, Huawei²
College of Intelligence and Computing, Tianjin University (peng) eeeggiie.guu gu, boan) @ntu. edu. sg, (zhaomengchen, haojianye] Baaaaayyeeeaaaaa @huawei.
ABSTRACT
Autonomous agents often need to work together as team accomplish complex cooperative tasks. Due to privacy and other realistic constraints, agents might need to collaborate with previously unknown teammates the fly. This problem is known ad hoc teamwork, which remains a core research challenge. Prior works usually rely heavily on strong assumptions like full observability, fixed and predefined teammates' types. This paper relaxes these assumptions with a novel reinforcement learning framework called ODITS, which allows the autonomous agent adapt to arbitrary teammates in an online fashion. Instead of limiting teammates into finite set of predefined types, ODITS automatically learns latent variables of teammates' behaviors infer how to cooperate with new teammates effectively. To overcome partial observability, we introduce an information-bassee regularizer to derive proxy representations of the learned variables from local observations. Extensive experimental results show that ODITS significantly outperforms various baselines in widely used ad hoc teamwork tasks.
1 INTRODUCTION
Recently, autonomous agents including robotics and software agents are being widely deployed in different environments. In many tasks, they are progressively required to cooperate with other unknown teammates on the fly. For example, in case of search and rescue tasks in a disaster, due to privacy or lack of time, deployed robots need to interact with robots from other companies or laboratories, whose coordination protocols might not be explicitly provided in advance (Barrett and Stone 2013). Besides, in the domain of game Al (Yannakakis) 2012), virtual agents are required assist different agents controlled by human players. To effectively complete these tasks, autonomous agents must show high adaptation ability to collaborate with intrinsically diverse and unknown teammates. This problem known in the literature as ad hoc teamwork (Stone et al. 2010p.
Existing approaches on ad hoc teamwork usually assume that all teammates' behaviors are categorized into several predefined and fixed types, which corresponds to different coordination strategies (Barrett and Stone 2015 Durugkar et al., (2200] Mirsky et al. 2020). Then, by reasoning over the type of interacting tcammates, the agent switches its behavior to the corresponding policy. If the types are correctly recognized and the strategies are effective, the agent would accomplish the given cooperation task well. However, defining sufficiently descriptive types of teammates requires prior domain knowledge, especially in uncertain and complex environments. For example, in human-AI collaboration in Hanabi (Bard et al. 2020), there are often wide variety of cooperative behaviors showed by human players. It is challenging for predefined types to cover all possible human players' behaviors. Further, teammates' strategies might be rapidly evolving throughout the entire teamwork. If the agent assumes that teammates' behavioral types are static and cannot adapt to current teammates' behaviors in an online fashion, teamwork would suffer from serious miscoordination (Ravula et al. 2019 Chen et al.) 2020). Rescue and search tasks are an essential class of such examples (Ravula et al. 2019). On the other hand, existing techniques (Barrett and Stone, 2015 Albrecht and Stone 2017 Chen al. 2020 Ravula et al. 2019) try utilize Bayesian posteriors over teammate types to obtain optimal responses. To effectively compute posteriors, they usually assume that the agent could always know other teammates' observations and actions. However, this
"orrespondingg author

---

Published as a conference paper at ICLR 2022
assumption is unrealistic in partial observable environments, in which each agent is not aware of other agents' observations.
To address the issues mentioned above, this paper introduces an adaptive reinforcement learning framework called Online aDaptation via Inferred Teamwork Situations (ODITS). Our key insight is that teamwork performance is jointly affected by the autonomous agent and other teammates' behaviors. So, the agent's 's optimal behavior depends on the current feamwork situation, which indicates the influence on the environmental dynamics caused by other teammates. If the agent identifies the current teamwork situation in an online fashion, can choose actions accordingly to ensure effective coordination. In this way, we introduce a multimodal representation learning framework (Suzuki et al., 2016 Yin al. 2017). It automatically encodes the core knowledge about the teamwork situations into a latent probabilistic variable. We show that without any prior knowledge, after learning from the interactive experience with given teammates, the latent variable is sufficiently descriptive to provide information about how to coordinate with new teammates' behaviors. To overcome partial observability. we propose an information-bns Then, the autonomous agent adapts to new teammates' behaviors dynamically and quickly by conditioning its policy on the inferred variables.
proxy encoder to implicitly infer the learned variables from local observations.
Instead of limiting teammates into several predefined and fixed types, ODITS considers mechanism of how an agent should adapt to teammates' behaviors online. It automatically learns continuous representations of teammates' behaviors to infer how to coordinate with current teammates' actions effectively. Without domain knowledge on current environments, it enables effective ad hoc teamwork performance and fast adaptation to varying teammates, which the agent might not thoroughly observe under partial observability. In our experimental evaluation, by interacting with a small set of given teammates, the trained agents could robustly collaborate with diverse new teammates. Compared with various type-based baselines, ODITS reveals superior ad hoc teamwork performance. Moreover, our ablations show both the necessity of learning latent variables of teamwork situations and inferring the proxy representations of learned variables.
2 RELATED WORKS
Ad Hoc Teamwork. The core challenge of achieving cooperative ad hoc teamwork is to develop an adaptive policy robust to various unknown teammates' behaviors (Stone et al., 2010). Existing type-based approaches try to predefine types of teammates and choose policies accordingly to cooperate with unknown teammates (Chen al. 2020 Ravula et al.) 2019: 201 Durugkar et al. 2020 Mirsky et al. 2020 [Brrettt and Stone 2015). Specifically,
PLASTIC (Barrett and Stone, 2015) infers types of teammates by computing Bayesian posteriors over all types. ConvCPD (Ravula et 2019) extends this work by introducing a mechanism to detect the change point of the current teammate's type. AATEAM (Chen et al., 2020) proposes attention based architecture to infer types in real time by extracting the temporal correlations from the state history. The drawback of these approaches is that finite and fixed types might not cover all possible situations in complex environments. One recent work avoids predefining teammates' types by leveraging graph neural networks (GNNs) to estimate the joint action value of an ad hoc team (Rahman et al. 202I). However, this work requires all teammates' observations as input, which might not always be available in the real world.
Agent Modeling. By modeling teammates' behaviors, approaches of agent modeling aims to provide auxiliary information, such as teammates' goals or future actions, for decision-making (He et al. 22016) Albrecht and Stone 2018). For example, McLIBA conditions the ad hoc agent's policy on a beliei over teammates, which is updated following the Bayesian rule (Zintgraf et al. (02T). However, existing agent models require the full observations of teammates as input (Raileanu et al.. Groverr et al.] Tacchetti et al.) 2019). If the agent cannot always observe other teammates' information (e.g. observations and actions), those approaches would fail to give accurate prediction about teammates" information. A recent work proposes to use VAE to learn fixed-policy opponents under partial observability. However, it does not generalize to the ad hoc setting where the teammates can be non- non-stationary (Papoudakís and Albrecht 2020).. There are also some works study how to generate diverge agent policies, which benefits the training of ad hoc agent (Canaan et al., 2019).
Multi-agent Reinforcement Learning (MARL). Cooperative MARL Foerster et al., 2017) with centralized training and decentralized execution (Oliehoek al. 2008) (CTDE) is relevant to this work. Related approaches (Sunehag et al.] 20188] Rashid et al.) 2018) utilize value function factorization to overcome the limitations of both joint and independent learning paradigms simuitaneously. However, these algorithms assume that the developed team is fixed and closed. The team configuration (e.g., team size, team formation, and goals) is
(Forreorr
2

---

Published as conference paper at ICLR 2022
unchanged, and agents will not meet other agents without pre-coordination. Several extended works improve the generalization ability for complex team configurations by leveraging other insights. like learning dynamic roles (Wang et al. 2021 2020), raadomiiedd entity- -wise factorization (Iqbal et al. 2020), and training regime based on game-theoretic principle (Muller et al. 2020). However, intrinsically, these approaches usually focus co-training group of highly-coupled agents instead of autonomous agent that adapt non-stationary teammates.
3 BACKGROUND
Problem Formalization. Our aim is to develop a single autonomous agent, which we refer to as the ad hoc agent, that can effectively cooperate with various teammates under partial observability without pre-coordination, such as joint-training. While we focus on training a single agent in this work, similar approaches can be applied construct ad hoc team.
Sample
Adhas
agent
Slured
Environment
Teammate
Figure 1: Visualization of the Dec-POMDP with an addtional teammate set r.
To evaluate the ad hoc agent's ability cooperate with unknown teammates, we formally define the problem a decentralized Partially observable Markov Decision Process (Dec-POMDP) (Oliehoek et al. 2008) with an additional assumption about the set of teammates' possible policies I..
where N denotes the number of agents required by the task, S denotes the global state of the environment. The joint action E A is formed by all agent's independent actions a² E A, where i is the index of the agent. In the environment, each agent only has access its partial observation O² O drawn according to the observation function (ss i), and it has an observation-action action history 7' E T = (O x A)". P(s', |s, a) denotes the probability that taking joint action in state results in a transition to state 8'. R(s, a) is reward function that maps state and a joint action to a team reward r E R. I represents a pool of various policies, which can be pretrained or predefined to exhibit cooperative behaviors. Without loss of generality, we denote by n; the policy of the ad hoc agent and by H the joint policy of all other agents.
It can be represented as a tuple A,T,
Fig shows the detailed schematics of this problem. Note that the orginal group size of teammate can be arbitrary. In Dec-POMDP with additional teammate set I, the objective of the ad hoc agent is maximize the expected team return when teams up with - arbitrary teammates sampled from I, though it has no prior knowledge about those teammates. Therefore, the ad hoc agent's optimal policy गे is required maximize the joint action value Q (8, a², a a=),, which indicates the expected accumulative team reward over different ad hoc teamwork:
4.00
(n(ssp = Bo oa M
Σ
n'r s, as = a,
(I)
(",aa' (", a), ViTA 8, ai; a
Marginal Utility is defined to measure the contribution of an ad hoc agent to the whole team utility (Genter and Stone (011). It represents the increase (or decrease) team's 's utility when ad hoc agent is added the team. Given teammates' actions a there is relationship between the marginal utility and the team utility
3

---

Published as a conference paper at ICLR 2022
(denoted by the joint action value) as follow:
arg max u' (s, a², a²) arg max a',
where us,s, a', a a) denotes the marginal utility when the ad hoc agent chooses the action a' under the state s. Note that the marginal utility is not necessarily equal the Q-value (Sunehag et al. 2018). The ad hoc agent chooses the action which maximizes the marginal utilityy to ensure the maximal team utility.
4 ODITS LEARNING FRAMEWORK
Our approach addresses the ad hoc teamwork problem with novel probabilistic framework ODITS. In this section, we first introduce the overall architecture of this framework and then present a detailed description all modules in this framework.
Only available the training phase
Joint Action Value
0g
Integrating
4.1 OVERVIEW
Teamwork Situation Decoder g
Teamwork Situation
ODITS aims to estimate the ad hoc agent's 's marginal utility for choosing corresponding actions to maxi- mize the team utility. To achieve the adaptive policy to unknown teammates, we model the marginal utility conditional function on the inferred latent vari- able, which implicitly represents the current feamwork situation. ODITS jointly optimizes the marginal util- ity function and the latent variable by two learning objectives in an end-to-end fashion. Fig shows the detailed schematics of ODITS. It splits the team into two parts: teammates and the ad hoc agent.
Encoder f
(5, a
MI
(r),
Marginal Utility
Network M
Ad horagent
Other teammates
Proxy Decoder g
Proxy Encoder f
Figure 2: Schematics of ODITS.
First, regard other tcammates a part of environmental dynamics perceived by the ad hoc agent. Since different combinations of tcammates lead to diverse and complex dynamics, we expect to learn a latent variable to describe the information of teammates' behaviors implicitly. To do this, introduce teamwork situation encoder f to learn the variable. Then, a loss function (Q loss), an integrating network G and a teamwork situation decoder g are jointly proposed to regularize the information embedded in the learned variable ct-
For the ad hoc agent, we expect to condition its policy on the learned variable of- However, the partial observability impedes the direct access ct. Thus, introduce proxy encoder f" to infer a proxy representation 2 of ef from local observations. We force 2 to be informationally consistent with ct by an information-batt function (MI loss). Then, we train a marginal utility network M to estimate the ad hoc agent's 'ss conditional marginal utility á' (r), 2f) u' u',,, ") For conditional behavior, part of parameters of M are generated by the proxy decoder g"
loss
Similar the CTDE scenario (Oliehoek al. 2008), relax the partial observability in the training phase. ODITS is granted access to the global state s and other teammates' actions a, during training. During execution, G, f. is removed; the ad hoc agent chooses the action which maximizes the conditional marginal utility function ü' (7f, at;2)): 2/).
4.2 LEARNING TO REPRESENT TEAMWORK SITUATIONS
For adaptive behaviors, we expect to condition the ad hoc agent's policy on other teammates. However, unknown teammates show complex behaviors. Directly conditioning the policy on them might lead volatile policy. To address this issue, we aim to embed the teammates' information into a compact but descriptive representation. To model the concept clearly, we formally define teamwork situation:
Definition 1 (teamwork situation) At each time step t, the ad hoc agent is in the teamwork situation of E C, which the current underlying teamwork state yielded by the environment state st and other teammates' actions a It reflects the high-level semantics about the leammates' behaviors.
4

---

Published as a conference paper at ICLR 2022
Though different teammates generate diverse state-action trajectories, assume that they can cause similar teamwork situations at some time, and the ad hoc agent's 's action would affect their transitions. When the current teamwork situation is identified, the ad hoc agent can choose the action accordingly to ensure online adaptation.
Teamwork Situation Encoder f. To model the uncertainty of unknown teammates, we encode teamwork situations in stochastic embedding space C. Thus, any teamwork situation probabilistic variable c¹ that is drawn from a multivariate Gaussian distribution N ((pe- To enable the dependency revealed in the definition, we use trainable neural network f to learn the parameters of the Gaussian distribution of c²:
be represented
latent
(Met, ;0) ~ ~N(Hat
(4)
where f are parameters of f.
Regularizing Information Embedded in c'. We introduce set of modules to jointly force ct to be sufficiently descriptive for reflecting the current teamwork situation. If ct is able to capture the core knowledge about other teammates' current behaviors, we can predict the joint action value (st, aj a, according to ei and the ad
hoc agent's marginal utility un [1] Thus, we propose an integrating network G for generating the joint action value's 's estimation G(u,, ei) z (T ai a, "). We adopt a modified asynchronous Q-learning's loss function (Q-loss) (Mnih et al. 2016) as the optimization objective:
L (ug [ire + y max ((
(5)
o++1
where G is a periodically updated target network. The expectation estimated with uniform samples from the replay buffer D, which saves the interactive experience with training teammates.
Integrating Network G. One simple approach for integrating c' with w' is formulate G an MLP that maps their concatenation into the joint value estimation. We instead propose to map c' into the parameters of G by a hypernetwork (Ha al. 2016). which we refer to as the teamwork situation decoder g. Then, maps the ad hoc agent's 's utility ut into the value estimation. This alternative design changes the procedure of information integration. The decoder provides multiplicative integration to aggregate information. By contrast, the concatenation-bat based operation only provides additive integration, leading to poor information integration ability (see Supplementary). We also empirically show that multiplicative integration stabilizes the training procedure and improves teamwork performance. In addition, BG
expect that there is monotonicity modeling the relationship between G and the marginal utility uf "bu; 2 0. Given any ci, the increase of the ad hoc agent's s marginal utility results in the improved joint action value. To achieve this property, foree 0g 2 0.
4.3 LEARNING CONDITIONAL MARGINAL UTILITY FUNCTION UNDER PARTIAL OBSERVABILITY
Apparently, the marginal utility of the ad hoc agent depends on other teammates' behaviors. Distinct behaviors result in different marginal utilities. Here, formalize the marginal utility network M deep recurrent Q network (DRQN) (Hausknecht and Stone 2015) parameterized by 0 To enable adaptation, we force the final layers' parameters of M condition on the learned variable ci.
Proxy Encoder ft. Because of partial observability, the teamwork situation encoder f is not available during execution. Thus, we introduce a proxy encoder f" to estimate cj from the local transition data bj (oi., Tt-1 r i) We assume that bú can partly reflect the current teamwork situation since the transition implicitly indicates the underlying dynamics, which is primarily influenced by other teammates' behav- iors. We denote the estimation of of as zf. Then, z would be fed into prexy decoder g (z); 0g-) parameterized by 0g to generate the parameters Our of M, enabling the marginal utility function to condition on the proxy representation zτ. Similar to c², we encode z into a stochastic embedding space:
(p)), (b:;0))) ~ ~N
(6)
where oy- are parameters of f*.
lut is a shorthand notation of u' (r/, 2f)
5

---

Published as conference paper at ICLR 2022
Regularizing Information Embedded in z'. To make z identifiable, expect of to be informatively consis- tent with c/. Thus, we introduce an information based loss function Lasz here to maximize the conditional mutual information [(2;; c)(b)) between the proxy variables and the true variables. However, estimating and maximizing mutual information is often infeasible. We introduce a variational distribution de (zilc), bt) parameterized by & to derive tractable lower bound for the mutual information (Alemi et al. 2017):
2: (= b)) log p(2 b5)
T(a c2)))) 2 Beg.ct.
where p(zi (b) the Gaussian distribution N (2 This lower bound can be rewritten as loss function be minimized:
LM (0 == B "'o,,""a"")>> D (p())()))) (#/lct, b)]]
(8)
where D is the replay buffer, DEL is the KL. divergence operator. The detailed derivation can be found Supplementary.
4.4 OVERALL OPTIMIZATION OBJECTIVE
Algorithm ODITS Training
Require: Batch of training teammates' behavioral policies { -ftr ",. learning rate ar scaling factor A.
I: initialize the replay buffer DD
2: while not done do
3: Sample the teammates' policies , from {,, 1 1
4: for k 1, K do
5: Sample data Di {(E,, at ar
T using the ad hoc agent's policy nO and
6: Add D2 into D
7: for steps in training steps do
Sample one trajectory D) for 1,... ,T TT do
9:
10:
Compute (Maz Rxt) f(s,, a and sample of ~ (
11:
Compute (12)) az2) (b;) and sample 2 ~ N(ugf
12:
Compute u,(T/, ai: 2f) and G(u):ct)
13:
Compute Lo, LMT
14:
0 <- + (LQ2)
15:
0 t 0f- - a (CMI)
16:
& + E + X - (
Algorithm 2 ODITS Testing
Require: Testing teammates' behavioral policies
I: for t 1, 1,... IT' do
2: Generate teaammtes' actions a, ~ π
3: Compute (Mag. oz;) f (b() and sample 2f ~ (1)
4: Do the action aj that maximizes wt (r}. a}; zf)
To the end, the overall objective becomes:
(()) LQ( (0mr((0r
(
where 0 = (0,, 0g 0M. 0y-, Op pp.$8 E), A is the scaling facor.
During the training phase, the ad hoc agent interacts with different training teammates for collecting transition data into the replay buffer D. Then, samples from D are fed into the framework for updating all parameters
6

---

Published as a conference paper at ICLR 2022
4cb
bwrb
2.1
Samn Bac
3.0
2.0
Figure 3: Performance comparison across various scenarios for Predator Prey (top panel) and Save the City (bottom panel).
by gradients induced by the overall loss. During execution, the ad hoc agent conditions its behavior on the inferred teamwork situations by choosing actions to maximize the conditional utility function u' (T/ a/s²f)) af). We summarize our training procedure and testing procedure in Algorithm and Algorithm 2
5 EXPERIMENTS
We now empirically evaluate ODITS on various new and existing domains. All experiments in this paper are carried out 4 different random seeds, and results shown with a 95% confidence interval over the standard deviation. In the following description, we refer to the teammates that interact with the ad hoc agent during the training phase as the training teammates, and refer to those teammates with unknown policies as testing teammates. And the "teammate types" correspond to the policy types of teammates. All experimental results illustrate the average teamwork performance when the ad hoc agent cooperates with different testing teammates. Additional experiments, further experiment details and implementation details of all models can be found at Supplementary.
5.1 PREDATOR PREY
Configurations. In this environment, m homogenous predators try capture randomly moviig preys in 7 x 7 grid world. Each predator has six actions: the moving actions in four directions, the capturing action, and waiting cell. Due to partial observability, each predator only access the environmental information within two cells nearby. Besides, there are two obstacles at random locations. Episodes are 40 steps long. The predators get team reward of 500 if two or more predators are capturing the same prey at the same time, and they are penalized for -10 if only one of them tries to capture a prey. Here, we adopt three different settings to verify the ad hoc agent's 's ability to cooperate with different number of teammates. They are 2 predators and 1 preys (2dly), 4 predators and 2 preys (4d2y) and 8 predators and 4 preys (8d4y), respectively.
We compare our methods with three type-based baselines: AATEAM (Chen et al. 2020), ConvCPD (Ravula al. 2019). PLASTIC (Barrett and Stone 2015). Note that these approaches assume that the ad hoc agent has full visibility on the environment. To apply them on partially observed settings, we replace the full state information used in them with partial observations of the ad hoc agent. Furthermore,
also compare two other strategies: (i) Random: The ad hoc agent chooses actions randomly. (ii) Combined: The ad hoc agent utilizes a DQN algorithm to learn single policy using the data collected from all possible teammates. This intuitive baseline provides the view of treating the problem as a vanilla single-agen ageet learning problem, where the agent ignores the differences between its teammates.
Before training all algorithms, we first require a teammate set that consists of various behavioral policies of teammates. Instead of crafting several teammates' cooperative policies by hand, we expect train set of distinct policies automatically. Therefore, we first utilize 5 different MARL algorithms (e.g. VDN (Sunehag et al. 2018) and QMIX (Rashid al.) 2018)) to develop several teams of agents. To ensure diversity, we use different random seeds for each algorithm and save the corresponding models at 3 different checkpoints (3 million steps, 4 million
7

---

Published as a conference paper at ICLR 2022
steps, and 5 million steps). Then, manually select 15 different policies showing distinct policy representations (Grover et a from all developed models. Finally, we randomly sampled 8 policies as the training set and the other 7 policies as the testing set. During training, we define 8 teammate types that correspond to 8 policies in the training set for the type-based approaches. Then, algorithms would develop their models according to the interactive experience with training teammates. For all algorithms, agents are trained for 4.5 million time steps. The number of captured preys when the ad hoc agent cooperates with testing teammates throughout training is reported. See Supplementary for further settings.
The top panel of Fig. 3 reports the results across 3 scenarios. We first observe that ODITS achieves superior results on the number of captured preys across a varying number of teammates, verifying its effectiveness. ODITS also tends to give more consistent results than other methods across different difficulties. The other 3 type-based baselines and ODITS show better results than random and combined policies, indicating that they can indeed lead to adaptive behaviors to different teammates. Furthermore, the random strategy captures some preys on 4d2y and 8y, but completely fails on 2d1y. This indicates that without the cooperative behaviors of the ad hoc agent, other teammates can also coordinate with each other to achieve the given goal. The combined policy shows worse results than the random policy on two scenarios (4d2y and 8 8d4y). This might be because the combined policy show behaviors that conflict with other teammates. With the number of teammates increasing, the increasing effects of conflicts lead to serious miscoordination.
Results.
5.2 SAVE THE CITY
This is grid world resource allocation task presented in (Iqbal et al. 2020 In this task, there are distinct types of agents, and their goal is to complete the construction of all buildings on the map while preventing them from burning down. Each agent has 8 actions: stay in place, move to the next cell in one of the four cardinal directions, put out the fire, and build. We set the agents to get a team reward of 100 if they have completed building and be penalized for -500 when one building is burned down. Agent types include firefighters (20x speedup over the base rate in reducing fires), builders (20x speedup in the building), or generalists (5x speedup in both as well 2x speedup in moving ). Buildings also consist of two varieties: fast-burning and slow-burning buildings burn four times faster. In our experiments, each agent can only access the environmental information within four cells nearby. We adopt three different scenarios here to verify all methods. They are 2 agents and I buildings (2a2b), 4 agents and buildings (4a3b), 6 agents and 4 buildings (6a4b).
Configurations.
Predatar Paay
08
com: CMX
04
cark
where the fast-burning
Ar comple- nurerge
fimo
Similar to training settings in Predator Prey, we select 15 distinct behavioral policies for the teammate set and randomly partition them into 8 training policies and 7 testing policies. For all algorithms, agents are trained for 4.5 million time steps. The number of completed buildings when the ad hoc agent cooperates with testings teammates throughout training is reported. See Supplementary for further settings.
Figure 4: ODITS V.S.
QMIX.
The bottom panel of Fig. 3 reports the results across scenarios. We first observe that ODITS outperforms other baselines, verifying its effectiveness. Since the setting force all agents in the environment to be heterogeneous, the results also underpin the robustness of ODITS. Interestingly, we find that the combined policy reveals better performance than other type-based approaches. This result is not consistent with that in Predator Prey. Our intuition is that the requirement of cooperative behaviors in Save the City is less than that in Predator Prey. Actually, one agent in Save the City complete buildings individually without the strong necessity of cooperating with other teammates' behaviors. By contrast, one predator cannot capture prey by itself. As a result, the combined policy leams universal and effective policy by learning from more interactive experience with different teammates, while type-based approaches fail because developing distinct cooperative behaviors leads to the instability of the ad hoc agent. This hypothesis is also empirically demonstrated in our ablations.
Results.
5.3 COMPARISON WITH MARL
In order to compare the performance of ODITS and MARL, we implement the commonly used algorithm QMIX (Rashid al. 2018) as our baseline. Similar the training procedure of ODITS, we fix one agent and train it with teammates randomly sampled from pool consisting of 8 policies. The gradients for updating the
8

---

Published as a conference paper at ICLR 2022
teammates' policies are blocked but the mixing network is updating as in the original implementation of QMIX. Figure the comparison of ODITS and QMIX on Predator Prey 4d2y and Save the City 4a3b. In both environments, QMIX performs significantly worse than ODITS. This is not quite surprising because MARL algorithms usually assume that all the teammates are fixed. Therefore, although trained with multiple teammates, the agent under the QMIX framework does not learn to cooperate with an impromptu team.
5.4 ABLATIONS.
We perform several ablations on the Predator Prey 4d2y and Save the City 4 4a to try and determine the importance of each compo- nent of ODITS.
Predator Prey 4etly
Save the City 4ad5
5013
Adaptive Behaviors. We first consider removing the information- based loss Las from the overall learning objective denoted as w/o info.), Fig. shows that without Cass regularizing the informa- tion embedded z4. ODITS induces worse teamwork performance. This indicates that improving the mutual information between the proxy variable and the true variable indeed results in better represen- tations of teamwork situations. We next consider how the inferred variables of teamwork situations affect the ad hoc agent's 's adaptive behaviors. We remove the proxy encoder and set =j as a fixed and randomly generated vector ( denoted as w/o infer.). As shown in Fig. 3 conditioning on a random signal leads to a further drop in performance, indicating that irrelevant signals cannot promote the ad hoc agent to develop adaptive policies.
Figure 5: Ablations for different compo- nents.
Integrating Mechanism. We remove the teamwork situation encoder well as L.s from the framework and feed a vector filled with 1 into the teamwork situation decoder (labeled with w/o integ.). This setting enables ODITS not to integrating the ad hoc agent's marginal utility with the information of teammates' behaviors. Compared with w/o info., it brings larger drop in teamwork performance. One intuition is that predicting the joint-action value plays an essential role in estimating the marginal utility. Suppose the integrating network has no information on other teammates' behaviors. In that case, it cannot accurately predict the joint-action value, resulting instability in marginal utility estimation. Despite the empirical evidence supporting this argument, however, it would be interesting to develop further theoretical insights into this training regime in future work. We finally consider the additive integration mechanism mentioned section 4.2 (labeled with additve). We observe that despite additive integration shows an excellent performance in Save the City, it suffers from poor performance in Predator Prey, indicating that multiplicative integration provides a more stable and effective ability to integrate information from teammates and the ad hoc agent. Interestingly, we also find that most ablations get worse results in Predator Prey than those in Save the city. We believe that the different levels of cooperative requirement in two environments result in this phenomenon. The prey captured when two nearby predators are simultaneously capturing them. By contrast, the burning building can be constructed by an individual agent. Therefore, removing mechanisms that promote the cooperative behaviors leads to the worse performance in Predator Prey.
6 CONCLUSIONS
This paper proposes novel adaptive reinforcement learning algorithm called ODITS to address the challenging ad hoc teamwork problem. Without the need to predefine types of teammates, ODITS automatically learns compact but descriptive variables to infer how to coordinate with previously unknown teammates' behaviors. To overcome partial observability, we introduce an information based regularizer to estimate proxy representations of learned variables from local observations. Experimental results show that ODITS obtains superior performance compared to various baselines on several complex ad hoc teamwork benchmarks.
ACKNOWLEDGMENTS
This research was supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: -ISS--RP-2019-0013 National Satellite of Excellence in Trustworthy Software Systems (Award No: NSOETSS2009-01), and NTU.
9

---

Published as conference paper at ICLR 2022
REFERENCES
S. V. Albrecht and P. Stone. Reasoning about hypothetical agent behaviours and their parameters. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, 2017.
S. V. Albrecht and P. Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-95, 2018.
A. Alemi, Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. International Conference Learning Representations, 2017.
N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. Song, Parisotto, Dumoulin, S. Moitra, E. Hughes, I. Dunning, S. Mourad, H. Larochelle, M. G. Bellemare, and M. Bowling. The hanabi challenge: A new frontier for research. Artificial Intelligence, 280: 103216, 2020.
S. Barrett and P. Stone. Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence Conference on Artificial Intelligence, 2015.
R. Canaan, J. Togelius, A. Nealen, and S. Menzel. Diverse agents for ad-hoc cooperation in hanabi. arXiv preprint arXiv:1907.03844 2019.
S. Chen, E. Andrejezuk, Z. Cao, and J. Zhang. Aateam: Achieving the ad hoc teamwork by employing the attention mechanism. In Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence Conference Artificial Intelligence, pages 7095-7102, 2020.
I. Durugkar, E. Liebman, and Stone. Balancing individual preferences and shared objectives in multiagent reinforcement learning. In Proceedings of the International Joins Conference on Artificial Intelligence, 2020.
J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the Proceedings of the AAAI Conference on Artificial Intelligence Conference on Artificial Intelligence, 2017.
N. A. Genter, Katie and P. Stone. Role-based ad hoc teamwork. In Workshops at the AAAI Conference on Artificial Intelligence, 2011.
Grover, M. Al-Shedivat, J. K. Gupta, Y. Burda, and H. Edwards. Learning policy representations in multiagent systems. In Proceedings of the Proceedings of the International Conference on Machine Learning, pages 1802-1811.
D. Ha, A. Dai, and Q. V. Le. Hypernetworks. arXiv preprint arXiv:1609009110 2016.
M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
H. He, J. Braderraber, K. Kwok, and H. Daumé. Opponent modeling in deep reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 1804-1813, 2016.
S. Iqbal, C. A. S. de Witt, B. Peng. W. Böhmer, S. Whiteson, and F. Sha. Randomized entity-wise factorization for multi-agent reinforcement learning. arXiv preprint arXiv:2006.04220 2020.
Mahajan, T. Rashid, M. Samvelyan, and S. Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, pages 7613-7624, 2019.
R. Mirsky, W. Macke, A. Wang, H. Yedidsion, and P. Stone. A penny for your thoughts: The value of communication in ad hoc teamwork. In Proceedings of the International Joint Conference on Artificial Intelligence, 2020.
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, Asynchronous methods for deep reinforcement learning, In Proceedings of the International Conference on Machine Learning, pages 1928-1937, 2016.
10

---

Published as a conference paper at ICLR 2022
P. Muller, S. Omidshafici, M. Rowland, K. Tuyls, J. Perolat, S. Liu, D. Hennes, L. Marris, M. Lanctot, E. Hughes, Z. Wang, G. Lever, N. Heess, T. Graepel. and R. Munos. A generalized training approach for multiagent learning. In International Conference on Learning Representations, 2020.
F. A. Olichock, M. T. Spaan, N. Vlassis, and S. Whiteson. Exploiting locality of interaction in factored dec- pomdps. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, pages 517-524, 2008.
G. Papoudakis and S. V. Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. In AAAI Workshop on Reinforcement Learning in Games, 2020.
M. A. Rahman, N. Hopner, F. Christianos, and S. V. Albrecht. Towards open ad hoc teamwork using graph-based policy learning. In Proceedings of the International Conference on Machine Learning, pages 8776-8786, 2021.
R. Raileanu, E. Denton, A. Szlam, and R. Fergus. Modeling others using oneself in multi-agent reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages 4257-4266.
R. Raileanu, M. Goldstein, A. Szlam, and R. Fergus. Fast adaptation to new environments via policy- policy-dynamics value functions. In Proceedings of the International Conference on Machine Learning, pages 7920-7931, 2020.
T. Rashid, M. Samvelyan, C. S. de Witt, G. Farquhar, Foerster, and S. Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the International Conference on Machine Learning, pages
2018.
M. Ravula, Alkobi, and P. Stone. Ad hoc teamwork with behavior switching agents. In Proceedings of the International Joint Conference on Artificial Intelligence, 2019.
M. Samvelyan, T. Rashid, C. S. de Witt, G. Farquhar, N. Nardelli,
Rudner, C.-M. Hung, H. Torr,
J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 04000,04443...0 2019.
P. Stone, G. A. Kaminka, S. Kraus, and J. S. Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, 2010.
P. Sunehag. G. Lever, A. Gruslys, W. M. Czarnecki, V. F. 7ambaldi, M. Jaderberg. M. Lanctot, N. Sonnerat, J. 7. Leibo, K. Tuyls, et Value-decomposiiion networks for cooperative multi-agent learning based on team reward. In Proceedings of the International Conference Autonomous Agents and Multiagent Systems, pages 2085-2087, 2018.
M. Suzuki, K. Nakayama, and Y. Matsuo. Joint multimodal learning with deep generative models. arXiv preprini arXiv: 1eee:6111.08880 1.01911, 2016.
Tacchetti, F. Song, P. M. Mediano, V. Zambaldi, J. Kramár, C. Rabinowitz, T. Graepel, M. Botvinick, and P. W. Battaglia. Relational forward models for multi-agent learning. In International Conference on Learming Representations, 2019.
M. Tan. Multi-agent reinforcement learning independent vs. cooperative agents. In Proceedings of the Interna- tional Conference on Machine Learning, 1993.
T. Wang, H. Dong, V. Lesser, and C. Zhang. Roma: Multi-agent reinforcement learning with emergent roles. Proceedings of the International Conference on Machine Learning, 2020.
T. Wang. T. Gupta, A. Mahajan, B. Peng, S. Whiteson, and C. Zhang. Rode: Learning roles to decompose multi-agent tasks. International Conference on Learning Representations, 2021.
G. N. Yannakakis. Game ai revisited. In Proceedings of the 9th conference on Computing Frontiers, pages 285 205-292,200 2012.
H. Yin, F. Melo, A. Billard, and A. Paiva. Associate latent encodings in learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, 2017.
L. Zintgraf, S. Devlin, K. Ciosek, S. Whiteson, and K. Hofmann. Deep interactive bayesian reinforcement learning via meta-learnin. arXiv preprint arXiv:2101.03866 2021.
11

---

Published as conference paper at ICLR 2022
A APPENDIX
A.1 MATHEMATICAL DERIVATION
A.1.1 ADDITIVE INTEGRATION VS. MULTIPLICATIVE INTEGRATION IN INTEGRATING NETWORK.
Concretely, let us consider a simple example that G one-layer network. The information aggregation of additive integration (abbreviated as Gadd) could be written as follow:
Gadd = F(Wuu* + Wock + b)
(10)
where W. and W. are the weight vectors of u' and c', respectively; b is the bias vector; F is the corresponding activation function.
By comparison, the multiplicative integration (abbreviated as Gmaa) could be written as follow:
Gmuit F(Wa(c")Wull + b(c²))
(11)
where ((')) and b(c²) are the function of c² for generating weight vectors.
Compared with Gadd, Gmul changes the integrating network from first order to second order, while introducing no extra parameters. The information of teamwork situation directly scales the information of the ad hoc agent through the term (C'))W,,, rather than via more subtle bias W., u 2 W.c
Further, Good also brings about advantages over Gadd as it alters the integrating network's 's gradient properties. In the additive integration, the gradient of @G.dd can be computed as follow:
0Godd
W.F
(12)
Oc
where F FF= + Wee* + b). This equation reveals that the gradient heavily depends on the matrix W.. but Wt. and u² play a limited role: they only come in the derivative of J mixed with W.u By comparison, aGads is as follow: de
0G [W.u Oct
' W' b'F3"
(13)
where W = W''(')) and b' = b'c)). In this equation. W.u is directly involved in the gradient computation by gating W hence more capble of altering the updates of the learning procedure. This naturally leads to a better information integration.
A.1.2 MUTUAL INFORMATION LOSS FUNCTION LMI
For forcing the proxy representation of the learned latent variable to be infomationally consistent with the true latent variable, propose to maximize the mutual information between the proxy representations and the latent variables. In this paper, we introduce a posterior estimator and derive a tractable lower bound the mutual information term:
p(=Blet bf) p(2)(b))
[(2]:c((b)) = Ezj,cio
log
log 9 (c(2)lc.bb)) bi) p(2)(b))
Ess.ci.08
(14)
+ B.g.o (D (n( (l)llum (DD
log q8 (z(2l bf]
2 E.g.
p( (b:)
where the last inequality holds because of the non-negativity of the KL divergence. Then it follows that:
12

---

Published as a conference paper at ICLR 2022
Defict log (0(2f)c,, bf.
p(zi(b))
=I s [log de (=) ce b)] - s (og((((1)))
(15)
o. [log qE (2) ce b,)] + E, (2)(b))
-Bet.0 /pleslet b)) log qe ct bi def + Eo; (z((6)))))
The proxy encoder is conditioned on the transition data. Given the transitions, the distribution of the proxy representations p(z{) are independent from the local histories. Thus, we have
T(=]: c((b)) =E [CEpp(2]cc: be)lla (xf cl 8))d2(1 + Edj (=(1b))
(16)
where In practice, we sample data from the replay buffer D and minimize
(mr(0g,,EE) -Ld.a.d))-oo (6)l 6))zf(] + Esg (=]]6))
(17)
Poo.sraacyyoolPxxxox//e/lllllle bi]
A.2 ARCHITECTURE, HYPERPARAMETERS, AND INFRASTRUCTURE
300
A.2.1 ODITS
Details of the neural network architectures used by ODITS in all environments are provided in Fig. 7 We control the scale of the loss function by controlling the optimization procedure. It is conducted using RMSprop with a learning rate of 5 x 10-4, a of 0.99, and with no momentum or weight decay. For the lambda value, we search over le 5, le 4, 5e 4, le 3, 5e 3, le 2}. We finally adopt AwS = le - 3.AMr = ll - 3, and AMI 5e for Modified Coin Game, Predator Prey, and Save the City, respectively, since they induce the best performance compared with other values. For the dimension of the latent variables 2 and c,. we search over {1, 2, 5, 10} and finally adopt |a| 10 Save the city and |=| 1 in the other environments. In addition, we set |el = |z]. For
0.5
Figure performance com- parison of two algorithms on the modified coin game. exploration, we use e-greedy with from 1.0 to 0.05 over 50, 000 time steps and kept constant for the rest of the training. Batches of 128 episodes are sampled from the replay buffer, and all components in the framework are trained together in an end-to-end fashion.
A.2.2 BASELINES
We compare ODITS with baselines. For the random strategy, we force the ad hoc agent to choose its action at each time step randomly.
For the comb ned strategy, a three-layer DQN trained to get the policy of the ad hoc agent. Details of DQN's architecture are illustrated in Fig. 7 We sample a type of training teammates each training episode and collect the interacting experience data into the replay buffer. The optimization procedure and the exploration scheme are the same those of ODITS. We set the batch size of samples 128 and the target update interval as 200 episodes.
For the other three type-based baselines, assume that each training policy of teammates corresponds to predefined type of teammates. So, the number of predefined types equals to the number of training polices. We construct set of three-layer DQN to learn the policies for all training types. Each DQN only learns the corresponding teammates' of interacting experience. Training settings for these DQNs are the same as those used in the combines strategy. Furthermore, apply these baselines in partially observable environments, we replace the state information used in them with partial observations. For P PLASTIC (Barrett and Stone 2015),
13

---

Published as a conference paper at ICLR 2022
J
BateWoomalzzttnn
rC, Turils
Parametenang
FC, |A] unns
R
Weights and Rloses
Softrras ax
RelU
LeakeyfleLJJ
FC, 128 units
BatehNoorma zobion
Absolutizator
FC, 64 uniss
FC.
ReLU
FC, 16769
ReeU
FC, 128 units
FC, 64 units
Fi, 20 un ts
ReLU
Tearwork Sttation Encoder
Intagrating Network
Tearwork Stustion Decodirr
DQN
FC, 100 unis
ReLU
unts
LL
matohoooallzttoo
Re.U
Pa ammmteniinggg
Maspool 2X2
FC. [A] units
FC Lnits
Cony, 4DX3X3
Lookey ReUU
Weights Bieses
Le
CRU, 5r unils
Racchoomalbriion
(%
UeeUU
CPD Network
64 units
61 (Ait
FC, [4 unis
FC, 128 units
(o).0(4)
Margiaa Utilty Network
Prozy Encoder
Procy Decoder
Variational Posterior Estimato
Figure 7: Architecture details of ODITS and baselines.
we set the parameter 1) used in the UpdateBelief function as 0.2. For ConvCPD (Ravula et al. 2019), we follow the implementation details mentioned in the original paper. We construct the architecture of the Change Point Detection (CPD) Network as illustrated in Fig. 7 where Rip is the number of predefined types, n, is the number of time steps that needed be considered ConvCPD, Ticlosses = nup x (Tep- - 1) + is the number of classess of change points. The ConCPD algorithm is trained with Necaasee x 1000 samples in which each claess has 1000 samples (batch size- = 64, learning rate- = 0.01, decay=0.1, optimizer=SGD). For AATEAM (Chen et al., 2020), we follow its proposed implementation details. The number of hidden layers for GRU is 5. The maximum length of a state sequence that attention networks process is 60. The rate of dropout is 0.1. During training, the default learning rate is 0.01, and the batch size is 32.
A.3 MODIFIED COIN GAME
To show the difference between ODITS and type-based approaches, we introduce a simple modified coin game. The game takes place on a 7 x 7 map which contains 6 coins of 3 different colors (2 coins of each color). The aim of the team is to only collect any two kinds of coins (correct coins with a reward of 100) and avoid collecting the other kind of coins (false coins with a reward of -200). The policies of the teammates
predefined and illustrated in the order of colors in Fig g left (2 training types and 2 testing types). For example, the first training type (red- - green) indicates that the policy of this teammate is to collect red and green coins, and it will collect red coins firstly. Therefore, while the correct coins of the first training type (green - red) and the second testing type (red + green) are the same, they are different policies since their paths to collect coins are apparently different. Each agent has five actions: move up, down, left, right, or pass. Once an agent steps on a coin, that
14

---

Published as a conference paper at ICLR 2022
coin disappears from the cell. The game ends after 20 steps. To maximize the team return, the aim of the ad hoc agent is to infer its current teammates' desired coins and collect them as much as possible.
Here, we adopt one state-of-the the art type-based approach (AATEAM (Chen et al. 2020 as the baseline. Fig 6 shows the testing performance. We observe that ODITS shows superior performance and converges quickly while AATEAM shows unstable curve. We believe this discrepance results from the key difference between our method and type-hased approaches. The baseline is hard to cooperate with new types of teammates. For example, when the baseline agent interacts with the teammate of the second testing type (green - red) and observes that the teammate is collecting the green coins at the start stage, it would switch its own policy to the one corresponding to the second training type of teammate (green + blue), so it would collect green coins and blue coins (false coins) simultaneously, leading to poor teamwork performance. By contrast, ODITS can be easily generalized the testing types of teammates. During training, ODITS learns how to cooperate with the teammate according to its current behavior instead of its types. If it observes that its teammate is collecting one kind of coins, will collect the same kind of coins, and this knowledge is automatically embedded in e and z.
A.4 DETAILS OF EXPERIMENTS
Modifled Coin Game
Predator Prey
Save the City
agpet
Capturna Moving
Co
Falldd Cape
Predaton
Pross
Polces of Trainin tpcs
Osstadi
H
Foliees Tertin typs
Bucudili capter
Figure 8: Illustration of Modified Coin Game (left), Predator Prey (middle) and Save the City (right)
Modified Coin Game In this environment, the ad hoc agent needs to cooperate with its teammate to collect target coins which depends on the current teammates' type. The teammates' types are illustrated as two ordered colors of coins in Fig. g The teammate follows heuristic strategy and has complete visibility of the entire environmental state. It would first move along an axis for which it has the smallest distance from the coin of its first color. If it collides with obstacles boundaries, it would first choose random actions in 3 steps, and then choose actions to move to the target coin again. If there is no coin of its first color, it would move to the coin of its second color following the same strategy. The ad hoc agent can only access the information 3 grids nearby itself. We use one-hot coding to represent different entities in each grid and concatenate them together as vector to construct the observation of the ad hoc agent. The game ends after 20 steps and takes place on 7 7 grid containing 6 coins of 3 different colors (2 coins of each color). Each agent has five actions: move up, down, left, right, or pass. Once an agent steps on coin, that coin disappears from the grid. This game requires agents to collect as many target coins as possible, which are indicated by the teammates' types. One right collected coin gives a reward of 100, while one false coin gives a reward of -200. As a result, the hoc agent needs to infer its teammate's i goals according to teammates' behaviors and move to right coins while avoiding meeting false coins.
Predator Prey. In this environment, m homogenous predators try to capture n randomly- preys in a 7 x 7 grid world. Each predator has six actions: the moving actions in four directions, the capturing action, and waiting at a grid. Besides, there are two obstacles at random locations. Due to partial observability, each predator can only access the environmental information within two grids nearby. The information of each grid is embedded into a one-hot vector to represent different entities: obstacles, blank grids, predators, preys. Episodes are 40 steps long. The predators get team reward of 500 if two more predators are capturing the same prey simultancously, and they are penalized for -10 if only one of them tries to capture a prey. Here, we adopt three different scenarios to verify the ad hoc agent's ability to cooperate with the different number of teammates. They are 2 predators and preys (2dly), 4 predators and 2 preys (4d2y) and 8 predators and 4 preys (8d4y), respectively.
In this environment, to simulate complex cooperative behaviors, we utilize 5 algorithms (VDN (Sunchag et al. 2018), MAVEN (Mahajan et al., 2019), ROMA (Wang et al. 2020), QMIX (Rashid et al. 2018). IQL Tan
15

---

Published as conference paper at ICLR 2022
2d1y
4d2y
8d4y
Predator Prey
2a2b
4a3b
6a4b
Save the City
Training policies
Testing policies
Unselected policies
Figure 9: t-SNE plots of the learned teammates' policy representations for Predator Prey (top panel) and Save the City (bottom panel) in 6 scenarios. For each scenario, we first developed candidate policies by using existing MARL apen-source implementations. Then, we train a self-supervised policy encoder mentioned in (Raileanu et al. 2020) by learning information from collected behavioral trajectories {(ot a{)} of all candidates to represent their policies. Finally, we select 15 policies whose average policy embeddings distinct from each other, and split them into 8 training policies and 7 testing policies for each scenario.
1993)) to train the candidate policies of teammates using their open-source implementations based on PyMARL Samvelyan et al. 2019). We 4 different random seeds for each algorithm and save the corresponding models at 3 different training steps (3 million, 4 million, and 5 million). As result, we get 60 candidate teammates' policies for each scenario. To ensure diversity of cooperative behaviors,
visualize policy representations of all candidate models by using the open-source implementation mentioned in (Railcanu et al. 2020). This mechanism encodes the agent's behavioral trajectories {(o]. af)} into an embedding space, which different policies generate different representations. Then, we manually select 15 distinct policies and randomly split them training policies and testing policies for each scenario. We illustrate the policy representations in Fig. g During training or testing, in each episode, we sample one policy from the policy set and equip team of agents with the policy. Then, we replace one random agent the team with the ad hoc agent to construct an ad hoc team.
Save the City. This is a 14 x 14 grid world resource allocation task presented (Iqbal et al. 2020). In this task, there are 3 distinct types of agents, and their goal is to complete the construction of all buildings map while preventing them from burning down. Each agent has actions: stay in place, move to the next grid in one of the four cardinal directions, put out the fire, and build. We set the agents get a team reward of 100 if they have completed a building and be penalized for -500 when one building is burned down. Agent types include firefighters (20x speedup over the base rate in reducing fires), builders (20x speedup in the building), or generalists (5x speedup in both as well 2x speedup in moving ). Buildings also consist of two varieties:
the
16

---

Published as conference paper at ICLR 2022
fast-burning and slow-burning, where the fast-burning buildings burn four times faster. In experiments, each agent can only access the environmental information within four grids nearby. The observation each agent contains the information (type, position, and the completion procedure of the building) of each entity in the environment. If one entity is not in the sight range of the agent, the observation vector of this entity would be filled with O. We adopt three different scenarios here to verify all methods. They 2 agents and I buildings (2a2b), 4 agents and 3 buildings (4abb), agents and 4 buildings (6a4b). Similar to Predator Prey, we also manually selected 15 teammates' policies for each scenario. We illustrate their policy representations Fig. / Furthermore, since each team has a random combination of agent types, teammates' behaviors on this environments show greater diversity. In addition, all buildings' types are randomly assigned each episode.
LIMITATIONS AND FUTURE WORKS
In this paper, the experiments were carried with relatively small teams (2-8 agents) and with agent behavior largely coming from pre-trained RL policies. In this case, the diversity of teammates' policies can be easily verified by visualizing the policy representations in latent space. However, with the team size increasing, there might be more complex and diverse teamwork situations. This leads to a higher requirement for the training speed and the scalability of the architecture. addition, the diversity of the policy sets for training has considerable influence on the performance. However, the characterization and quantization of such an influence remain unexplored. We leave these issues for future work.
17


Reviews:
[{"review_id": "lnXP1VeOYo", "reviewer": "Reviewer_X2GL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I liked the paper. The proposed approach strikes me as novel in the context of ad hoc teamwork, although the idea of building multiple encoder-decoder models that learn a common latent representation from diverse sources of information by joint training through the addition of mutual-information loss terms has been explored in the context of multimodal representation learning (see, for example, references [a, b] below). It is also clearly different from the \"main trend\" on ad hoc teamwork, which mostly disregards partial observability.\n\nAt the same time, there is one aspect about the paper that does not fully convince me and which could be further discussed. My main objection is the assumption that---during training---the agent has full access to the other agent's and the environment's state. I'm fully aware that previous approaches (such as PLASTIC or ConvCPD) assume a \"training phase\" whereby the ad hoc agent is allowed to learn about the teammates. However, the interaction between the ad hoc agent and teammates is the same during training and testing---something which, in my view, is key to the ad hoc teamwork formulation: the ad hoc agent builds on its interaction with previous teams to quickly adjust to the current (unknown) team. The notion that the ad hoc agent, during training, is allowed full access to the teammates seems more in line with other settings---such as MARL or, perhaps, transfer in MARL---than with ad hoc teamwork.\n\nRefs.\n\n[a] M. Suzuki, K. Nakayama, Y. Matsuo. \"Joint multimodal learning with deep generative models.\" arXiv:1611.01891, 2016.\n[b] H. Yin, F. Melo, A. Billard, A. Paiva. \"Associate latent encodings in learning from demonstrations.\" AAAI, 2017. ", "summary_of_the_paper": "The paper addresses the problem of ad hoc teamwork, where an agent learns how to coordinate with a team of unknown teammates. Previous works typically assume that the teammates belong to one of a finite set of possible (known) types; at test time, the ad hoc agent uses the observed teammate behavior to identify their corresponding type and act accordingly.\n\nThe proposed method, instead, builds a predictor of the ad hoc agent's marginal utility (which depends on the teammates) that relies on local information alone. The predictor takes as input the ad hoc agent's current observations, $b_t$, and a latent vector, $c$, representing the current \"teamwork situation\". The teamwork situation representation, $c$, roughly plays the role of \"teammate types\" in previous works. To train the predictor, the proposed method makes use of an encoder-decoder model that, at training time, learns the \"teamwork situation\" representation $c$ using full state-action information. The output of the model is used in an \"integration network\" that estimates the $Q$-function for the underlying MMDP from $c$ and the agent's estimated marginal utility, $u_t$. Finally, since at test time the agent cannot access the full state-action information (and hence cannot estimate $c$ using the aforementioned encoder-decoder model), during training it learns an proxy encoder that builds an estimate $z$ for $c$ using only local information. The overall architecture thus includes two encoder-decoder models jointly trained so that they estimate a coherent representation for the \"teamwork situation\"---the first using only local information, and the second using full state-action information. At test time, the agent can use the estimated representation, $z$, as input to its marginal utility predictor and act accordingly.\n\nThe paper tests the proposed approach in 3 domains (modified coins environment, predator-prey, save the city) showing positive results over existing architectures and a well-established MARL approach (QMIX), and also presents a brief ablation study establishing the relevance of using the two encoder-decoder models to the overall performance of the method. ", "summary_of_the_review": "I believe the paper is interesting and provides a novel contribution in terms of ad hoc teamwork (addressing partial observability). It would be good if additional discussion could be made regarding the assumption of \"centralized training\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VfJSVACIWFz", "reviewer": "Reviewer_Z8fc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Just as a forewarning, I'm not highly familiar with this problem so what I say should be taken with a grain of salt, but here are my observations nonetheless.\n\n### Originality\n\nIt seems like the main claims to originality verge on the fact that ODITS:\n1) doesn't require pre-defining agent types\n2) is able to adapt in an online fashion to changing and unseen agents/situations\n3) is able to deal with partially observable environments\n\nTwo works which are not cited, seem to potentially be related (and check at least some of the boxes above) are:\n- Deep Interactive Bayesian Reinforcement Learning via Meta-Learning. From the paper: \"A popular way of modelling other agents is type-based modelling which assumes that the other agent has one of several types that can be pre-defined by an expert, or learned from data [2, 5, 47]. In this paper, we learn a latent variable model, in which the latent variables can be seen as a continuous representation of agent types learned in an unsupervised way via interaction.\". My understanding is that this would check boxes 1 and 2, and might be extendible to also achieve 3.\n- Georgios Papoudakis and Stefano V Albrecht. Variational autoencoders for opponent modeling in multi-agent systems. My understanding is that in this paper they focus on partial observability (3) potentially at the expense of adaptation (2).\n\nIn any case, it would be helpful to add these to the related work and address what the main differences are. If particularly related, they could also constitute additional baselines for the method. MeLIBA in particular seems like a potentially natural choice of baseline to the method presented.\n\n### Clarity\n\nThe paper is well written and overall quite clear. Some minor suggestions/questions:\n- Figure 1: it's unclear what the groups on the right hand side are supposed to represent. For example, why are there 3 green agents and 5 yellow ones? Is this just supposed to represent the proportions with which such agents are sampled?\n- In the \"Marginal Utility\" heading, it might be useful to briefly give a definition of Marginal Utility, rather than just referring to other work. I'm not sure how common this term is in this subfield – it might be standard enough for this to be reasonable.\n- In various places throughout the text, you use the expression \"key insight\". It might be useful to vary this expression or use it more sparingly for the thing that is actually most \"key\" – especially the usage in the introduction seems quite underwhelming.\n- Figure 3: I would separate the legend (containing \"Teammate\", \"Other agent\", and \"Coins\") from the training and testing types. I found the illustration quite confusing.\n- Section 5.1. Partial observability means that the main agent might not see what coins the teammate has taken. I would describe qualitatively what the main agent does in order to address this? Do they try to find the teammate as soon as possible and then follow them? Do they scan the entire grid until they are certain of what coins the teammate took? Neither?\n\nI appreciated the use of ablations in Sec. 5.5 showing the relative importance of the various components of the architecture.", "summary_of_the_paper": "This paper proposes a new method, ODITS, to enable effective ad hoc cooperation without requiring to predefine a static set of teammate types. This is done by training encoder networks to extract latent variables which capture the \"situation type\", which is trained to be a sufficient statistic for predicting the joint action value. This enables the agents to adapt in online fashion (it doesn't assume static other agents), and additionally the methodology is able to deal with partially observable environments.", "summary_of_the_review": "I don't feel qualified to comment on the method's limitations in the broader context of previous work. Outside of the method's quality, the experiments seem well executed and the work is relatively clear. It seems like the method presented does better than the baselines considered, although these might be limited (as mentioned in \"originality\"). Overall, I'll mostly remit to other reviewer's judgement on the method, and wouldn't be opposed to this paper being accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DjZ0Y4zYUjL", "reviewer": "Reviewer_v5Yz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is a pretty cool paper. The core insight, which is learning a flexible and adaptable way of utilizing teammate observation in ad hoc teamwork via latent representations seems to be a very useful approach, and the overall architecture seems sound. I do have some comments and questions, however:\n\n- The relationship between the proxy encoder-decoder and the fully-observable encoder-decoder is not entirely clear to me, and it's not entirely obvious to me how robust the current formulation is. There's also the concern that the entire framework might be very difficult to train for even moderately sized environments such as half field offense.\n\n- To be fair, in practice, most ad hoc teamwork approaches, even if they formally assume a finite set of teammate types, handle variable or novel teammates by learning a distribution over previously modeled types (this can be framed as a k experts problem, or a multi-armed bandit, among other approaches). While AATEAM arguably does this implicitly via the attention mechanism it would still be of interest to consider a simpler baseline whose situational encoder is a distribution over existing learned representations per type - how much does the encoder-decoder approach actually buys you?\n\n- It is unclear to me that the approaches utilized in Section 5.2 to engender variability among teammate types actually gets you this - what concrete measures are there to show that there's real variability in single agent policies learned by varying seeds in QMIX, for instance? One would argue that in the very least a separate effort should be put into maximizing variability, or at least promoting it actively.\n\n- I concede that I did not fully understand the take home message in Section 5.4, is it meant to show that simply following single agent QMIX policy works poorly for an ad hoc agent? That does not seem like a very strong result to show. I would argue that a better experiment would be using  a single agent learning algorithm (say PPO or A3C) and showing that it does worse than the ad-hoc policy learning approach (I would very much expect it to, but at least I understand what that experiment aims to show).\n", "summary_of_the_paper": "In this paper the authors tackle the challenge of ad hoc teamwork with unknown teammate types. Unlike most previous work, the proposed framework, ODITS, does not assume some finite set of teammate types or full observability, and adapts to current teammates via the utilization of a teammate situation encoder-decoder framework, which learns a latent representation of the teammate configuration the agent is currently observing (this observation can be partial). The authors subsequently show this approach outperforms state of the art baselines such as AATEAM on a several domains, namely a modified version of coin game, predator prey, and save the city.", "summary_of_the_review": "Overall this is a reasonably solid paper which presents a novel approach for ad-hoc teamwork policy learning via learning latent embeddings of partial teammate observations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DXviSmyeci", "reviewer": "Reviewer_7i5n", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nAd-hoc teaming is an important problem from the perspecitve of real world applications. Removing some of limitations of required prior knowledge is quite important to allow full data-driven learning of such behavior. Paper's clear focus on this problem leads to integration of many ideas from latent-variable generative modeling in the generative modeling community with multi-agent RL solution techniques in the ODITS framework. The general idea of distilling high-level semantics of teammate behavior into a latent vectors makes sense and has been explored in some of the related works, but doing that in an online manner remained unexplored. Other ideas about using a hypernetwork for integration (as done by QMIX) and utilizing the CTDE assumption to its fullest while learning a proxy encoder for decentralized execution are all well utilized. There is a clear train-test agent separation protocol in the expriments and comparisons with multiple baselines and different numbers of agents and types.\n\n**Weaknesses**\n\nFig 3 fill color seems to suggest something is wrong with AATEAM. In RL the failures are seldom so consistent that unlearning would have such a low spread around the mean. Techniques using some form of bi-level optimization or diversity should be relevant related works [1,2].\nIt's understandable that environments with more complicated dynamics were used given the number of training iterations required for the experiments (which would have been slower too because of additional computation added), it would be useful to add a limitations section to the paper. The experiments were still with relatively small teams and with agent behavior largely coming from pre-trained RL policies. How different the RL policies were might be hard to really understand but it might be useful to report action distribution divergences on pre-selected trajectories along with the t-SNE plot. Overall it's still difficult to say if the selected policies for testing are representative enough as say doing human evaluation where the style of play might be completely different.\n\n[1] https://openreview.net/forum?id=Bkl5kxrKDr\n[2] https://arxiv.org/abs/1907.03840\n\n\n", "summary_of_the_paper": "The paper focuses on the important problem of building agents that can function as useful teammembers in a ad-hoc team where agents can change behavior over an episode. The paper propose a new framework named ODITS that allows training an agent for ad-hoc teaming applications without strong assumptions on observability or pre-defined roles and categories. It does so my learning a latent variable representation of teammate behaviors in a data-driven fashion during policy training. The framework also adds an information-based regularizer to allow learning these variables from partial local observations of an agent (using all available information during training in a CTDE manner). The paper performs experiments on three different environments to demonstrate their outperformance.\n", "summary_of_the_review": "Overall it's a good combination of exisiting ideas in latent-variable modeling and CTDE MARL to demonstrate effective ad-hoc teaming behavior.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for cooperative ad-hoc collaboration by learning latent representations of the teammates. The method is evaluated in three domains. All the reviewers agree that the method is novel and adds an interesting contribution to the important and difficult problem of the ad-hoc collaboration, making fewer assumptions about the team and the teammates.

The next version of the paper should comment:

- On the societal impact of the centralized training.
- Wang et al, CoRL 2020, https://arxiv.org/abs/2003.06906, which addresses the cooperative tasks in the ad-hoc teams without privileged knowledge and assumptions about the teammates.
Decision: Accept (Poster)