Online Ad Hoc Teamwork under Partial Observability

1 INTRODUCTION . Recently , autonomous agents including robotics and software agents are being widely deployed in different environments . In many tasks , they are progressively required to cooperate with other unknown teammates on the fly . For example , in case of search and rescue tasks in a disaster , due to privacy or lack of time , deployed robots need to interact with robots from other companies or laboratories , whose coordination protocols might not be explicitly provided in advance ( Barrett and Stone , 2015 ) . Besides , in the domain of game AI ( Yannakakis , 2012 ) , virtual agents are required to assist different agents controlled by human players . To effectively complete these tasks , autonomous agents must show high adaptation ability to collaborate with intrinsically diverse and unknown teammates . This problem is known in the literature as ad hoc teamwork ( Stone et al. , 2010 ) . Existing approaches on ad hoc teamwork usually assume that all teammates ’ behaviors are categorized into several predefined and fixed types , which corresponds to different coordination strategies ( Barrett and Stone , 2015 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ) . Then , by reasoning over the type of interacting teammates , the agent switches its behavior to the corresponding policy . If the types are correctly recognized and the strategies are effective , the agent would accomplish the given cooperation task well . However , defining sufficiently descriptive types of teammates requires prior domain knowledge , especially in uncertain and complex environments . For example , in human-AI collaboration in Hanabi ( Bard et al. , 2020 ) , there are often a wide variety of cooperative behaviors showed by human players . It is challenging for predefined types to cover all possible human players ’ behaviors . Further , teammates ’ strategies might be rapidly evolving throughout the entire teamwork . If the agent assumes that teammates ’ behavioral types are static and can not adapt to current teammates ’ behaviors in an online fashion , teamwork would suffer from serious miscoordination ( Ravula et al. , 2019 ; Chen et al. , 2020 ) . Rescue and search tasks are an essential class of such examples ( Ravula et al. , 2019 ) . On the other hand , existing techniques ( Barrett and Stone , 2015 ; Albrecht and Stone , 2017 ; Chen et al. , 2020 ; Ravula et al. , 2019 ) try to utilize Bayesian posteriors over teammate types to obtain optimal responses . To effectively compute posteriors , they usually assume that the agent could always know other teammates ’ observations and actions . However , this assumption is unrealistic in partial observable environments , in which each agent is not aware of other agents ’ observations . To address the issues mentioned above , this paper introduces an adaptive reinforcement learning framework called Online aDaptation via Inferred Teamwork Situations ( ODITS ) . Our key insight is that teamwork performance is jointly affected by the autonomous agent and other teammates ’ behaviors . So , the agent ’ s optimal behavior depends on the current teamwork situation , which indicates the influence on the environmental dynamics caused by other teammates . If the agent identifies the current teamwork situation in an online fashion , it can choose actions accordingly to ensure effective coordination . In this way , we introduce a multimodal representation learning framework ( Suzuki et al. , 2016 ; Yin et al. , 2017 ) . It automatically encodes the core knowledge about the teamwork situations into a latent probabilistic variable . We show that without any prior knowledge , after learning from the interactive experience with given teammates , the latent variable is sufficiently descriptive to provide information about how to coordinate with new teammates ’ behaviors . To overcome partial observability , we propose an information-based proxy encoder to implicitly infer the learned variables from local observations . Then , the autonomous agent adapts to new teammates ’ behaviors dynamically and quickly by conditioning its policy on the inferred variables . Instead of limiting teammates into several predefined and fixed types , ODITS considers a mechanism of how an agent should adapt to teammates ’ behaviors online . It automatically learns continuous representations of teammates ’ behaviors to infer how to coordinate with current teammates ’ actions effectively . Without domain knowledge on current environments , it enables effective ad hoc teamwork performance and fast adaptation to varying teammates , which the agent might not thoroughly observe under partial observability . In our experimental evaluation , by interactive with a small set of given teammates , the trained agents could robustly collaborate with diverse new teammates . Compared with various type-based baselines , ODITS reveals superior ad hoc teamwork performance . Moreover , our ablations show both the necessity of learning latent variables of teamwork situations and inferring the proxy representations of learned variables . 2 RELATED WORKS . Ad Hoc Teamwork . The core challenge of achieving cooperative ad hoc teamwork is to develop an adaptive policy robust to various unknown teammates ’ behaviors ( Stone et al. , 2010 ) . Existing type-based approaches try to predefine types of teammates and choose policies accordingly to cooperate with unknown teammates ( Chen et al. , 2020 ; Ravula et al. , 2019 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ; Barrett and Stone , 2015 ) . Specifically , PLASTIC ( Barrett and Stone , 2015 ) infers types of teammates by computing Bayesian posteriors over all types . ConvCPD ( Ravula et al. , 2019 ) extends this work by introducing a mechanism to detect the change point of the current teammate ’ s type . AATEAM ( Chen et al. , 2020 ) proposes an attention-based architecture to infer types in real time by extracting the temporal correlations from the state history . The drawback of these approaches is that finite and fixed types might not cover all possible situations in complex environments . One recent work avoids predefining teammates ’ types by leveraging graph neural networks ( GNNs ) to estimate the joint action value of an ad hoc team ( Rahman et al. , 2021 ) . However , this work requires all teammates ’ observations as input , which might not always be available in the real world . Agent Modeling . By modeling teammates ’ behaviors , approaches of agent modeling aims to provide auxiliary information , such as teammates ’ goals or future actions , for decision-making ( He et al. , 2016 ; Albrecht and Stone , 2018 ) . For example , MeLIBA conditions the ad hoc agent ’ s policy on a belief over teammates , which is updated following the Bayesian rule ( Zintgraf et al. , 2021 ) . However , existing agent models require the full observations of teammates as input ( Raileanu et al . ; Grover et al . ; Tacchetti et al. , 2019 ) . If the agent can not always observe other teammates ’ information ( e.g . observations and actions ) , those approaches would fail to give an accurate prediction about teammates ’ information . A recent work proposes to use VAE to learn fixed-policy opponents under partial observability . However , it does not generalize to the ad hoc setting where the teammates can be non-stationary ( Papoudakis and Albrecht , 2020 ) . There are also some works study how to generate diverge agent policies , which benefits the training of ad hoc agent ( Canaan et al. , 2019 ) . Multi-agent Reinforcement Learning ( MARL ) . Cooperative MARL ( Foerster et al. , 2017 ) with centralized training and decentralized execution ( Oliehoek et al. , 2008 ) ( CTDE ) is relevant to this work . Related approaches ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ) utilize value function factorization to overcome the limitations of both joint and independent learning paradigms simultaneously . However , these algorithms assume that the developed team is fixed and closed . The team configuration ( e.g. , team size , team formation , and goals ) is unchanged , and agents will not meet other agents without pre-coordination . Several extended works improve the generalization ability for complex team configurations by leveraging other insights , like learning dynamic roles ( Wang et al. , 2021 ; 2020 ) , randomized entity-wise factorization ( Iqbal et al. , 2020 ) , and training regime based on game-theoretic principle ( Muller et al. , 2020 ) . However , intrinsically , these approaches usually focus on co-training a group of highly-coupled agents instead of an autonomous agent that can adapt to non-stationary teammates . 3 BACKGROUND . Problem Formalization . Our aim is to develop a single autonomous agent , which we refer to as the ad hoc agent , that can effectively cooperate with various teammates under partial observability without pre-coordination , such as joint-training . While we focus on training a single agent in this work , similar approaches can be applied to construct an ad hoc team . To evaluate the ad hoc agent ’ s ability to cooperate with unknown teammates , we formally define the problem as a decentralized Partially observable Markov Decision Process ( Dec-POMDP ) ( Oliehoek et al. , 2008 ) with an additional assumption about the set of teammates ’ possible policies Γ . It can be represented as a tuple 〈N , S , A , O , T , P , R , O , Γ〉 , where N denotes the number of agents required by the task , s ∈ S denotes the global state of the environment . The joint action a ∈ AN is formed by all agent ’ s independent actions ai ∈ A , where i is the index of the agent . In the environment , each agent only has access to its partial observation oi ∈ O drawn according to the observation function O ( s , i ) , and it has an observation-action history τ i ∈ T ≡ ( O ×A ) ∗ . P ( s′ , |s , a ) denotes the probability that taking joint action a in state s results in a transition to state s′ . R ( s , a ) is the reward function that maps a state s and a joint action a to a team reward r ∈ R. Γ represents a pool of various policies , which can be pretrained or predefined to exhibit cooperative behaviors . Without loss of generality , we denote by πi the policy of the ad hoc agent and by π−i the joint policy of all other agents . Fig.1 shows the detailed schematics of this problem . Note that the orginal group size of teammate can be arbitrary . In a Dec-POMDP with an additional teammate set Γ , the objective of the ad hoc agent is to maximize the expected team return when it teams up with N − 1 arbitrary teammates sampled from Γ , though it has no prior knowledge about those teammates . Therefore , the ad hoc agent ’ s optimal policy π∗i is required to maximize the joint action value Qπi ( s , ai , a−i ) , which indicates the expected accumulative team reward over different ad hoc teamwork : Qπi ( s , ai , a−i ) = Eait=1 : +∞∼πi , a−it=1 : +∞∼π−i , π−i∼Γ [ +∞∑ t=0 γtrt ∣∣∣∣s0 = s , a0 = a , P ] ( 1 ) Qπ ∗ i ( s , ai , a−i ) ≥ Qπi ( s , ai , a−i ) , ∀πi , s , ai , a−i ( 2 ) Marginal Utility is defined to measure the contribution of an ad hoc agent to the whole team utility ( Genter and Stone , 2011 ) . It represents the increase ( or decrease ) in a team ’ s utility when an ad hoc agent is added to the team . Given teammates ’ actions a−i , there is a relationship between the marginal utility and the team utility ( denoted by the joint action value ) as follow : arg max ai ui ( s , ai , a−i ) = arg max ai Qπi ( s , ai , a−i ) ( 3 ) where ui ( s , ai , a−i ) denotes the marginal utility when the ad hoc agent chooses the action ai under the state s. Note that the marginal utility is not necessarily equal to the Q-value ( Sunehag et al. , 2018 ) . The ad hoc agent chooses the action which maximizes the marginal utility to ensure the maximal team utility . 4 ODITS LEARNING FRAMEWORK Our approach addresses the ad hoc teamwork problem with a novel probabilistic framework ODITS . In this section , we first introduce the overall architecture of this framework and then present a detailed description of all modules in this framework .

Summary: The paper addresses the problem of ad hoc teamwork, where an agent learns how to coordinate with a team of unknown teammates. Previous works typically assume that the teammates belong to one of a finite set of possible (known) types; at test time, the ad hoc agent uses the observed teammate behavior to identify their corresponding type and act accordingly.  The proposed method, instead, builds a predictor of the ad hoc agent's marginal utility (which depends on the teammates) that relies on local information alone. The predictor takes as input the ad hoc agent's current observations, $b_t$, and a latent vector, $c$, representing the current "teamwork situation". The teamwork situation representation, $c$, roughly plays the role of "teammate types" in previous works. To train the predictor, the proposed method makes use of an encoder-decoder model that, at training time, learns the "teamwork situation" representation $c$ using full state-action information. The output of the model is used in an "integration network" that estimates the $Q$-function for the underlying MMDP from $c$ and the agent's estimated marginal utility, $u_t$. Finally, since at test time the agent cannot access the full state-action information (and hence cannot estimate $c$ using the aforementioned encoder-decoder model), during training it learns an proxy encoder that builds an estimate $z$ for $c$ using only local information. The overall architecture thus includes two encoder-decoder models jointly trained so that they estimate a coherent representation for the "teamwork situation"---the first using only local information, and the second using full state-action information. At test time, the agent can use the estimated representation, $z$, as input to its marginal utility predictor and act accordingly.  The paper tests the proposed approach in 3 domains (modified coins environment, predator-prey, save the city) showing positive results over existing architectures and a well-established MARL approach (QMIX), and also presents a brief ablation study establishing the relevance of using the two encoder-decoder models to the overall performance of the method.