Online Ad Hoc Teamwork under Partial Observability

1 INTRODUCTION . Recently , autonomous agents including robotics and software agents are being widely deployed in different environments . In many tasks , they are progressively required to cooperate with other unknown teammates on the fly . For example , in case of search and rescue tasks in a disaster , due to privacy or lack of time , deployed robots need to interact with robots from other companies or laboratories , whose coordination protocols might not be explicitly provided in advance ( Barrett and Stone , 2015 ) . Besides , in the domain of game AI ( Yannakakis , 2012 ) , virtual agents are required to assist different agents controlled by human players . To effectively complete these tasks , autonomous agents must show high adaptation ability to collaborate with intrinsically diverse and unknown teammates . This problem is known in the literature as ad hoc teamwork ( Stone et al. , 2010 ) . Existing approaches on ad hoc teamwork usually assume that all teammates ’ behaviors are categorized into several predefined and fixed types , which corresponds to different coordination strategies ( Barrett and Stone , 2015 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ) . Then , by reasoning over the type of interacting teammates , the agent switches its behavior to the corresponding policy . If the types are correctly recognized and the strategies are effective , the agent would accomplish the given cooperation task well . However , defining sufficiently descriptive types of teammates requires prior domain knowledge , especially in uncertain and complex environments . For example , in human-AI collaboration in Hanabi ( Bard et al. , 2020 ) , there are often a wide variety of cooperative behaviors showed by human players . It is challenging for predefined types to cover all possible human players ’ behaviors . Further , teammates ’ strategies might be rapidly evolving throughout the entire teamwork . If the agent assumes that teammates ’ behavioral types are static and can not adapt to current teammates ’ behaviors in an online fashion , teamwork would suffer from serious miscoordination ( Ravula et al. , 2019 ; Chen et al. , 2020 ) . Rescue and search tasks are an essential class of such examples ( Ravula et al. , 2019 ) . On the other hand , existing techniques ( Barrett and Stone , 2015 ; Albrecht and Stone , 2017 ; Chen et al. , 2020 ; Ravula et al. , 2019 ) try to utilize Bayesian posteriors over teammate types to obtain optimal responses . To effectively compute posteriors , they usually assume that the agent could always know other teammates ’ observations and actions . However , this assumption is unrealistic in partial observable environments , in which each agent is not aware of other agents ’ observations . To address the issues mentioned above , this paper introduces an adaptive reinforcement learning framework called Online aDaptation via Inferred Teamwork Situations ( ODITS ) . Our key insight is that teamwork performance is jointly affected by the autonomous agent and other teammates ’ behaviors . So , the agent ’ s optimal behavior depends on the current teamwork situation , which indicates the influence on the environmental dynamics caused by other teammates . If the agent identifies the current teamwork situation in an online fashion , it can choose actions accordingly to ensure effective coordination . In this way , we introduce a multimodal representation learning framework ( Suzuki et al. , 2016 ; Yin et al. , 2017 ) . It automatically encodes the core knowledge about the teamwork situations into a latent probabilistic variable . We show that without any prior knowledge , after learning from the interactive experience with given teammates , the latent variable is sufficiently descriptive to provide information about how to coordinate with new teammates ’ behaviors . To overcome partial observability , we propose an information-based proxy encoder to implicitly infer the learned variables from local observations . Then , the autonomous agent adapts to new teammates ’ behaviors dynamically and quickly by conditioning its policy on the inferred variables . Instead of limiting teammates into several predefined and fixed types , ODITS considers a mechanism of how an agent should adapt to teammates ’ behaviors online . It automatically learns continuous representations of teammates ’ behaviors to infer how to coordinate with current teammates ’ actions effectively . Without domain knowledge on current environments , it enables effective ad hoc teamwork performance and fast adaptation to varying teammates , which the agent might not thoroughly observe under partial observability . In our experimental evaluation , by interactive with a small set of given teammates , the trained agents could robustly collaborate with diverse new teammates . Compared with various type-based baselines , ODITS reveals superior ad hoc teamwork performance . Moreover , our ablations show both the necessity of learning latent variables of teamwork situations and inferring the proxy representations of learned variables . 2 RELATED WORKS . Ad Hoc Teamwork . The core challenge of achieving cooperative ad hoc teamwork is to develop an adaptive policy robust to various unknown teammates ’ behaviors ( Stone et al. , 2010 ) . Existing type-based approaches try to predefine types of teammates and choose policies accordingly to cooperate with unknown teammates ( Chen et al. , 2020 ; Ravula et al. , 2019 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ; Barrett and Stone , 2015 ) . Specifically , PLASTIC ( Barrett and Stone , 2015 ) infers types of teammates by computing Bayesian posteriors over all types . ConvCPD ( Ravula et al. , 2019 ) extends this work by introducing a mechanism to detect the change point of the current teammate ’ s type . AATEAM ( Chen et al. , 2020 ) proposes an attention-based architecture to infer types in real time by extracting the temporal correlations from the state history . The drawback of these approaches is that finite and fixed types might not cover all possible situations in complex environments . One recent work avoids predefining teammates ’ types by leveraging graph neural networks ( GNNs ) to estimate the joint action value of an ad hoc team ( Rahman et al. , 2021 ) . However , this work requires all teammates ’ observations as input , which might not always be available in the real world . Agent Modeling . By modeling teammates ’ behaviors , approaches of agent modeling aims to provide auxiliary information , such as teammates ’ goals or future actions , for decision-making ( He et al. , 2016 ; Albrecht and Stone , 2018 ) . For example , MeLIBA conditions the ad hoc agent ’ s policy on a belief over teammates , which is updated following the Bayesian rule ( Zintgraf et al. , 2021 ) . However , existing agent models require the full observations of teammates as input ( Raileanu et al . ; Grover et al . ; Tacchetti et al. , 2019 ) . If the agent can not always observe other teammates ’ information ( e.g . observations and actions ) , those approaches would fail to give an accurate prediction about teammates ’ information . A recent work proposes to use VAE to learn fixed-policy opponents under partial observability . However , it does not generalize to the ad hoc setting where the teammates can be non-stationary ( Papoudakis and Albrecht , 2020 ) . There are also some works study how to generate diverge agent policies , which benefits the training of ad hoc agent ( Canaan et al. , 2019 ) . Multi-agent Reinforcement Learning ( MARL ) . Cooperative MARL ( Foerster et al. , 2017 ) with centralized training and decentralized execution ( Oliehoek et al. , 2008 ) ( CTDE ) is relevant to this work . Related approaches ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ) utilize value function factorization to overcome the limitations of both joint and independent learning paradigms simultaneously . However , these algorithms assume that the developed team is fixed and closed . The team configuration ( e.g. , team size , team formation , and goals ) is unchanged , and agents will not meet other agents without pre-coordination . Several extended works improve the generalization ability for complex team configurations by leveraging other insights , like learning dynamic roles ( Wang et al. , 2021 ; 2020 ) , randomized entity-wise factorization ( Iqbal et al. , 2020 ) , and training regime based on game-theoretic principle ( Muller et al. , 2020 ) . However , intrinsically , these approaches usually focus on co-training a group of highly-coupled agents instead of an autonomous agent that can adapt to non-stationary teammates . 3 BACKGROUND . Problem Formalization . Our aim is to develop a single autonomous agent , which we refer to as the ad hoc agent , that can effectively cooperate with various teammates under partial observability without pre-coordination , such as joint-training . While we focus on training a single agent in this work , similar approaches can be applied to construct an ad hoc team . To evaluate the ad hoc agent ’ s ability to cooperate with unknown teammates , we formally define the problem as a decentralized Partially observable Markov Decision Process ( Dec-POMDP ) ( Oliehoek et al. , 2008 ) with an additional assumption about the set of teammates ’ possible policies Γ . It can be represented as a tuple 〈N , S , A , O , T , P , R , O , Γ〉 , where N denotes the number of agents required by the task , s ∈ S denotes the global state of the environment . The joint action a ∈ AN is formed by all agent ’ s independent actions ai ∈ A , where i is the index of the agent . In the environment , each agent only has access to its partial observation oi ∈ O drawn according to the observation function O ( s , i ) , and it has an observation-action history τ i ∈ T ≡ ( O ×A ) ∗ . P ( s′ , |s , a ) denotes the probability that taking joint action a in state s results in a transition to state s′ . R ( s , a ) is the reward function that maps a state s and a joint action a to a team reward r ∈ R. Γ represents a pool of various policies , which can be pretrained or predefined to exhibit cooperative behaviors . Without loss of generality , we denote by πi the policy of the ad hoc agent and by π−i the joint policy of all other agents . Fig.1 shows the detailed schematics of this problem . Note that the orginal group size of teammate can be arbitrary . In a Dec-POMDP with an additional teammate set Γ , the objective of the ad hoc agent is to maximize the expected team return when it teams up with N − 1 arbitrary teammates sampled from Γ , though it has no prior knowledge about those teammates . Therefore , the ad hoc agent ’ s optimal policy π∗i is required to maximize the joint action value Qπi ( s , ai , a−i ) , which indicates the expected accumulative team reward over different ad hoc teamwork : Qπi ( s , ai , a−i ) = Eait=1 : +∞∼πi , a−it=1 : +∞∼π−i , π−i∼Γ [ +∞∑ t=0 γtrt ∣∣∣∣s0 = s , a0 = a , P ] ( 1 ) Qπ ∗ i ( s , ai , a−i ) ≥ Qπi ( s , ai , a−i ) , ∀πi , s , ai , a−i ( 2 ) Marginal Utility is defined to measure the contribution of an ad hoc agent to the whole team utility ( Genter and Stone , 2011 ) . It represents the increase ( or decrease ) in a team ’ s utility when an ad hoc agent is added to the team . Given teammates ’ actions a−i , there is a relationship between the marginal utility and the team utility ( denoted by the joint action value ) as follow : arg max ai ui ( s , ai , a−i ) = arg max ai Qπi ( s , ai , a−i ) ( 3 ) where ui ( s , ai , a−i ) denotes the marginal utility when the ad hoc agent chooses the action ai under the state s. Note that the marginal utility is not necessarily equal to the Q-value ( Sunehag et al. , 2018 ) . The ad hoc agent chooses the action which maximizes the marginal utility to ensure the maximal team utility . 4 ODITS LEARNING FRAMEWORK Our approach addresses the ad hoc teamwork problem with a novel probabilistic framework ODITS . In this section , we first introduce the overall architecture of this framework and then present a detailed description of all modules in this framework .

Summary: This paper proposes a new method, ODITS, to enable effective ad hoc cooperation without requiring to predefine a static set of teammate types. This is done by training encoder networks to extract latent variables which capture the "situation type", which is trained to be a sufficient statistic for predicting the joint action value. This enables the agents to adapt in online fashion (it doesn't assume static other agents), and additionally the methodology is able to deal with partially observable environments.