Online Ad Hoc Teamwork under Partial Observability

1 INTRODUCTION . Recently , autonomous agents including robotics and software agents are being widely deployed in different environments . In many tasks , they are progressively required to cooperate with other unknown teammates on the fly . For example , in case of search and rescue tasks in a disaster , due to privacy or lack of time , deployed robots need to interact with robots from other companies or laboratories , whose coordination protocols might not be explicitly provided in advance ( Barrett and Stone , 2015 ) . Besides , in the domain of game AI ( Yannakakis , 2012 ) , virtual agents are required to assist different agents controlled by human players . To effectively complete these tasks , autonomous agents must show high adaptation ability to collaborate with intrinsically diverse and unknown teammates . This problem is known in the literature as ad hoc teamwork ( Stone et al. , 2010 ) . Existing approaches on ad hoc teamwork usually assume that all teammates ’ behaviors are categorized into several predefined and fixed types , which corresponds to different coordination strategies ( Barrett and Stone , 2015 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ) . Then , by reasoning over the type of interacting teammates , the agent switches its behavior to the corresponding policy . If the types are correctly recognized and the strategies are effective , the agent would accomplish the given cooperation task well . However , defining sufficiently descriptive types of teammates requires prior domain knowledge , especially in uncertain and complex environments . For example , in human-AI collaboration in Hanabi ( Bard et al. , 2020 ) , there are often a wide variety of cooperative behaviors showed by human players . It is challenging for predefined types to cover all possible human players ’ behaviors . Further , teammates ’ strategies might be rapidly evolving throughout the entire teamwork . If the agent assumes that teammates ’ behavioral types are static and can not adapt to current teammates ’ behaviors in an online fashion , teamwork would suffer from serious miscoordination ( Ravula et al. , 2019 ; Chen et al. , 2020 ) . Rescue and search tasks are an essential class of such examples ( Ravula et al. , 2019 ) . On the other hand , existing techniques ( Barrett and Stone , 2015 ; Albrecht and Stone , 2017 ; Chen et al. , 2020 ; Ravula et al. , 2019 ) try to utilize Bayesian posteriors over teammate types to obtain optimal responses . To effectively compute posteriors , they usually assume that the agent could always know other teammates ’ observations and actions . However , this assumption is unrealistic in partial observable environments , in which each agent is not aware of other agents ’ observations . To address the issues mentioned above , this paper introduces an adaptive reinforcement learning framework called Online aDaptation via Inferred Teamwork Situations ( ODITS ) . Our key insight is that teamwork performance is jointly affected by the autonomous agent and other teammates ’ behaviors . So , the agent ’ s optimal behavior depends on the current teamwork situation , which indicates the influence on the environmental dynamics caused by other teammates . If the agent identifies the current teamwork situation in an online fashion , it can choose actions accordingly to ensure effective coordination . In this way , we introduce a multimodal representation learning framework ( Suzuki et al. , 2016 ; Yin et al. , 2017 ) . It automatically encodes the core knowledge about the teamwork situations into a latent probabilistic variable . We show that without any prior knowledge , after learning from the interactive experience with given teammates , the latent variable is sufficiently descriptive to provide information about how to coordinate with new teammates ’ behaviors . To overcome partial observability , we propose an information-based proxy encoder to implicitly infer the learned variables from local observations . Then , the autonomous agent adapts to new teammates ’ behaviors dynamically and quickly by conditioning its policy on the inferred variables . Instead of limiting teammates into several predefined and fixed types , ODITS considers a mechanism of how an agent should adapt to teammates ’ behaviors online . It automatically learns continuous representations of teammates ’ behaviors to infer how to coordinate with current teammates ’ actions effectively . Without domain knowledge on current environments , it enables effective ad hoc teamwork performance and fast adaptation to varying teammates , which the agent might not thoroughly observe under partial observability . In our experimental evaluation , by interactive with a small set of given teammates , the trained agents could robustly collaborate with diverse new teammates . Compared with various type-based baselines , ODITS reveals superior ad hoc teamwork performance . Moreover , our ablations show both the necessity of learning latent variables of teamwork situations and inferring the proxy representations of learned variables . 2 RELATED WORKS . Ad Hoc Teamwork . The core challenge of achieving cooperative ad hoc teamwork is to develop an adaptive policy robust to various unknown teammates ’ behaviors ( Stone et al. , 2010 ) . Existing type-based approaches try to predefine types of teammates and choose policies accordingly to cooperate with unknown teammates ( Chen et al. , 2020 ; Ravula et al. , 2019 ; Durugkar et al. , 2020 ; Mirsky et al. , 2020 ; Barrett and Stone , 2015 ) . Specifically , PLASTIC ( Barrett and Stone , 2015 ) infers types of teammates by computing Bayesian posteriors over all types . ConvCPD ( Ravula et al. , 2019 ) extends this work by introducing a mechanism to detect the change point of the current teammate ’ s type . AATEAM ( Chen et al. , 2020 ) proposes an attention-based architecture to infer types in real time by extracting the temporal correlations from the state history . The drawback of these approaches is that finite and fixed types might not cover all possible situations in complex environments . One recent work avoids predefining teammates ’ types by leveraging graph neural networks ( GNNs ) to estimate the joint action value of an ad hoc team ( Rahman et al. , 2021 ) . However , this work requires all teammates ’ observations as input , which might not always be available in the real world . Agent Modeling . By modeling teammates ’ behaviors , approaches of agent modeling aims to provide auxiliary information , such as teammates ’ goals or future actions , for decision-making ( He et al. , 2016 ; Albrecht and Stone , 2018 ) . For example , MeLIBA conditions the ad hoc agent ’ s policy on a belief over teammates , which is updated following the Bayesian rule ( Zintgraf et al. , 2021 ) . However , existing agent models require the full observations of teammates as input ( Raileanu et al . ; Grover et al . ; Tacchetti et al. , 2019 ) . If the agent can not always observe other teammates ’ information ( e.g . observations and actions ) , those approaches would fail to give an accurate prediction about teammates ’ information . A recent work proposes to use VAE to learn fixed-policy opponents under partial observability . However , it does not generalize to the ad hoc setting where the teammates can be non-stationary ( Papoudakis and Albrecht , 2020 ) . There are also some works study how to generate diverge agent policies , which benefits the training of ad hoc agent ( Canaan et al. , 2019 ) . Multi-agent Reinforcement Learning ( MARL ) . Cooperative MARL ( Foerster et al. , 2017 ) with centralized training and decentralized execution ( Oliehoek et al. , 2008 ) ( CTDE ) is relevant to this work . Related approaches ( Sunehag et al. , 2018 ; Rashid et al. , 2018 ) utilize value function factorization to overcome the limitations of both joint and independent learning paradigms simultaneously . However , these algorithms assume that the developed team is fixed and closed . The team configuration ( e.g. , team size , team formation , and goals ) is unchanged , and agents will not meet other agents without pre-coordination . Several extended works improve the generalization ability for complex team configurations by leveraging other insights , like learning dynamic roles ( Wang et al. , 2021 ; 2020 ) , randomized entity-wise factorization ( Iqbal et al. , 2020 ) , and training regime based on game-theoretic principle ( Muller et al. , 2020 ) . However , intrinsically , these approaches usually focus on co-training a group of highly-coupled agents instead of an autonomous agent that can adapt to non-stationary teammates . 3 BACKGROUND . Problem Formalization . Our aim is to develop a single autonomous agent , which we refer to as the ad hoc agent , that can effectively cooperate with various teammates under partial observability without pre-coordination , such as joint-training . While we focus on training a single agent in this work , similar approaches can be applied to construct an ad hoc team . To evaluate the ad hoc agent ’ s ability to cooperate with unknown teammates , we formally define the problem as a decentralized Partially observable Markov Decision Process ( Dec-POMDP ) ( Oliehoek et al. , 2008 ) with an additional assumption about the set of teammates ’ possible policies Γ . It can be represented as a tuple 〈N , S , A , O , T , P , R , O , Γ〉 , where N denotes the number of agents required by the task , s ∈ S denotes the global state of the environment . The joint action a ∈ AN is formed by all agent ’ s independent actions ai ∈ A , where i is the index of the agent . In the environment , each agent only has access to its partial observation oi ∈ O drawn according to the observation function O ( s , i ) , and it has an observation-action history τ i ∈ T ≡ ( O ×A ) ∗ . P ( s′ , |s , a ) denotes the probability that taking joint action a in state s results in a transition to state s′ . R ( s , a ) is the reward function that maps a state s and a joint action a to a team reward r ∈ R. Γ represents a pool of various policies , which can be pretrained or predefined to exhibit cooperative behaviors . Without loss of generality , we denote by πi the policy of the ad hoc agent and by π−i the joint policy of all other agents . Fig.1 shows the detailed schematics of this problem . Note that the orginal group size of teammate can be arbitrary . In a Dec-POMDP with an additional teammate set Γ , the objective of the ad hoc agent is to maximize the expected team return when it teams up with N − 1 arbitrary teammates sampled from Γ , though it has no prior knowledge about those teammates . Therefore , the ad hoc agent ’ s optimal policy π∗i is required to maximize the joint action value Qπi ( s , ai , a−i ) , which indicates the expected accumulative team reward over different ad hoc teamwork : Qπi ( s , ai , a−i ) = Eait=1 : +∞∼πi , a−it=1 : +∞∼π−i , π−i∼Γ [ +∞∑ t=0 γtrt ∣∣∣∣s0 = s , a0 = a , P ] ( 1 ) Qπ ∗ i ( s , ai , a−i ) ≥ Qπi ( s , ai , a−i ) , ∀πi , s , ai , a−i ( 2 ) Marginal Utility is defined to measure the contribution of an ad hoc agent to the whole team utility ( Genter and Stone , 2011 ) . It represents the increase ( or decrease ) in a team ’ s utility when an ad hoc agent is added to the team . Given teammates ’ actions a−i , there is a relationship between the marginal utility and the team utility ( denoted by the joint action value ) as follow : arg max ai ui ( s , ai , a−i ) = arg max ai Qπi ( s , ai , a−i ) ( 3 ) where ui ( s , ai , a−i ) denotes the marginal utility when the ad hoc agent chooses the action ai under the state s. Note that the marginal utility is not necessarily equal to the Q-value ( Sunehag et al. , 2018 ) . The ad hoc agent chooses the action which maximizes the marginal utility to ensure the maximal team utility . 4 ODITS LEARNING FRAMEWORK Our approach addresses the ad hoc teamwork problem with a novel probabilistic framework ODITS . In this section , we first introduce the overall architecture of this framework and then present a detailed description of all modules in this framework .

Summary: The paper focuses on the important problem of building agents that can function as useful teammembers in a ad-hoc team where agents can change behavior over an episode. The paper propose a new framework named ODITS that allows training an agent for ad-hoc teaming applications without strong assumptions on observability or pre-defined roles and categories. It does so my learning a latent variable representation of teammate behaviors in a data-driven fashion during policy training. The framework also adds an information-based regularizer to allow learning these variables from partial local observations of an agent (using all available information during training in a CTDE manner). The paper performs experiments on three different environments to demonstrate their outperformance.