# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
###### Abstract  
When agents need to collaborate without previous coordination, the multi-agent cooperation problem transforms into an ad hoc teamwork (AHT) problem. Mainstream research on AHT is divided into type-based and type-free methods. The former depends on known teammate types to infer the current teammate type, while the latter does not require them at all. However, in many real-world applications, the complete absence and sufficient knowledge of known types are both impractical. Thus, this research focuses on the challenge of AHT with limited known types. To this end, this paper proposes a method called a Few type-based Ad hoc Teamwork via meta-reinforcement learning (FEAT), which effectively adapts to teammates using a small set of known types within a single episode. FEAT enables agents to develop a highly adaptive policy through meta-reinforcement learning by employing limited priors about known types. It also utilizes this policy to generate a diverse type repository automatically. During the ad hoc cooperation, the agent can autonomously identify known teammate types followed by directly utilizing the pre-trained optimal cooperative policy or swiftly updating the meta policy to respond to teammates of unknown types. Comprehensive experiments in the pursuit domain validate the effectiveness of the algorithm and its components. ad hoc teamwork; meta-reinforcement learning; self-play; agent types; online adapting +
Footnote †: journal:  
optimal reactive policies by pre-training with these teammates, enabling them to infer the types of current teammates during online cooperation based on information such as their trajectories. Consequently, they select cooperative policies accordingly. Representative works in this area include the tactical adaptation method proposed by [PERSON] et al. [5,6] for the CMDragons robot soccer competition. Since 2010, some type-based approaches have also addressed scenarios where the current teammate is not included in the type repository, such as HBA [7], E-HBA [8], PLASTIC [9], and TEAMSTER [10]. Their focus lies in learning the policies of unknown teammates and planning the actions of the ad hoc agent based on these policies. To accurately learn these policies and effectively plan behaviors, a comprehensive environmental model or multiple episode samples are generally required. However, in most real-world scenarios, achieving this is not feasible. The type-free approach argues that providing a set of known teammate types is impractical and ensuring the diversity of manually collected teammate types is challenging. Furthermore, if the type of a teammate is not included in the type repository, ad hoc cooperation performance tends to decline. Therefore, the focus of type-free approaches is to address this challenge by directly adapting to unknown teammates online, such as OPAT [11]; or by equipping agents with diversified behaviors to adapt to them, as demonstrated in EDRQN [12]; or by automatically generating a diverse range of teammate types for the agents, as seen in methods like BRDiv [13], L-BRDiv [14], and the research of [PERSON] et al. [15]. However, in practical scenarios, completely lacking prior knowledge of teammate types is also unrealistic. For instance, in the mentioned case of a pandemic outbreak, temporarily deployed personnel typically possess some level of medical treatment experience to fulfill their roles. These ad hoc agents may have prior collaboration experience with other agents, thereby mastering partial cooperative expertise. Utilizing the prior knowledge of a few known teammate types can also enhance the performance of the ad hoc agent in task completion. Therefore, this paper focuses on investigating a problem setting of more practical interests than both type-based and type-free assumptions, i.e., scenarios where priors about a limited number of teammate types can be leveraged by the ad hoc agent and only a single episode is allowed for adapting to an unknown teammate.