# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
###### Abstract
Meanwhile, to more accord with real-world application requirements, the agent is provided with sparse rewards but no environmental model. As a result, we present a method that relies on meta-reinforcement learning and self-play to utilize the prior knowledge about a small set of known types for adaptive cooperation with various teammates, called a Few type-based Ad hoc Teamwork via meta-reinforcement learning (FEAT). The general motivation of this paper is to achieve a combination of (a) offline pre-training for acquiring a repository of diverse teammate types and a highly adaptive policy and (b) online fast adaptation for performing agile cooperation with encountered teammates of known or unknown types. For offline pre-training, we treat the collaboration of the ad hoc agent with different known types of teammates as distinct tasks. Based on this idea, a meta-reinforcement learning (Meta-RL) policy that exhibits sensitivity to task parameters and rapid adaptability to new tasks is acquired. By employing the policy in self-play within environments characterized by stochastic reward perturbations, FEAT produces an extensive repository of clustered usable types along with corresponding collaborative policies, thereby avoiding inefficient online zero-shot adaptation. This reduces the probability of encountering an unknown teammate type during the testing phase, with which the ad hoc agent has no satisfactory way to cooperate. During online inference, if the agent can identify the actual type of the current teammate by leveraging prior knowledge of known teammate types, the agent can swiftly utilize the optimal cooperative policy. Otherwise, due to the high adaptability of the Meta-RL policy, the ad hoc agent can quickly update their policy within a single episode by imitating teammate behaviors without the need for environmental rewards and models. Experimental results in a pursuit domain have confirmed the effectiveness of our approach when there is already only a limited number of known teammate types. To sum up, the main contributions of this paper are as follows. First, we propose a method that maximizes the utilization of a few known teammate types to develop a highly adaptive ad hoc agent policy. This enhances the efficiency of the ad hoc agent in adapting to unknown teammates. Second, we utilize this policy to automatically generate a diverse type repository, thereby improving the usage efficiency of known types and expanding the coverage of the repository for test teammates. Finally, we introduce a training approach for the ad hoc agent based on environmental characteristics, which does not rely on environmental models or rewards, and we demonstrate its effectiveness in a pursuit domain. The organization of this paper is as described below. Section 2 introduces relevant research related to this work. Section 3 covers the background knowledge involved in this work. Section 4 presents the primary methods of this work. Section 5 conducts benchmark experiments and ablation studies to explore the effectiveness of the algorithm, and Section 6 concludes the paper.