# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 2 Related Work
### Type-Based Ad Hoc Teamwork  
In current research on AHT, studies focusing on type-based approaches are the most prevalent. This line of approaches typically involves preparing a range of potential teammate types for ad hoc agents. Key aspects of their investigation include (a) how to infer the actual type of teammates from already known types based on a limited set of observations and (b) how to devise cooperative policies for ad hoc agents based on known teammate types. For key point (a), the repository is typically composed of manually designed teammate types. For example, [PERSON] et al. [5] developed a comprehensive tactical library. Also, [PERSON] et al. [9] utilized externally created soccer robot teams as their known teammate types. Moreover, some studies acknowledge the absence of explicit teammate types in the repository, thus requiring learning teammate types online. For instance, [PERSON] et al. [16] proposed the TwoStageTransfer [17] algorithm to learn unknown teammate types based on weighted models from existing manually designed types. This method, utilizing the C4.5 decision tree learner, although straightforward, is computationally intensive. PLASTIC-Model [9] learns models of teammate types from scratch via supervised learning, while PLASTIC-Policy [9] employs Deep Q-Networks (DQN) [18] to learn ad hoc agent policies through online trajectory sampling; both exhibit low utilization rates of known teammate types. Moreover, PLASTIC-Policy often requires multiple episodes to learn effective cooperative policies in sparse-reward environments, whereas our approach only needs a single episode. Additionally, the CTCAT [19] algorithm addresses the issue of mixed types in the repository, which differs from the focus of this study. The manually designed types and the types requiring online learning together form the type repository, provided to the ad hoc agent during the testing phase, known as the testing type repository. Based on it, [PERSON] et al. [5,6] proposed, in the context of small-scale robot soccer leagues, that ad hoc agents maintain a selection weight for three tactics and employ a weight update algorithm to choose the most suitable tactic dynamically. However, this approach relies on environmental feedback rewards, which may be very sparse in real-world cases. The AATEAM [20] algorithm employs attention mechanisms to measure the similarity between current teammate types and those in the testing types repository. However, this method requires a large amount of data to train the attention network, which is inefficient. [PERSON] et al. and [PERSON] et al. [9,16,21] introduced the Bayesian theorem to compute beliefs that the actual teammate type belongs to the testing type repository, and they employed a polynomial weighting algorithm for updating. [PERSON] et al. [7] modeled the AHT problem as stochastic Bayesian games (SBG) and utilized time-specific weighted posteriors for belief updates in the known type repository. All these methods require a repository of a sufficient number of prepared types as support. Comparatively, in our study, there is no need to manually specify a sufficient type repository in advance. Concerning essential point (b), [PERSON] et al. [16; 22] utilized the Monte Carlo Tree Search (MCTS) [23] method to strategize agent behavior in a pursuit domain, while PLASTIC-Model [9] and TEAMSTER [10] employ the Upper Confidence Bounds for Tree (UCT) [23] algorithm for planning ad hoc agent behavior. However, all these methodologies require accurate teammate and environmental models, a condition not stipulated in our setup. Meanwhile, PLASTIC-Policy [9] introduces the Fitted Q Iteration (FQI) [24] algorithm, which eliminates the necessity for an environmental model. However, it mandates multiple episodes for task completion, unlike our method, which accomplishes environmental adaptation within a single episode. [PERSON] et al. [25] introduced the Meta-RL algorithm into the domain of AHT problems, which is similar to our approach.