# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
Subsequently, random perturbations were introduced to the environment's reward function, enabling the Meta-RL policy to participate in self-play within this perturbed setting and cultivate a diverse type repository. Based on this type repository, ad hoc agents initialized with the Meta-RL policy identify teammate policies while simultaneously imitating teammate behaviors to achieve effective cooperation. Comprehensive experimental analyses in the pursuit domain validated the effectiveness of the individual module designs within the FEAT algorithm, which outperforms representative methods such as PLASTIC [9] and EDRQN [12]. Our approach represents the first explicit algorithm designed for AHT, which focuses on a limited number of known types. The algorithm shows its capability to adapt to teammates within a single episode in partially observable and sparse-reward environments. However, prior related studies usually needed multiple episodes for successful collaboration with teammates. Moreover, based on the foundational assumptions of our algorithm, we suggest several promising directions for future research. Firstly, we assume a role-invariant experimental environment, which allows us to utilize teammate behavior imitation for adaptation. To address environments with variable roles, integrating opponent and environmental modeling with the Meta-RL policy seems to provide a feasible solution. Secondly, we assume that ad hoc agents have access to teammate observations and actions. To relax this assumption, we recommend exploring the concept of ODITS [30]. Finally,

Figure 7: The variation in policy performance of ad hoc agents when cooperating with six distinct types of teammates, as shown in (**a**–**f**). The red line represents the performance of the ad hoc agent after updating its policy via imitation learning without inferring the teammate type. despite creating a diverse type repository, the main purpose of it within the algorithm is to identify teammate policies. To further enhance this capability, we propose integrating transfer learning to infer teammate policies using the type repository. It is important to mention that our experimental setup, which involves only two agents, can be adapted to environments with different numbers of agents without impacting the performance of the algorithm. Conceptualization, Q.F. and J.Z. ; methodology, Q.F. and H.X. ; validation, Q.F. and H.X. ; formal analysis, Q.F. ; investigation, Q.F. and J.Z. ; data curation, Q.F. ; writing--original draft preparation, Q.F. ; writing--review and editing, Q.F., J.Z. and Y.H. ; visualization, J.Z. ; supervision, Y.H., J.Z. and Q.Y. ; funding acquisition, Y.H. All authors have read and agreed to the published version of the manuscript. This work was supported by the Natural Science Foundation of China (grant number 62306329, 62103420) and the Natural Science Foundation of Hunan Province of China (grant number 2023J)40676). Not applicable. The code of EDRQN can be found at [[https://github.com/dxing-cs/EDRQN](https://github.com/dxing-cs/EDRQN)]([https://github.com/dxing-cs/EDRQN](https://github.com/dxing-cs/EDRQN)), accessed on 23 October 2023. The source code of PLASTIC can be found at [[https://github.com/jmribeiro/PLASTIC-Algorithms](https://github.com/jmribeiro/PLASTIC-Algorithms)]([https://github.com/jmribeiro/PLASTIC-Algorithms](https://github.com/jmribeiro/PLASTIC-Algorithms)), accessed on 12 October 2023. And the pursuit environment is contained at [[https://github.com/dxing-cs/EDRQN](https://github.com/dxing-cs/EDRQN)]([https://github.com/dxing-cs/EDRQN](https://github.com/dxing-cs/EDRQN)), accessed on 23 October 2023. We thank [PERSON] et. al. and [PERSON] for the opensource code. The authors declare no conflicts of interest. ## References

* [PERSON] et al. (2020) [PERSON]; [PERSON]; [PERSON] [PERSON]; [PERSON]; [PERSON]; [PERSON] Monotonic value function factorisation for deep multi-agent reinforcement learning. _[PERSON]. Learn. Res._**2020**, _21_, 1-51. * [PERSON] et al.