# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
The environment is deemed **role-invariant** provided there exists any permutation \(pm:\{1,\cdots,n\}\to\{1,\cdots,n\}\) and its corresponding inverse permutation \(pm^{-1}\) such that \(pm^{-1}(pm(\Pi_{\mathcal{E}}))=\Pi_{\mathcal{E}}\) and the following conditions are satisfied:

\[pm^{-1}(pm(\Pi_{\mathcal{E}})(\mathbf{o}_{t}))=\Pi_{\mathcal{E}}(\mathbf{o}_{ t})=\{\pi^{1}(\mathbf{o}_{t}^{1}),\pi^{2}(\mathbf{o}_{t}^{2}),\cdots,\pi^{n}( \mathbf{o}_{t}^{n})\}. \tag{15}\]

Considering an environment with two agents, at each time step, the ad hoc agent randomly selects \(b\) instances \((\mathbf{o}_{t}^{i},\mathbf{a}_{t}^{1})\) from the replay buffer. The loss function is computed as follows. \[\mathcal{L}_{IL}\Big{(}\pi^{0}(\phi)\Big{)}=-\log\pi^{0}(\phi)(\mathbf{a}_{t}^ {1}|\mathbf{o}_{t}^{1}). \tag{16}\]

Utilizing stochastic gradient descent, the loss function is updated to formulate a new policy for the ad hoc agent, facilitating improved cooperation with the teammate, as the Algorithm 2 shows. Furthermore, as the ad hoc agent is initialized using the Meta-RL policy, it displays high adaptability, leading to effective outcomes with limited imitation samples. This approach can also be applied to environments with multiple agents, in which a larger number of samples is added to the replay buffer at each time step, thereby providing more favorable conditions for training the ad hoc agent's policy. ## 5 Experiments

In this section, our experimental study is presented to demonstrate the effectiveness of FEAT. Section 5.1 outlines the experimental environment. Section 5.2 describes the primary settings and research objectives of the experiments. Section 5.3 presents the main experimental results. Section 5.4 showcases the ablation study results, primarily focusing on the design functionalities of various components within the algorithm. ### Domain Description

The experimental environment utilized in this study is identified as the Pursuit Domain, which is a setting where numerous studies about the AHT problem have been undertaken. A variant of the Pursuit Domain similar to that utilized by [PERSON] et al. [12] is employed, with the distinction that in the present environment, a reward of \(1\) is awarded only when both predators capture the same prey simultaneously, contrasting with their setting, where a reward of \(1\) is given as soon as one predator captures the prey. The experimental environment is depicted in Figure 2(a). The map comprises a \(20\times 20\) square grid, representing a toroidal world. Four light blue regions are present on the map, each hosting one prey that is restricted to move randomly within the confines of its respective region. Two predators are randomly placed at the center of each region with the objective of simultaneously capturing one of the four prey. If a predator occupies one of the four adjacent squares around a prey, the prey is considered captured by that predator, as depicted in Figure 2(b). Subsequently, the predator and prey maintain their positions for the next time step. The objective of this experimental setting is for two predators to capture a prey simultaneously. A successful capture yields a reward of \(1\), whereas capturing different prey or exceeding the maximum game steps results in a reward of \(-1\). For all other scenarios, no reward is awarded, defining it as a sparse-reward environment. The task terminates when one of the following conditions is met:

* Both predators simultaneously capture the same prey;
* Both predators capture different prey;
* The task execution steps exceed the maximum step length of the environment. Predators have five potential actions: moving up, down, left, right, or staying stationary. Movement order for the predators is determined randomly, and should an obstacle construct the intended destination, they will remain stationary. Observations encompass their own positions, their teammate's position, and the position of the nearest prey.