# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
### Experiment Setup

To evaluate the effectiveness of our method, we select PLASTIC-Policy [9], a highly representative method among type-based approaches within the PLASTIC algorithm. We excluded PLASTIC-Model, another variant within the PLASTIC algorithm, due to its requirement for an accurate model of the environment, which is unavailable in our experiments. Subsequently, we selected the EDRQN [12] algorithm, representing type-free approaches. In brief, the configuration for the compared algorithms is established as follows:

* **FEAT**: The algorithm proposed in this paper;
* **PLASTIC-\(\Theta\)**: An implementation of a typical type-based algorithm PLASTIC-Policy under the setting of leveraging a constrained set of types \(\Theta\) as the known teammate types, on the online testing phase;
* **PLASTIC-\(\Theta^{\prime}\)**: An implementation of a typical type-based algorithm PLASTIC-Policy under the setting of leveraging the diverse type repository \(\Theta^{\prime}\) obtained in Section 4.2 as the known teammate types, on the online testing phase;
* **EDRQN**: A representative type-free algorithm that the ad hoc agent learns a policy demonstrating diverse behaviors without utilizing any knowledge about known types in the pre-training phase. Subsequently, during the online testing phase, the agent directly cooperates with its teammates;
* **Random**: A baseline wherein the ad hoc agent selects actions randomly at each time step, establishing the lower bound for experimental performance. This study supplies sets of known teammate types numbered 2, 3, and 4, with their performances documented in Table 1. At the end of each episode, the teammates collaborating with the ad hoc agent are switched. Additionally, during the online testing phase, we established three configurations for teammate scenarios:

* **In\(\Theta\)**: Possible teammate types are within \(\Theta\);
* **In\(\Theta^{\prime}\)**: Possible teammate types are within \(\Theta^{\prime}\);

Figure 3: Illustration of the pursuit domain environment. (**a**) The initial state of the environment features two orange predators randomly placed in the central area, and four blue prey randomly placed in the surrounding light blue areas. (**b**) An illustration of prey capture, depicting both predators simultaneously positioned within the neighboring grid to the prey. * **NFIn\(\Theta^{\prime}\)**: Possible teammate policies are not fully within \(\Theta^{\prime}\). The results of the experiments are detailed in Section 5.3. ### Main Results

Leveraging prior knowledge from a range of known types, a total of 35 types were generated. The quantity of these types within the constructed type repository is detailed in Table 2a. Within this table, \(\Theta\) denotes the known types, \(\Theta^{\prime}\) the type repository generated by the algorithm, and \(\Theta_{T}\) the externally supplied types of teammates required for collaboration. During the testing phase for various algorithms, combinations involving the known teammate type repository and the required types for collaboration are detailed in Table 2b. It is noteworthy that neither the EDRQN nor the random algorithms require known teammate types. Additionally, the approach for selecting teammates during testing aligns with that of the aforementioned algorithms. In the testing phase, we implemented a sampling method without replacement to select a type of teammate from \(\Theta_{T}\) for collaboration in each episode. Each algorithm and teammate executed ten collaborative interactions, except for EDRQN, which participated in fifty. Based on this methodology, we calculated the win ratios of the ad hoc agent and the average number of steps required to successfully complete their tasks. The detailed results are presented in Tables 3 and 4. As demonstrated in Tables 3 and 4, the FEAT algorithm exhibits superior performance and robust stability across various experimental setups. It facilitates efficient teamwork even without constraining the actions of teammates to achieve optimality.