# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
The results are depicted in Figure 4. We set \(N=2\), \(N=3\), and \(N=4\), with teammate performance configurations matching those in Table 1.

As depicted in Figure 4, testing rewards steadily improve and converge as training steps increase. Obviously, there is an improvement in policy performance after a gradient descent. Figure 4 clearly indicates that the Meta-RL policy trained with \(N=4\) demonstrates superior performance compared to \(N=3\) and \(N=2\), with \(N=2\) showing the weakest performance. This suggests that the Meta-RL policy exhibits a parameter-sensitive characteristic, requiring minimal samples and iterations to collaborate effectively with unknown teammates. Furthermore, enhancing pre-training with a broader range of known teammate types can significantly enhance their performance in collaborating with unknown teammates. In problem 2, using the policies obtained from \(N=4\), we acquired an additional 35 types to construct the type repository \(\Theta^{\prime}\). We assessed the type repository using the similarity and compatibility metrics outlined in Section 4.2. The visualization of similarity and compatibility results is shown in Figure 5. Herein, the diagonal denotes the outcome of self-play within the same type, indicating the highest similarity and compatibility. The analysis reveals that the types exhibit a moderate level of similarity and substantial differences in compatibility. Furthermore, types with high compatibility tend to have greater similarity, as shown by \(\mathbf{M}(\pi_{30}^{\prime},\pi_{35}^{\prime})\) and \(\mathbf{C}(\pi_{30}^{\prime},\pi_{35}^{\prime})\). This demonstrates the diversity of the type repository. Figure 4: The variation in average episode rewards with training steps during testing with ad hoc team members. (**a**) After each training step, we sampled three teammates and generated 20 trajectories for each teammate. The average reward of 60 trajectories was then plotted on the vertical axis as the average episode reward during testing. (**b**) After each training iteration, an evaluation of the current policy was conducted. Initially, a new teammate was randomly selected, and the current policy collaborated with this teammate to produce 20 trajectories. These trajectories were then utilized as samples to update the policy through a single round of gradient descent. Subsequently, the updated policy collaborated with the same teammate once more, generating an additional 20 trajectories. This procedure was repeated with three different teammates, resulting in a total of 60 trajectories, with the average trajectory rewards depicted on the y-axis. Next, we contrasted the performance of self-play in the environment using policies initialized with the Meta-RL policies and randomly initialized policies using the PPO algorithm, as shown in Figure 6. The reward configuration corresponds to the environment before adjustments were made. The graph clearly shows that the Meta-RL policy requires fewer steps and achieves higher efficiency in obtaining a reliable policy. For problem 3, we did not utilize the automatically generated diverse type repository \(\Theta^{\prime}\) or utilize the Bayesian formula to identify the actual type of teammates during ad hoc collaboration. The performance of the algorithm is illustrated in Table 5. A total of 67 possible teammates are in \(\Theta_{T}\). Despite this, the algorithm achieves a high success rate and completes tasks in a relatively short number of steps. Moreover, as the number and diversity of known teammates types increase, the algorithm's performance gradually

Figure 5: Description of the similarity and compatibility matrices of the type repository. (**a**) Each square depicts the similarity between corresponding policies of type pairs, as identified by the horizontal and vertical coordinates. (**b**) Each square depicts the compatibility between corresponding policies of type pairs, as determined by the horizontal and vertical coordinates.