# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
Figure 6: A comparative analysis of performance outcomes in self-play, which contrasts the effects of initializing with the Meta-RL policy that of random initialization. The horizontal axis denotes the number of training rounds. (**a**) This plots the average reward gained across 100 episodes following each training round. (**b**) This illustrates the average number of steps needed to complete a task across 100 episodes following each training round. improves. This improvement can be attributed to the enhanced generalization of the Meta-RL policy, enabling the algorithm to better adapt to unknown teammates. Regarding problem 4, we initiated the ad hoc agent using known teammate types with \(N=3\). The agent collaborated with various teammates in the environment for ten episodes and we randomly sampled one of them, which had more than 50 time steps. Throughout this process, we did not utilize the teammate type identification module. Before each episode, the ad hoc agent's policy was reinitialized using the Meta-RL policy. We sampled 20 trajectories of the ad hoc agent collaborating with the current teammate at each time step and calculated the average reward. Figure 7 shows the results of the ad hoc agent collaborating with six different types of teammates, and Table 6 details the performance of each teammate. From Table 6, it is clear that the compatibility between the teammate depicted in Figure 7c and the current Meta-RL policy is minimal. Consequently, as depicted in Figure 7c, their average cooperation reward is approximately \(-100\), which indicates a significant challenge in completing tasks. However, after imitation learning, their rewards experience a rapid and substantial improvement, highlighting the sensitivity of the Meta-RL policy to parameters and the significance of updating policies through imitation learning. In Figure 7b, teammates exhibit maximal compatibility with the Meta-RL policy, as evidenced by uniformly positive cooperation rewards and consistent task completion. The performance experienced a modest enhancement following imitation learning. Further, we analyzed the impact of the cooperative teammates' performance on the joint outcomes, as depicted in Figure 7e,f. Notably, when teammates require fewer steps to complete tasks independently, as demonstrated in Figure 7f, there are fewer samples available for the ad hoc agent to learn from. Despite performance enhancements in the context of previously low compatibility, the limited number of training samples challenges the achievement of superior cooperative performance. \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Win Rate (\%)** & **Causht Prey Id** & **Average Num of Steps** \\ \hline Figure 7a & 93.00 & 0 & 97.68 \(\pm\) 65.58 & 0.67 \\ Figure 7b & 97.00 & 1 & 66.30 \(\pm\) 44.18 & 0.74 \\ Figure 7c & 98.00 & 2 & 11.95 \(\pm\) 2.33 & 0.23 \\ Figure 7d & 97.00 & 3 & 93.53 \(\pm\) 63.53 & 0.41 \\ Figure 7e & 95.00 & 3 & 25.00 \(\pm\) 8.72 & 0.53 \\ Figure 7f & 100.00 & 3 & 11.99 \(\pm\) 2.23 & 0.24 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Diverse teammate performances are depicted in Figure 7, as well as their compatibility with the Meta-RL policy. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Win Rate (\%)** & **Average Num of Steps** \\ \hline \(N=2\) & \(74.33\pm 3.26\) & \(41.92\pm 4.80\) \\ \(N=3\) & \(76.57\pm 1.77\) & \(50.75\pm 6.20\) \\ \(N=4\) & \(83.28\pm 1.86\) & \(45.32\pm 5.08\) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: The efficacy of Meta-RL ad hoc agents, which are initialized with various known teammate types, in collaboration with teammates from \(\Pi_{T}\) without employing Bayesian inference or a diversity type repository. ## 6 Conclusions

In this paper, we introduced an algorithm termed FEAT for AHT, which is predicated on a small set of known teammate types. To optimally utilize prior knowledge of these types, FEAT employs meta-reinforcement learning algorithms to derive a Meta-RL policy that is highly responsive to new tasks.