# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification  
In Section 4.2, a diverse type repository has been provided for the ad hoc agent. If the ad hoc agent can identify the current teammate's policy based on limited observations and deduce the teammate's current actions, it can execute the task more effectively, as the Algorithm 2 shows. ```
0:\(\Pi^{\prime}\): Automatically generated type repository
0:\(\pi^{0}(\theta^{*})\): The Meta-RL policy
1: Initialize ad hoc agent policy \(\pi^{0}(\phi)\leftarrow\pi^{0}(\theta^{*})\)
2: Initialize policy \(\pi^{0}_{*}\leftarrow\) None
3:while not done do
4: Get observation \(\mathbf{o}^{0}_{t}\),\(\boldsymbol{\sigma}^{-0}_{t}\),\(\mathbf{a}^{-0}_{t-1}\)
5: Add \(\mathbf{o}^{-0}_{t}\), \(\mathbf{a}^{-0}_{t-1}\) into the replay buffer \(\mathcal{B}\)
6:\(b_{\mathit{finter}}\), \(\mathbf{b}_{\mathit{finter}}\leftarrow\) UpdateBelief(\(\Pi^{\prime},\mathcal{B}\))
7:for\(\Pi^{\prime}_{i}\) in \(\Pi^{\prime}\)do
8:if\(\Pi^{\prime}_{i}\) satisfies the condition (14) with \(\mathbf{b}\mathbf{f}_{\mathit{inter}}\)then
9:\(b_{\mathit{fintra}}\), \(\mathbf{b}\mathbf{f}_{\mathit{intra}}\leftarrow\) UpdateBelief(\(\Pi^{\prime}_{i},\mathcal{B}\))
10:for\(\pi^{\prime}_{ij}\) in \(\Pi^{\prime}_{i}\)do
11:if\(\pi^{\prime}_{ij}\) satisfies the condition (14) with \(\mathbf{b}\mathbf{f}_{\mathit{intra}}\)then
12:\(\pi^{0}_{*}=\arg\max_{\pi^{0}\in\Pi^{\prime}}C(\pi^{0},\pi^{\prime}_{i,j})\)
13:\(\pi^{0}(\phi)\leftarrow\pi^{0}_{*}\)
14:endif
15:endfor
16:endif
17:endfor
18:if\(\pi^{0}_{*}\) is None and \(\mathrm{len}(\mathcal{B})\leq N_{B}\)then
19: Compute \(\mathcal{L}_{IL}(\pi^{0}(\phi))\) based on Equation (16)
20: Update \(\phi\leftarrow\phi-\
abla\mathcal{L}_{IL}(\pi^{0}(\phi))\)
21:endif
22:endwhile
23:Function UpdateBelief (\(\Pi\), \(\mathcal{B}\)):
24:\(n\leftarrow\) the size of \(\Pi\)
25: Initialize belief distribution \(bf\leftarrow[\frac{1}{n},\cdots,\frac{1}{n}]_{n}\)
26: Initialize belief distribution list bf
27:for\(\Pi_{i}\) in \(\Pi\)do
28:for\((\mathbf{o}^{-0}_{t-1},\mathbf{a}^{-0}_{t-1})\) in \(\mathcal{B}\)do
29: Update \(P(\Pi_{i})\) with \((\mathbf{o}^{-0}_{t-1},\mathbf{a}^{-0}_{t-1})\) based on Equation (12)
30:\(bf[i]\gets P(\Pi_{i})\)
31:endfor
32:endfor
33:\(bf=bf/\sum_{i}bf_{i}\)
34: Add \(bf\) in bf
35:return\(bf\), bfBefore initiating collaboration with the teammate, we initialize the Meta-RL policy \(\pi^{0}(\theta^{*})\) (denoted as \(\pi^{0}(\phi)\)) as the initial policy for the ad hoc agent. We assume that the ad hoc agent receives the teammate's previous action \(\mathbf{a}_{t-1}^{-0}\) and current observation \(\mathbf{o}_{t}^{-0}\) at each time step and stores them in the replay buffer \(\mathcal{B}\). Next, the process involves discerning among the policies in the policy repository based on the data in the replay buffer. However, as the types in the repository are automatically generated in an environment with random perturbations, resulting in a large number of generated types, we adopt the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [29] algorithm to classify the type repository based on the similarity matrix \(\mathbf{C}\), obtaining a total of \(u\) type clusters, the policy clusters of which are denoted as \(\bigcup_{i=\{1,\cdots,u\}}\Pi_{i}^{\prime}=\Pi^{\prime}\). For each sample pair \((\mathbf{o}_{t}^{-0},\mathbf{a}_{t}^{-0})\sim\mathcal{B}\) in the replay buffer, we compute the probability of each policy cluster \(\Pi_{i}^{\prime}=\{\pi_{i,1},\cdots\pi_{i,l}\}\) in taking action \(\mathbf{a}_{t}^{-0}\) under observation \(\mathbf{o}_{t}^{-0}\) based on the policy repository.