# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Teammate Identification
\[P(\mathbf{a}_{t}^{-0}|\mathbf{o}_{t}^{-0};\Pi_{i}^{\prime})=\frac{1}{\mathcal{ I}}\sum_{j}\pi_{i,j}(\mathbf{a}_{t}^{-0}|\mathbf{o}_{t}^{-0}). \tag{11}\]

Using this likelihood, the posterior probability of the current teammate's policy in different policy classes is calculated as follows:

\[P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o}_{t}^{-0},\mathbf{a}_{t}^{- 0})\Big{)} =(1-\eta\cdot\text{loss})\cdot P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o }_{t-1}^{-0},\mathbf{a}_{t-1}^{-0})\Big{)}, \tag{12}\] \[\text{where, loss } =\ 1-P(\mathbf{a}_{t}^{-0}|\mathbf{o}_{t}^{-0};\Pi_{i}^{\prime}), \tag{13}\]

where \(\eta=0.5\), the prior probability \(P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o}_{0}^{-0},\mathbf{a}_{0}^{-0})\Big{)}\), determining the type class to which the teammate type belongs, is uniformly initialized. Following computation at each time step, the posterior probabilities of all type classes are normalized. The teammate policy is considered to belong to policy class \(\Pi_{i}^{\prime}\) at time step \(t\) when the following conditions are met. The first condition specifies that over \(h\) consecutive time steps, based on the current observation, the \(\Pi_{i}^{\prime}\) obtained from \(\Pi^{\prime}\) that is most similar to the current teammate policy remains consistent. The second condition indicates that over \(h\) successive time steps, the posterior probability of \(\Pi\) converges based on the current observation. \[ \begin{cases}\arg\max_{\Pi_{i}^{\prime}}P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o}_ {t}^{-0},\mathbf{a}_{t}^{-0})\Big{)}=\cdots=\arg\max_{\Pi_{i}^{\prime}}P\Big{(} \Pi_{i}^{\prime}|(\mathbf{o}_{t-h+1}^{-0},\mathbf{a}_{t-h+1}^{-0})\Big{)}\\ \sum_{j=1}^{h}\sum_{p=1}^{h}\bigg{|}P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o}_{t-j} ^{-0},\mathbf{a}_{t-j}^{-0})\Big{)}-P\Big{(}\Pi_{i}^{\prime}|(\mathbf{o}_{t-p }^{-0},\mathbf{a}_{t-p}^{-0})\Big{)}\bigg{|}\leq\zeta\end{cases} . \tag{14}\]

Subsequently, each policy within this policy class is subject to iterative updates of the previously mentioned posterior probability, utilizing observations and actions from the replay buffer. Ultimately, identify the policy \(\pi_{ij}^{-0}\in\Pi_{i}^{\prime}\) that most closely resembles the teammate's policy within the policy repository. Following this, the ad hoc agent continues to explore the repository to identify the policy that is most compatible with the previously identified policy, serving as its current collaborative policy, denoted as \(\pi_{*}^{0}=\arg\max_{\pi^{0}\in\Pi^{\prime}}C(\pi^{0},\pi_{i,j}^{-0})\). ### Teammates' Policies Adaption

If the current teammate type is absent from the type repository, the ad hoc agent is unable to accurately determine the teammate type through updates of posterior probabilities. In the absence of knowledge regarding the teammate type, it is imprudent to blindly adopt behaviors. Additionally, in the context of a sparse-reward environment in which rewards are unattainable during task completion, the ad hoc agent is incapable of updating its policy using reinforcement learning methods within a single episode. Therefore, leveraging environmental characteristics is crucial to facilitate the updating of the policy for the ad hoc agent. Nevertheless, it is noteworthy that should the environment be role-invariant, the ad hoc agent has the capability to update its policy using observations. For an environment \(\mathcal{E}\) comprising \(n\) agents, each with its own policy \(\pi^{i}\), they together form a joint policy \(\Pi_{\mathcal{E}}=\{\pi^{1},\pi^{2},\cdots,\pi^{n}\}\).