# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Policy Repository Generation
The obtained Meta-RL policy \(\pi^{0}(\theta^{*})\) then engages in self-play in a series of perturbed environments, with the loss function computed as follows:  
\[\mathcal{L}_{SP}\Big{(}\pi^{0}(\theta^{*})\Big{)}=-\Big{(}\pi^{0}(\theta^{*}), \pi^{0}(\theta^{*})\Big{)}=-\mathbb{E}_{\mathbf{o}_{t},\mathbf{a}_{t}\sim\{\pi ^{0}(\theta^{*}),\pi^{0}(\theta^{*})\}}\left[\sum_{t=0}^{H}\gamma^{t}\bar{R}_{t }(\mathbf{o}_{t},\mathbf{a}_{t})\right]. \tag{8}\]  
After multiple training rounds, we generate a considerable number of effective policies, which augment the initial set of \(m\) policies to form the policy repository \(\Pi^{\prime}\), also known as the type repository \(\Theta^{\prime}\). Subsequently, to enhance the efficiency of identifying the teammate and obtaining the best-response cooperative policy, we evaluate the diverse policies within this repository from two perspectives \((\pi_{i},\pi_{j})\subseteq\Pi^{\prime}\):  
* **Similarity**. Since policies represent the probability distributions of different actions under specific observations, the comparison of the similarity between two policies involves the comparison of the similarity between two probability distributions. We chose the Jensen-Shannon (JS) divergence to measure the similarity between policies. Although JS divergence exhibits symmetry, it has a limitation in its inability to update gradients when two distributions do not completely overlap. This situation results in a constant divergence value regardless of the distance between their centers, leading to a gradient of zero and preventing gradient updates. However, this limitation does not influence our method, since we do not employ gradients. Therefore, the formula for computing the similarity of policy pairs is as follows, where \(q=\frac{1}{2}(\pi_{i}+\pi_{j})\): \[M(\pi_{i},\pi_{j}):=\frac{1}{2}\mathcal{D}_{KL}(\pi_{i}||q)+\frac{1}{2} \mathcal{D}_{KL}(q||\pi_{j}).\] (9)
* **Compatibility**. This concept is derived from LIPO [28]. We extend its scope to measure the compatibility between pairs of policies, defining it in terms of task completion, task performance, and task completion time. The expressions are as follows: \[C(\pi_{i},\pi_{j}):=\frac{1}{K}\sum_{k=1}^{K}d(\pi_{i},\pi_{j})\Bigg{[}\lambda _{1}\sum_{t}R_{t}+\lambda_{2}(1-N_{step}/N_{\text{max}})\Bigg{]}.\] (10) In this context, \(d(\pi_{i},\pi_{j})\) measures the ability of the policy pair to complete the task, where \(d=1\) indicates successful completion and \(d=0\) denotes failure. \(\sum_{t}R_{t}\) assesses the rewards gathered by the policy pair within a single episode, over a total of \(K\) episodes. The third term \((1-N_{step}/N_{\text{max}})\) measures the duration of task completion, where \(N_{step}\) signifies the number of steps required for task completion, and \(N_{max}\) represents the maximum number of steps permissible in the environment. In reinforcement learning training, a discount factor \(\gamma\in(0,1]\) is typically applied to the accumulated  
Figure 2: Visualization of the perturbed environment reward function. (**a**) Initial reward distribution in the environment, with four highlighted red areas representing the target regions for the agents to reach. (**b**) A situation of reward distribution after one perturbation. (**c**) Another situation of reward distribution after one perturbation. rewards to regulate the agent's focus on short-term rewards. It also serves to minimize the number of steps required to complete the task during training; the more steps taken, the smaller the final returns \(\gamma^{t}R_{t}\) accumulated. We aim to prevent the agent's returns from decreasing exponentially with an increase in steps; therefore, we compute the task completion reward and the completion steps independently, using \(\lambda_{1}\) and \(\lambda_{2}\) to manage their respective importance. After calculating the similarity and compatibility for all policy pairs in \(\Pi^{\prime}\), we obtain the similarity matrix \(\mathbf{M}\) and the compatibility matrix \(\mathbf{C}\).