# Learning Ad Hoc Cooperation Policies from Limited Priors via Meta-Reinforcement Learning
## 4 Method
### Policy Repository Generation  
The policies of known type repository \(\Theta\) are denoted as \(\Pi=\{\pi_{1}^{-0},\cdots,\pi_{m}^{-0}\}\). The performance of different types varies, affecting the efficiency and behavior of the ad hoc agent when cooperating with them. Following the definition in Section 3.2, we have \(\mathcal{T}=\{\mathcal{T}_{1},\cdots,\mathcal{T}_{m}\}\), comprising a total of \(m\) cooperation source tasks, with the loss function defined across all tasks as follows:  
\[\mathcal{L}_{\mathcal{T}}\Big{(}\pi^{0}(\theta),\Pi\Big{)}=\sum_{\mathcal{T}_ {i}\sim p(\mathcal{T})}\mathcal{L}_{\mathcal{T}_{i}}\Big{(}\pi^{0}(\theta^{ \prime}),\pi_{i}^{-0}\Big{)}=-\sum_{\mathcal{T}_{i}\sim p(\mathcal{T})}J\Big{(} \pi^{0}(\theta^{\prime}),\pi_{i}^{-0}\Big{)}. \tag{6}\]  
Through stochastic gradient descent, the optimization of the above function yields the Meta-RL policy \(\pi^{0}(\theta^{*})=\arg\max\mathcal{L}_{\mathcal{T}}\) for the ad hoc agent, as detailed in Algorithm 1. ```
0:\(\mathcal{T}\): A few tasks of different known types
0:\(N_{in}\): The number of inner-loop iterations, \(N_{out}\): The number of outer-loop iterations
0:\(\pi^{0}(\theta)\): The meta-RL policy of the ad hoc agent
1: Initialize ad hoc agent's policy \(\pi^{0}(\theta)\)
2:for iteration \(n_{o}\) in \(N_{out}\)do
3:for\(\mathcal{T}_{i}\) in \(\mathcal{T}\)do
4:for iteration \(n_{i}\) in \(N_{in}\)do
5: Use policy \(\pi^{0}(\theta)\) to sample \(L\) trajectories \(\tau_{i}\) in \(\mathcal{T}_{i}\)
6: Compute loss \(\mathcal{L}_{\mathcal{T}_{i}}(\pi^{0}(\theta),\pi_{i}^{-0})\) based on Equation (4)
7:\(\theta^{\prime}\leftarrow\theta-\pi\
abla_{\theta}\mathcal{L}_{\mathcal{T}_{i} }(\pi^{0}(\theta),\pi_{i}^{-0})\)
8: Use policy \(\pi^{0}(\theta^{\prime})\) to sample \(L\) trajectories \(\tau_{i}^{\prime}\) in \(\mathcal{T}_{i}\)
9:endfor
10: Compute loss \(\mathcal{L}_{\mathcal{T}}(\pi^{0}(\theta),\Pi)\) based on Equation (6)
11:\(\theta\leftarrow\theta-\beta\
abla_{\theta}\mathcal{L}_{\mathcal{T}}(\pi^{0}( \theta),\Pi)\)
12:endfor
13:endfor
```  
**Algorithm 1** Obtaining Meta-RL policy  
In order to automatically generate a diverse type repository using the obtained Meta-RL policy, we introduce Gaussian noise \(\epsilon\sim N(\mu,\sigma)\) to the environment reward function \(R\), thereby inducing random perturbations and resulting in a series of perturbed reward functions \(\bar{R}\). Figure 1(a) illustrates the initial reward distribution in the environment, characterized by rewards of 1 at the four corners of the grid. Two essential properties must be preserved when perturbing the environment:  
* **Consistency**: The agent's goals should remain within the original goals. * **Reachability**: The agent should be able to achieve its goals. To maintain consistency, a more significant value for the perturbation \(\mu\) of rewards at the agent's original target states and a smaller value for \(\sigma\) are selected. For reachability, smaller values for both \(\mu\) and \(\sigma\) are applied for perturbations at states other than the agent's original target states. Consequently, the perturbed reward function is defined as follows:  
\[\bar{R}= \begin{cases}R(s)+\epsilon_{1},s\in\text{Goal}\\ R(s)+\epsilon_{2},s\
ot\in\text{Goal}\end{cases} . \tag{7}\]  
Among these, we can take \(\epsilon_{1}\sim N(20,5),\epsilon_{2}\sim N(0,1)\). The perturbation outcomes are illustrated in Figure 1(b),c, where rewards at the four target states remain significantly higher than at other states, and the perturbed rewards are adjusted to prevent the agent from being trapped in local optima.