# Multi-Agent Chronological Planning with Model-Agnostic Meta Reinforcement Learning
## 3 Methods
### Procedure  
Our method consists of two stages, meta-training and meta-testing. In the meta-training stage, we optimize the model parameters of each agent to minimize their losses on a set of training tasks. In the meta-testing stage, we use the learned meta-parameters to adapt to unseen but related tasks. We collect experiences from a new task and use them to fine-tune the meta parameters to reduce the loss on a specific task.  
As we have shown in the framework of our proposed method (Figure 2), the overall method is a two-level optimization, where the inner loop corresponds to the base learner that only considers a single task, while the outer loop trains the meta parameters \(\theta\) that determine the base learner in the inner loop. In Figure 2, different tasks are implicitly expressed in their corresponding environments. The pseudo-code of meta training is shown in Algorithm 1.  
```
1 Randomize \(\theta\)
2 forachepochdo
3 Sample tasks set \(\mathcal{T}\) from the task distribution.
4 for\(T_{i}\in\mathcal{T}\)do
5\(\phi_{i}\leftarrow\theta\)
6 fork = 1, 2,..., Kdo
7 Sample \(D_{i}=\{(s,\mathbf{a},r,s^{\prime})\}\) using \(f_{\phi_{i}}\) in task \(T_{i}\)
8 Calculate \(\mathcal{L}_{T_{i}}(\phi_{i})\)
9 Inner Update: \(\phi_{i}\leftarrow\phi_{i}-\alpha\
abla_{\phi_{i}}\mathcal{L}_{T_{i}}\big{(} \phi_{i}\big{)}\)
10 end for
11
12
13
14 Meta Update: \(\theta\leftarrow\theta+e\sum\limits_{i=1}^{|\mathcal{T}|}(\phi_{i}-\theta)\)
15
16 end for
17 Return \(\theta\)
```  
**Algorithm 1**_Reptile-MADDPG_  
For every cycle of the outer loop, one or more tasks are sampled from a set of tasks that share some common structure. The current meta parameters \(\theta\) and the sampled task(s) are given to the inner loop, while in the inner loop, agents initialize their network parameters \(\phi\) with \(\theta\) and then interact with the sampled training tasks and optimize their parameters  
Figure 4: Overview of Reptile algorithm.  
for the maximal average reward. In detail, \(N\) agents collect experiences from each training task and store them in the corresponding replay buffers. For every training step, we randomly sample a batch of transitions \(\{(s_{k},\mathbf{a}_{k},r_{k},s^{\prime}_{k})\}^{B}\) from the replay buffer, where \(B\) means the batch size. Then, each agent uses the batch to update its actor and critic based on Equations (8) and (9).  
\[\
abla_{\phi}J(u^{i})=\frac{1}{B}\sum_{k}[\
abla_{\phi}\mu^{i}(\sigma^{i}_{k}) Q^{i}_{\mu}(\mathbf{x},a^{1}_{k},\dots,a^{N}_{k})|_{a^{i}_{k}=\mu^{i}(\sigma^{i}_{k})}] \tag{8}\]  
\[\begin{split} L(\phi)=\frac{1}{B}\sum_{k}[(Q^{i}_{\mu}(\mathbf{x}_{k},a^{1}_{k},\dots,a^{N}_{k})-y_{k})^{2}],\\ y_{k}=r^{i}_{k}+\gamma Q^{\prime i}(\mathbf{x}^{\prime}_{k},a^{\prime 1 }_{k},\dots,a^{\prime N}_{k})|_{a^{i}_{k}=\mu^{j}(\sigma^{i}_{k})}.\end{split} \tag{9}\]  
Every time the policy networks and \(Q\) networks are updated, the corresponding target networks will be soft-updated with hyperparameters \(\tau\) like Equation (10).  
\[\phi^{\prime}\leftarrow(1-\tau)\phi^{\prime}+\tau\phi. \tag{10}\]  
When the inner loop finishes training, the outer loop concludes its results on the batch of tasks to update the meta parameters. To be specific, the outer loop collects the learned policy parameters \(\{\phi_{i}\}_{i=1\dots|\mathcal{T}|}\) and updates the meta parameters \(\theta\) as Equation (11).  
\[\theta\leftarrow\theta+\epsilon\sum_{i=1}^{|\mathcal{T}|}(\phi_{i}-\theta). \tag{11}\]  
After training, the meta parameters \(\theta\) that can quickly adapt to a specific task in the stage of meta-testing are acquired. The process of meta-testing is the same as the inner loop, representing a typical MARL process. With the initialization of \(\theta\), meta-testing could potentially learn good parameters quickly.