# Multi-Agent Chronological Planning with Model-Agnostic Meta Reinforcement Learning
## 2 Related Work
### Meta Reinforcement Learning  
Early works on the idea of meta-learning (also known as _learning to learn_) date back to the 1990s [30; 31]. Recently, meta-learning has received much attention and has been applied in few-shot recognition [32], network routing [33], traffic control [34], and in biomedical [35] and other domains [36; 37; 38; 39; 40; 41]. Meta-learning is regarded as the key to achieving human-level intelligence because it raises the learning level from data to tasks [42]. Here, we focus on meta-learning in the reinforcement learning community, meta reinforcement learning (meta-RL). The goal of meta-RL is to adapt to a new test task quickly using only a small amount of experience in the test setting [9] without learning from scratch. In meta-RL, the train and test tasks are usually different but drawn from the same family of problems [43]. In this paper, we divide the recent methods of meta-reinforcement learning into recurrent neural network-based and optimization-based methods. RNN-based meta-RL is very similar to the standard RL algorithms. The current state \(s_{t}\), the last reward \(r_{t-1}\), and the last action \(a_{t-1}\) are also fed into the policy network of model-based meta-RL. [PERSON] et al. [44] proposed a meta learning method that trains a gated recurrent unit (GRU) to remember the construct of the current MDP, so that it can adapt to an unseen but familiar MDP task quickly by fine-tuning the parameters of the GRU. During the training procedure, the hidden states of the GRU are not cleared between episodes. Similar to [44], [PERSON] et al. [45] used an LSTM as the memory module. [PERSON] et al. [46] present a hierarchical meta reinforcement learning method that learns hierarchically structured policies and uses shared primitives to improve sample efficiency on new tasks. Optimization-based meta-RL aims to update the model parameters to achieve good generalization across new tasks. [PERSON] et al. [9] proposed a general method called model-agnostic meta learning (MAML). MAML aims to optimize initialized parameters on some training tasks so that the parameters can quickly adapt to similar unseen tasks. MAML is model-agnostic so that it can combine with any learning models trained with gradient-descent. To reduce the expensive computation from the use of second-derivatives in MAML, [PERSON] et al. [10] proposed a first-order meta learning method, Reptile, that performs SGD that updates the initialized parameters towards the average of updated task-specific weights. Evolved Policy Gradient aims to build a differentiable loss function that is parameterized via temporal convolutions over the agent's historical information [47]. When tackling new tasks, one can optimize the policy by minimizing the loss function. Meta Q-Learning (MQL) [48] is an off-policy meta learning method and draws upon ideas in propensity estimation to amplify the amount of available data for adaptation. Recently, some works tried to apply meta learning to multi-agent systems. [PERSON] et al. [49] studied the continuous adaptation problem in a multi-agent environment, RoboSumo, and enabled a learning agent to adapt to different enemies in a short timeframe. However, they did not actually research multi-agent learning, but focused on a single agent in a two-agent competitive environment. [PERSON] et al. [50] combined meta learning with MADDPG and successfully enabled MADDPG to adapt to new tasks, with different friction coefficients or numbers of agents. [PERSON] et al. [18] trained a meta-actor and a meta-critic to distill the meta-knowledge of a team that can help a new agent to better integrate into the group. Both [18; 50] tested their meta learning methods in multi-agent systems. Existing works primarily focus on variations in kinetic parameters or the number of agents between training and testing tasks, neglecting the interrelationships among agents. In contrast, our study addresses the complexity of chronological constraints between agents.