# Multi-Agent Chronological Planning with Model-Agnostic Meta Reinforcement Learning
## 1 Introduction  
In recent years, Multi-Agent Reinforcement Learning (MARL) has attracted a great deal of interest in the AI community. As a cutting-edge research direction, MARL is closely related to decision theory, game theory, optimization methods, agent-based modeling, and so forth. In real-world scenarios, MARL has a broad range of promising applications, such as robotics control [1; 2; 3], power grid [4; 5; 6], real-time strategy games [7; 8], etc. The use of MARL has achieved remarkable performance and exhibits potential economic benefits. Multi-agent chronological planning problems, where multiple agents need to complete sequential tasks with precedence constraints, are important in both theoretical and real-world domains. However, these problems are yet to be well addressed. In this paper, we consider a chronological scenario where several agents have to cooperatively occupy some landmarks in sequence. This scenario is applicable in many multi-agent systems, such as real-time strategy games and pickup-and-delivery problems. Many problems related to multi-agent systems can be described with Markov Decision Processes (MDPs) and addressed within the framework of MARL. However, when it comes to chronological planning, the process of training policies for multiple agents could be extremely time-consuming to obtain optimal policies since there might be too many task completion orders. Meanwhile, learned policies can be unstable in performance and easily deteriorate when faced with tiny disturbances or other noise. The existing bottleneck inthis domain is that the learned policies may suffer from unsatisfactory performance in a new environment (a new task completion order) even though the drift of an underlying Markov decision process, e.g., system dynamics or the reward mechanism, is quite slight. These challenges render the direct deployment of a previously learned policy in a new task order impractical. However, learning from scratch is not only time-consuming but also fails to leverage prior experience effectively. To circumvent the above-mentioned concerns about MARL, researchers have proposed a novel paradigm--meta-learning--to improve the generalizability of machine learning algorithms with regard to adapting to new environments. Meta-learning is a universal paradigm that can be combined with supervised learning and reinforcement learning. Recent advances in meta-learning enable reinforcement learning to learn from a distribution of different yet related tasks and solve new tasks within a few trials. In the realm of reinforcement learning, meta-learning can be used to learn good initialization parameters [9; 10], improve exploration efficiency [11; 12], and learn proper hyperparameters [13]. The use of meta-learning techniques enables us to leverage learned skills in the past from MARL, and simultaneously improve the robustness to the disturbance of MDPs. Although meta-learning has proved to be effective in adapting policies to new environments, applying meta-learning to multi-agent chronological planning remains less investigated. To close this knowledge gap, we first design a reward setting for the chronological scenario that can effectively describe scenarios of different chronological tasks. Then, we use a classical MARL algorithm--Multi-Agent Deep Deterministic Policy Gradients (MADDPG)--to train the policies of several cooperative agents in the environment. To improve the generalization across different tasks of the learned policies, we encapsulate the MARL part into a meta-learning framework--Reptile--to produce more general policies that can adapt to a new chronological planning scenario with a few interactions. We respectively examine methods on MADDPG with and without a meta-learning framework. We pre-train these two methods on training tasks and then fine-tune the learned policies on two testing tasks to assess the generalization capability. We also consider a random policy as a baseline when fine-tuning on testing stage.