# Multi-Agent Chronological Planning with Model-Agnostic Meta Reinforcement Learning
## 2 Related Work
### Multi-Agent Reinforcement Learning  
MARL extends RL to a Multi-Agent System (MAS). MARL is much more complex because the agents need to make decisions based on observations and interactions with dynamic environments and their partners [18]. The most natural approach to finding policies for MARL is that each agent learns its policy independently and treats the rest of the agents as part of the environment. This idea is implemented in the Independent Q-Learning (IQL) method [19]. In 2015, [PERSON] et al. [20] proposed Independent Deep Q-Network (IDQN), which extends IQL with DQN. These methods suffer from the non-stationarity of the environment. That means the policies of other agents keep changing during the training process and result in unstable training for the learning algorithm. Non-stationarity also prevents the straightforward use of experience replay in methods like DQN. Meanwhile, policy gradient RL methods usually exhibit very high variance when the coordination of multiple agents is required [21].  
Many approaches in MARL adopt the framework of centralized training and decentralized evaluation (CTDE). In this framework, the policies of a group of agents are trained in a centralized way and granted access to other agents' information and the global states during the centralized training process. While in the decentralized execution phase, each agent makes its own decision based on its local action-observation information. Among these works, DIAL [22], CommNet [23], ATOC [24], and SchedNet [25] aim to enable agents to learn how to communicate with each other in multi-agent systems. Some other approaches try to use a fully observable critic to let agents learn to achieve cooperation directly from their own local observations. MADDPG is the first general-purpose MARL algorithm for stabilizing training. It uses an actor-critic learning framework and can be applied to mixed competitive and cooperative environments. With the fully observable critics, MADDPG eliminates the non-stationary in MARL by explicitly conditioning on the actions of other agents. As the critics in MADDPG concatenate all the local observations, it faces the curse of dimensionality when agents increase. To alleviate this problem, [PERSON] et al. [26] proposed Multiple-Actor-Attention-Critic (MAAC) that involves an attention mechanism to select relevant information for each agent during training.  
In cooperative settings, a group of agents has to coordinate to maximize a shared team reward. Therefore, the global \(Q\) function \(Q^{\pi}(\mathbf{o},\mathbf{a})\) is conditioned on the joint states and actions of all the agents. However, in a multi-agent system, some agents may get lazy and not learn to cooperate as they are supposed to [27], which may lead to the breakdown of the whole system. To solve this issue, some works in MARL focus on the factorization of this global reward. [PERSON] et al. [28] proposed Value-Decomposition Networks (VDN) that learn an optimal linear decomposition for \(Q\) values: \(Q^{\pi}(\mathbf{o},\mathbf{a}):=\sum_{i=1}^{N}Q^{i}(o^{i},a^{i})\). The implicit value function learned by each agent depends only on local observations. As the structure of \(Q\) values considered by VDN is too simple, [PERSON] et al. [7] proposed QMIX, which makes an improvement over VDN by adding the constraints that the joint-actionvalue is monotonic in the per-agent values. [PERSON] et al. [29] also proposed QPLEX, which takes a duplex dueling network architecture to factorize the joint value function, making value function learning more efficient.  
Even though the above MARL methods perform well in some scenarios, when the task varies, one has to retrain the policies from scratch, which takes plenty of time.