# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 1 Introduction
### Multi-Agent Reinforcement Learning  
The traditional MARL approach [21], which is formulated based on a Markov game [22], enables multiple agents to interact with its operating environment and neighboring agents in a trial and error manner, learn a policy (e.g., a control policy), and perform sequential decision making for optimizing system-wide performance. A Markov game is the generalization of the MDP in the multi-agent and shared environment; specifically, it represents the \(I\)-agent MDP problem \((S,A^{1}\times\ldots\times A^{i},R^{1},\ldots,R^{i},P)\), where \(I=1,\ldots,i,\ldots|I|\) is a set of interacting agents solving a cooperative task [23], \(S\) is the global state represented by the set of states observed by the \(|I|\) agents, and \(a^{i}\in A^{i}\) is an action, as part of the set of joint action \(A^{1}\times\cdots\times A^{i}\), selected by agent \(i\). The transition probability function \(P:S\times A^{1}\times\cdots\times A^{i}\to P(S)\) describes the state transition from \(s_{t}\in S\) to \(s_{t+1}\in S\). Each agent \(i\) receives its own reward, given the reward function \(R^{i}:S\times A^{1}\times\cdots\times A^{i}\to R\) that provides the immediate reward \(r_{t+1}^{i}(s_{t+1}^{i})\in R^{i}\) as a feedback signal for each agent \(i\). Specifically, (a) each agent \(i\in I\) observes state \(s_{t}^{i}\in S\), and selects and executes action \(a_{t}^{i}\in A^{i}\) based on individual policy \(\pi^{i}:S\to P(A^{i})\) at time step \(t\); (b) the environment transits from the current state \(s_{t}\in S\) to next state \(s_{t+1}\in S\); and (c) each agent \(i\in I\) receives immediate reward \(r_{t+1}^{i}(s_{t+1}^{i})\in R^{i}\) based on the joint action \(A^{1}\times\cdots\times A^{i}\) and the transition probability \(P\). Unlike RL, the aim of the agents in MARL is to change their policies in a way that maximizes expected long-term system-wide rewards in the future. In Markov games, it is assumed that each agent \(i\) observes state \(s_{t}^{i}\) that captures all decision-making information of the system. However, in practice, agents may have partial observations about the complex environment only, and so they make sequential decisions under uncertainty. This is referred to as the partially observable Markov game [24] (commonly called partially observable Markov decision process), which is a generalization of both Markov game and MDP, represented by \((S,O^{1}A^{1}\times\ldots\times O^{i}A^{i},R^{1},\ldots,R^{i},P)\), where \(o^{i}\in O^{i}\) is the observation of a local state, as part of the global state \(S\), observed by agent \(i\). Hence, agent \(i\) learns individual policy \(\pi^{i}:O_{t}^{i}\to P(A^{i})\) that maps local observations to individual actions at time step \(t\). For simplicity, an agent \(i\) observation \(o_{t}^{i}\) is represented by a local state \(s_{t}^{i}\). To solve a Markov game, MARL enables multiple agents to interact with the operating environment and exchange knowledge (e.g., Q-value \(Q_{t}^{i}(s_{t}^{i},a_{t}^{i})\) and immediate reward \(r_{t+1}^{i}(s_{t+1}^{i})\)) among themselves, to learn the best possible joint policy \(\pi(a_{t}^{1},\dots,a_{t}^{i}|s_{t})\) in order to achieve global optimization in a collaborative manner as time goes by. While the state transition in RL is attributed to the environmental dynamics of an agent only, the _local_ state transition in MARL is attributed to both environmental dynamics and the non-stationary policies of neighboring agents learning at the same time. Hence, each agent's actions can cause further dynamicity in the operating environment. This problem is called moving target [25], whereby an agent's best possible policy can affect other agents' performances, hence their policies, since all agents select and take their respective actions simultaneously in a shared operating environment. In this case, the convergence property of MDP no longer applies, so the single-agent RL approach may not converge when it is applied in a multi-agent environment.