# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 1 Introduction
### Reinforcement Learning  
The traditional RL approach [13], which is formulated based on Markov decision process (MDP) [8; 12; 14; 15], enables a _single_ agent (or a decision maker) to interact with its operating environment in a trial and error manner, learn a policy (e.g., a control policy), and perform sequential decision making for optimizing system performance. One of the popular SARL (single-agent reinforcement learning) (called RL for simplicity henceforth) approaches is Q-learning. In Q-learning, an agent learns action-value functions, and it has three main advantages in that an agent: (a) models the system performance, instead of individual factors affecting the performance; (b) does not require prior knowledge about the dynamic operating environment; and (c) does not require transition probability. Q-learning enables an agent \(i\) to observe state \(s^{i}_{t}\in S\) and select the best possible action \(a^{i}_{t}\in A^{i}\) at time \(t\), and then receive an immediate reward \(r^{i}_{t+1}(s^{i}_{t+1})\) at time \(t+1\). Subsequently, the agent \(i\) updates Q-value \(Q^{i}_{t}(s^{i}_{t},a^{i}_{t})\) that represents the appropriateness of taking action \(a^{i}_{t}\) under state \(s^{i}_{t}\) for the state-action pair \((s^{i}_{t},a^{i}_{t})\) as follows:  
\[Q^{i}_{t+1}(s^{i}_{t},a^{i}_{t})\leftarrow(1-\alpha)Q^{i}_{t}(s^{i}_{t},a^{i}_ {t})+\alpha[r^{i}_{t+1}(s^{i}_{t+1})+\gamma\text{max}_{a\in A}Q^{i}_{t}(s^{i}_ {t+1},a)] \tag{1}\]  
where \(\alpha\in[0,1]\) represents the learning rate, \(\gamma\text{max}_{a\in A}Q^{i}_{t}(s^{i}_{t+1},a)\) represents the discounted reward, and \(\gamma\in[0,1]\) represents the preference for the discounted reward compared to the immediate reward. A two-dimensional look-up Q-table is used to store \(|S^{i}|\times|A^{i}|\) entries of agent \(i\)'s Q-values; hence, the Q-table size increases exponentially with increasing number of states \(|S^{i}|\) and actions \(|A^{i}|\), particularly in complex operating environments, resulting in the disadvantages of the curse of dimensionality and longer convergence time to the optimal action. Algorithm 1 shows the Q-learning algorithm. An agent \(i\) observes the current state \(s^{i}_{t}\) (step 2) and chooses an action \(a^{i}_{t}\) (step 3) at time \(t\). The agent \(i\) selects one of the two types of actions: (a) the exploitation action \(a^{i}_{t}=\text{argmax}_{a\in A}Q^{i}_{t}(s^{i}_{t},a)\) has the maximum Q-value, and it is chosen to maximize value function \(v^{i}_{\tau}(s^{i}_{t})\); and (b) the exploration is a random action, and it is chosen to discover better actions in order to adapt to the operating environment. The agent can use techniques such as \(\varepsilon\)-greedy [16] and the softmax [13] approach, to achieve a balanced trade-off between exploration and exploitation based on the dynamicity of the operating environment. The agent \(i\) receives immediate reward \(r^{i}_{t+1}(s^{i}_{t+1})\) (step 4) and updates Q-value \(Q^{i}_{t+1}(s^{i}_{t},a^{i}_{t})\) using Equation (1) at time \(t+1\) (step 5) \begin{table}
\begin{tabular}{p{85.4 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Reference, Year** & **Foci** & **Approach** & **Multi-Agent Environment** \\ \hline [PERSON] et al. (2021) [11] & Theoretical & analysis of MADRL & Game Theory & \(\bullet\) & Collaborative \\  & & squaretees & & \(\bullet\) & Competitive \\ \hline [PERSON] (2021) [12] & Theoretical & analysis of MADRL approaches in terms of training schemes and the emergent patterns of agent behavior & Deep Learning & \(\bullet\) & Collaborative \\ \hline This survey & MADRL models and algorithms applied to various multi-agent environments & Deep Learning & \(\bullet\) & Collaborative \\  & & & & \(\bullet\) & Competitive \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Cont. 5). The steps are repeated as time goes by. The algorithm assumes a single agent in the operating environment, and so it optimizes the local performance regardless of the global performance.