# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 3 Multi-Agent Deep Reinforcement Learning
### Multi-Agent Deep Q-Network  
The multi-agent deep Q-network (MADQN) is based on DQN, and so it uses DNN to address the curse of dimensionality, and replay memory and target network to increase the convergence rate to the optimal joint action. Most importantly, MADQN extends the single-agent DQN approach. Multiple agents exchange knowledge among themselves in order to achieve global optimization in a collaborative manner, and so it addresses the moving target problem. Knowledge exchange enables agents to consider their own and neighboring agents' performances [32]. Algorithm 4 shows the MADQN algorithm. MADQN enables an agent \(i\in N\) to observe its state \(s^{i}_{t}\in S^{i}\) (step 3), exchange knowledge (steps 4 and 5), perform steps 5 to 13 in Algorithm 3, and update its Q-value \(Q^{i}_{t+1}(s^{i}_{t},a^{i}_{t})\) using the Q-function (step 9). The agent uses the exchanged knowledge (i.e., Q-value) to calculate the Q-value of each state-action pair as follows [33]:  
\[ \begin{split} Q^{i}_{t+1}(s^{i}_{t},a^{i}_{t})&=(1 -\alpha)Q^{i}_{t}(s^{i}_{t},a^{i}_{t};\theta^{i})+\alpha[r^{i}_{t+1}(s^{i}_{t+1 })\\ +\gamma\text{max}_{a\in A^{i}}Q^{i}_{t}(s^{i}_{t+1},a;\theta^{i -})-Q^{i}_{t}(s^{i}_{t},a^{i}_{t};\theta^{i})]\\ &\hskip 14.226378 pt+\sum_{j\in N}\eta^{i,j}Q^{i}_{t-1}(s^{j}_{t-1}, a^{j}_{t-1};\theta^{i})\end{split}  \tag{6}\]  
The traditional DQN approach and MADQN approaches have been applied in a multi-agent environment with centralized and distributed agents. The traditional DQNapproach can be embedded in the centralized agent to: (a) gather data, information, or observations from distributed agents; (b) learn from the gathered data and update Q-values to maximize the expected long-term system-wide rewards; and (c) send optimal actions to the distributed agents, which execute the optimal actions accordingly. For achieving scalability, DQN can be embedded in the distributed agents with two assumptions [34]: (a) each agent ignores the presence of the rest of the agents' actions and assumes that they are part of the operating environment; and (b) the changes of the rest of the agents' actions in response to each agent's actions being part of the dynamicity of the operating environment. Unfortunately, these assumptions can cause instability, particularly when the rest of the agents start learning and their actions become non-stationary, causing the agents to fail to converge on the optimal joint action. Hence, MADQN enables agents to exchange knowledge with each other and learn the best possible actions in order to achieve scalability (O.5). MADQN can be embedded in distributed agents so that each of them is aware of the rest of the agents. To enable this, agents exchange knowledge and use it to update their respective Q-values, \(Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i};\theta^{i})\), based on a weighted sum of delayed rewards or value functions from themselves and neighboring agents using Equation (6), where \(J^{i}\) is a set of agent \(i\)'s neighboring agents, and \(\eta^{i,j}\) represents the importance of neighboring agent \(j\) at agent \(i\). For instance, with \(\sum_{j}\eta^{i,j}/|J^{i}|\), the Q-value for neighboring agent \(j\) has an equal effect on an agent \(i\)'s Q-value \(Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i};\theta^{i})\). Therefore, the agents select their respective actions while ensuring that the global Q-value converges to a unique and optimal equilibrium. The agents may exchange knowledge via direct communication or through prediction. For instance, the agents create the models of the operating environment and neighboring agents to predict their Q-values. As an example of an extension to DRL, MADRL uses actor-critic networks [35; 36]. Each agent has unique actor and critic networks: (a) the critic network corrects the immediate reward using temporal difference; and (b) the actor network updates the Q-values using the temporal difference. Table 2 highlights the differences between RL methods and their convergence criteria.