# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 3 Multi-Agent Deep Reinforcement Learning
### Deep Reinforcement Learning  
DRL addresses the shortcomings of RL and MARL--particularly the curse of dimensionality, whereby storing knowledge in a tabular manner is unable to cater for large scale and complex problems--by using neural networks as function approximators to generalize and approximate the value function. Deep Q-network (DQN), which is a value-based method is a popular single-agent DRL approach [30]. Using DQN, a single agent: (a) addresses the curse of dimensionality using DNN to represent and store states, actions, and knowledge; and (b) ensures convergence and increases the convergence rate to the optimal joint action (or increases the learning speed) using distinguishing features, particularly replay memory and target network. This enables DQN to solve complex problems. For instance, in [31], DQN was embedded in a centralized agent to gather global information from distributed agents, learn, select optimal actions, and then send the selected actions back to the distributed agents that performed the selected actions. The MADQN approach, as shown in Figure 1, consists of multiple agents interacting with the operating environment and with each other. Each agent is based on the single-agent DQN approach. Hence, each agent in the multi-agent environment has: (a) a main network with a fully connected DNN to provide Q-value \(Q_{t}^{i}(s_{t}^{i},a_{t}^{i};\theta^{i})\); (b) a target network, which is a duplicate of the main network, to provide a stable target for learning; and (c) a replay memory to store the agent's experiences. The fully connected DNN has: (a) the input layer which consists a set of neurons to receive a state that consists of substates; (b) the hidden layer, and (c) the output layer which consists a set of neurons to provide the Q-value \(Q_{t}^{i}(s_{t}^{i},a_{t}^{i};\theta^{i})\) of each possible action \(a_{t}^{i}\in A^{i}\). Each link between neurons \(k^{i}\) and \(j^{i}\) has a weight \(w_{k}j^{i}\), and the output of neuron \(k^{i}\) is as follows:  
\[y_{k}^{i}=\varphi(\sum_{j=0}^{m}w_{kj}^{i},x_{j}^{i}) \tag{3}\]  
where \(\varphi\) is an activation function (e.g., sigmoid function); the input \(x_{j}^{i}\) is assigned a weight \(w_{kj}^{i}\) representing its importance compared to other inputs \((x_{0}^{i},\dots,x_{j}^{i},\dots,x_{m}^{i})\). DQN has two types of networks that use the fully connected neural network architecture, namely, the main network and the target network. The main network produces Q-values for all the possible actions \(A^{i}\) in its output layer, and the action \(a_{t}^{i}\) with the maximium Q-value is selected during exploitation. The target network, which is a duplicate of the main network every \(C\) steps, produces a stable target for learning. Algorithm 3 shows the DQN algorithm. There are two main phases. During the action selection phase, the agent \(i\) observes the current state \(s^{i}_{t}\) (step 3) and chooses an exploitation or exploration action \(a^{i}_{t}\) (step 5) at time \(t\). The agent \(i\) receives immediate reward \(r^{i}_{i+1}(s^{i}_{t+1})\) and next state \(s^{i}_{t+1}\) (step 6), and stores the transition (or experience) \((s^{i}_{t},a^{i}_{t},r^{i}_{t+1}(s^{i}_{t+1}),s^{i}_{t+1})\) in its replay memory \(D^{i}_{t}\) (step 7). During the training phase, the agent \(i\) selects a mini-batch of samples randomly from its replay memory \(D^{i}_{t}\) to train the main network (step 8). The agent \(i\) calculates a target \(y^{i}_{j}\) using a target Q-value \(Q^{i}_{t}(s^{i}_{t},a^{i}_{t},\theta^{i,-})\), which is generated using the target network \(\theta^{i,-}\), over several steps \(j=1,\ldots,N\) (step 10). The target network \(\theta^{i,-}\) is a duplicate of the main network \(\theta^{i}\). The target \(y^{i}_{j}\) is used to compute the loss \(L^{i}_{j}(\theta^{i})\) of the selected action \(a^{i}_{j}\); and the loss \(L^{i}_{j}(\theta^{i})\), which is expected to reduce as time goes by, is used to train the main network (step 11).