# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s MADRL Approach with Bootstrapping in a Multi-Agent Environment  
Similarly to [33, 36, 76, 79], in [83], [PERSON] et al. embedded MADRL in distributed agents, which are the traffic lights (A.2), to enhance their exploration strategy. A multi-agent environment is collaborative (X.2.2) in nature. The two key features are that: (a) distributed agents use bootstrapping, whereby the value function \(v_{\pi}^{i}(s_{t}^{i})\) is sampled per episode to increase learning speed (O.2); and (b) distributed agents share knowledge (i.e., rewards) with neighboring agents (X.1.1) to ensure stability (O.1). The proposed approach addresses the challenge of high dynamicity (C.2) due to the dynamic condition of the road traffic pattern.  
The proposed approach improves exploration by identifying uncertainties in Q-value estimation using bootstrapping. Bootstrapping enables an agent \(i\) to perform multiple actions and learn at the end of an episode rather than to learn right after taking an action. Figure 7 shows the bootstrapped DQN architecture that consists of \(k>10\) heads sharing a convolutional network. Bootstrapped DQN uses a masking distribution \(M\) over possible values to generate a mask \(m_{t}^{k}\), which is a binary vector of length \(k\) used to decide whether or not the Q-value of the \(k^{th}\) head and an experience should be updated during training at time step \(t\). Each head \(k\), which is a separate DQN trained based on different data slices, bootstraps Q-value functions for different actions in parallel. Each head \(k\) uses a DQN, whereby the Q-value function of the main network \(Q_{t}^{ik}(s_{t}^{i},a_{t}^{i};\theta^{i})\) is trained against its own target network \(Q_{t}^{ik}(s_{t}^{i},a_{t}^{i};\theta^{i,-})\). This allows the agent \(i\), which is a traffic light, to learn the posterior distribution over different Q-value functions \(Q_{t}^{ik}(s_{t}^{i},a_{t}^{i};\theta^{i})\) to minimize the loss function without replay memory data shared among the heads. During action selection, in each episode, the agent \(i\) observes state \(s_{t}^{i}\), samples a single optimal Q-value \(Q_{t}^{ik}(s_{t}^{i},a_{t}^{i};\theta^{i})\) per episode of interactions, takes the optimal actions \(a_{t}^{i}\) according to the sampled Q-value  
Figure 6: Architecture of cooperative MADQN.  
function throughout the episode, receives reward \(r^{i}_{t+1}(s^{i}_{t+1})\), and then uses the \(m^{k}_{t}\) mask to select whether or not to store the \((s^{i}_{t},a^{i}_{t},r^{i}_{t+1}(s^{i}_{t+1}),m^{k}_{t})\) experience in a replay buffer \(B^{k}\) (i.e., a distinct memory buffer for the \(k\)-head). The experience \((s^{i}_{t},a^{i}_{t},r^{i}_{t+1}(s^{i}_{t+1}),m^{k}_{t})\) is used to train the \(k\)-th head if it is activated with \(m^{k}_{t}[k-1]=1\).  
At an intersection, the traffic light \(i\) represents state \(s^{i}_{t}\) with the waiting time of all vehicles in all incoming lanes. The action \(a^{i}_{t}\) represents a selected traffic phase. The reward \(r^{i}_{t+1}(s^{i}_{t+1})\) represents the number of vehicles waiting at lanes with red signals. The proposed scheme has been shown to reduce the congestion level (P.6).