# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s MADRL Approach Using Target Updated with Neighboring Agents' States and Actions Information  
In [36], [PERSON] et al. embedded MADRL in distributed agents, which are the traffic lights (A.2). The MADRL approach enables a distributed agent to exchange state and action information with neighboring agents (X.1.1), estimate their policies, and use the estimation to select its own actions. This enables a distributed agent to perform learning that mimic the centralized learning approach, and select action based on local states. A multi-agent environment is collaborative (X.2.2) in nature. The MADRL approach has two main features: (a) enabling collaboration with neighboring distributed agents to provide a partial observation of the operating environment; and (b) addressing the curse of dimensionality due to the complexity of the joint actions of multiple neighboring agents. Using the proposed approach, distributed agents: (a) share knowledge with each other to increase the learning speed (O.2) and stability (O.1); and (b) learn in a distributed manner to increase scalability (O.5). The proposed approach addresses the challenges of high dynamicity (C.2) due to the dynamic condition of the road traffic pattern, and high dimension (C.3) with a large number of state-action pairs characterizing the large number vehicles at an intersection.  
At an intersection, the traffic light \(i\) represents state \(s_{t}^{i}\) with a five-tuple information, including: (a) the current traffic phase; (b) the queue length of vehicles in each lane; (c) the location of the vehicles at each lane; (d) the velocity of the vehicles; and (e) the number of pedestrians waiting to cross the intersection. The action \(a_{t}^{i}\) represents a selected traffic phase. The reward \(r_{t+1}^{i}(s_{t+1}^{i})\) is a weighted value based on: (a) the sum of the queue lengths of vehicles in all its lanes; (b) the total waiting time of vehicles in all its lanes; (c) the system delay; (d) the total number of vehicles crossing the intersection at time \(t\); (e) the changes of the blinking condition, which is an intermediate signal (e.g., a flashing red signal when the signal changes from red to green) in the current traffic phase; and (f) the total waiting time of pedestrians waiting to cross the intersection.  
Figure 5 shows the proposed MADRL architecture. Both main and target networks are based on LSTM to use historical states and actions during training and action selection. The target network of an agent \(i\) receives state and action information from neighboring agents, and use this information to estimate its own Q-value \(Q_{t}^{i,j}(s_{t}^{i,j},a_{t}^{i,j};\theta^{i,-})\), which is used to generate the target \(y_{jr}^{i}\) subsequently used to minimize the local loss function \(L^{i}(\theta^{i})\) and update the main network. In other words, each agent \(i\) uses global states and actions in its target network to estimate neighboring agents' policies. During action selection, each agent \(i\) selects actions based on local states and its own policy using the main network in an independent manner. Specifically, the proposed approach undergoes centralized learning and distributed execution. The proposed scheme has been shown to reduce the congestion level (P.6).  
A similar approach has been applied in [82]. In [82], [PERSON] et al. embedded this approach in distributed agents (i.e., base stations) that allocate resources (i.e., transmission opportunities) (A.1) to mobile users based on the SINR level and the capacity of the base station (i.e., the number of mobile users served by the base station), while addressing the challenge of high dynamicity (C.2) due to the dynamic channel conditions, in order to increase energy efficiency (P.4). Distributed agents share experience (i.e., states and actions) with neighboring distributed agents (X.1.1) to ensure stability (O.1) and increase the learning speed (O.2).