# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s SADRL Approach for a Hierarchical Multi-Agent Environment  
In [76], [PERSON] et al. embedded SADRL in a centralized agent, which is the centralized controller, to coordinate the traffic phases of distributed agents, which are traffic lights (A.2), based on traffic conditions in order to manage the congestion level. The agents maximize their individual rewards in a competitive (X.2.1) manner. The key features are that: (a) distributed agents use neighboring agents' knowledge from the last time step rather than the current time step, which increases stability (O.1) and learning speed (O.2); and (b) the centralized agent forwards knowledge to distributed agents according to their roles and positions in the multi-agent environment, which ensures higher scalability (O.5). The proposed approach addresses the challenges of high dynamicity (C.2) due to the dynamic condition of the road traffic pattern, and high dimension (C.3) with a large number of state-action pairs characterizing the large numbe of vehicles at an intersection. At an intersection, the traffic light \(i\), which is a distributed agent, represents state \(s^{i}_{t}\) with a four-tuple information, including: (a) the cumulative delay of the first vehicle in each incoming lane; (b) the total number of vehicles along each incoming lane within a distance (e.g., 50 m) from an intersection; and (c) the historical Q-values. The action \(a^{i}_{t}\) represents a selected traffic phase. The reward (or cost) \(r^{i}_{t+1}(s^{i}_{t+1})\) represents the queue length of vehicles along each lane. The proposed approach extends the traditional MADRL approach with two main mechanisms. Firstly, each agent has an enhanced communication-learning mechanism with three main blocks to overcome the shortcomings of partial observations among distributed agents: (a) the information extract block extracts relevant information from high-dimensional raw data, and uses local states and Q-values of the distributed agents in the last time step rather than that in the current time step to increase robustness against delay; (b) the information exchange block has a centralized agent that learns the significance of the information of each distributed agent, and sends the information to the rest of the distributed agents; and (c) the policy learning block predicts the Q-values of the actions of each distributed agent based on the observations and information gathered by the information extract and exchange blocks. In short, this mechanism enables each distributed agent to use relevant information based on its significance. Secondly, an enhanced communication among agents. The hierarchical approach provides three main advantages: (a) provides the abstraction of complex problems; specifically, agent \(i\) learns from relevant information rather than high-dimensional raw data; (b) enables agent \(i\) to learn over temporal spans using knowledge from the last time step; and (c) provides the segregation of agents into smaller groups based on their capabilities (i.e., roles and positions), which naturally helps to decompose complex problems into smaller ones. The proposed scheme has been shown to reduce the congestion level (P.6). SADRL with Actors and Critics Applied to Centralized Entities in a Hierarchical Multi-Agent Environment  
DRL can be implemented using actor and critic networks, in which an agent \(i\) has: (a) a _critic network_ to calculate the temporal difference (TD)-error, which represents the quality of agent \(i\)'s current action, based on the states and actions of all distributed agents; and (b) a _actor network_ to update its Q-values or policy using temporal difference, rather than the immediate reward. The actor-critic approach has been shown to expedite learning.