# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s SADRL Approach for a Hierarchical Multi-Agent Environment
For reducing complexity (O.3) and increasing learning speed (O.2), the centralized agent performs the complex learning task, rather than the distributed agents [35]. [PERSON]'s Hybrid SADRL Approach with Actors and Critics  
In [35], [PERSON] et al. embedded SADRL with actors and critics in the centralized agent, which is the base station shown in Figure 4. The agent maximizes individual rewards in a competitive (X.2.1) manner. The _centralized agent_ gathers experiences (including the channel information, such as channel quality and interference levels) from a predefined number of neighboring distributed agents, learns, and distributes learned knowledge (i.e., the weights of the actors) to distributed agents, which are the transmitter and receiver node pairs (or links) (X.3.1). Based on the knowledge given by the centralized agent, the distributed agents select their resources (i.e., the transmission opportunities characterized by the operating channel, the time of transmission, and the transmission power) for D2D communication so that they can communicate with each other directly without passing through the base station while meeting the SINR constraint (A.1). Node pairs use different operating channels to reduce interference, and to increase the reliability and capacity of both cellular communication (i.e., between the node pair and the cellular base station) and D2D communication if they are physically close with each other. The node pairs can reuse the same operating channels to improve spectral efficiency if they are far apart from other node pairs. The proposed approach addresses two challenges: (a) ultra-densification (C.1) caused by a large number of node pairs; and (b) high dynamicity (C.2) due to the dynamic condition of the operating channels. Consider a predefined set of all distributed agents \(N\) maintained by the centralized agent, and a predefined set of neighboring distributed agents \(I\in N\). Each D2D node pair  
Figure 4: The centralized entity gathers experiences from agents \(1\ldots i\ldots I\), stores them in its replay memory, and performs training. Shown in the figure is one of the distributed agents, namely, agent \(i\), that performs action selection. (or link) \(i\) represents state \(s^{i}_{t}\) with a four-tuple information, including: (a) the channel information of the link \(i\); (b) the channel information of the cellular link; (c) the interference level of the D2D link \(n\) at the previous time instant \(t-1\); and (d) the transmission opportunity of the D2D link \(n\) at the previous time instant \(t-1\). The action \(a^{i}_{t}\) represents a transmission opportunity at time \(t\). The reward \(r^{i}_{t+1}(s^{i}_{t+1})=\log(1+\xi^{i}_{t})\) represents the link rate of the D2D link \(i\), where \(\xi^{i}_{t}\) represents the SINR of the receiver of the D2D link \(i\) based on the Shannon capacity. The reward is applicable when the SINR of cellular nodes is higher than a predefined threshold; otherwise, the reward is a negative constant value. Each distributed agent receives the weights of the target actor \(\mu^{\prime}_{n}(s_{n}|\theta^{\mu^{\prime}}_{n})\) from the centralized entity, observes its state \(s^{n}_{t}\), selects action \(a^{i}_{n}\) independently, and receives reward \(r^{i}_{t+1}(s^{i}_{t+1})\). Subsequently, the agent \(i\) sends the \((s^{n}_{t},a^{n}_{t},r^{n}_{t+1}(s^{n}_{t+1}),s^{n}_{t+1})\) experience to the centralized entity for subsequent learning. The SADRL with actor and critic networks approach has been shown to increase accumulated reward (P.2) and throughput (P.3), and reduce outage probability (P.1). A similar approach has been applied in the literature [52]. In [52], [PERSON] et al., use this approach so that multiple distributed agents (i.e., node pairs) with different channel conditions can select their resources (i.e., transmission opportunities) for D2D communication (A.1), which helps them to achieve fairness (P.5) among the distributed agents and co-exist in a dynamic environment.