# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s SADRL Approach for a Hierarchical Multi-Agent Environment
In Figure 4, the centralized entity \(i\) gathers the \((s_{t}^{i},a_{t}^{i},r_{t+1}^{i}(s_{t+1}^{i}),s_{t+1}^{i})\) experience from distributed agents \(N=I\), stores it in a replay memory \(D^{i}\), learns, and distributes learned knowledge (i.e., the weights of the actors \(\mathbf{\mu}\)) to distributed agents. At each time step, the centralized entity \(i\) selects a random set of experiences from the replay memory \(D\) and updates four networks as follows:  
* _Actor network_\(\mathbf{\mu}=(\mu_{1}(s_{1}|\theta_{1}^{\mu}),\ldots,\mu_{I}(s_{1}|\theta_{I}^{\mu}))\) is parameterized by weight \(\mathbf{\theta}^{\mathbf{\mu}}=(\theta_{1}^{\mu},\ldots,\theta_{I}^{\mu})\). It updates the weight \(\theta_{i}^{\mu}\) of agent \(i\) using sampled policy gradient according to \(\
abla_{\theta_{i}^{\mu}}I_{i}=\mathbb{E}_{s,\mathbf{\mu}\sim D}[\
abla_{a_{i}}Q_ {i}(\mathbf{s},\mathbf{a}|\theta_{i}^{Q})]_{a_{i}=\mu_{i}(s_{i})}|\
abla_{\theta_{i}^{ \mu}}H_{i}(s_{i}|\theta_{i}^{\mu})\) where \(J_{i}=\mathbb{E}[R_{i}]\) and \(\mathbb{E}\) represents environment. * _Target actor network_\(\mathbf{\mu}^{\prime}=(\mu_{1}^{\prime}(s_{1}|\theta_{1}^{\mu^{\prime}}),\ldots, \mu_{I}^{\prime}(s_{1}|\theta_{I}^{\mu^{\prime}}))\) is a copy of the actor network that updates the weight \(\theta_{i}^{\mu^{\prime}}\leftarrow\tau\theta_{i}^{\mu}+(1-\tau)\theta_{i}^{ \mu^{\prime}}\) slowly (e.g., with a small \(\tau=0.01\) value) to ensure convergence. * _Critic network_\(\mathbf{Q}=(Q_{1}(\mathbf{s},\mathbf{a}|\theta_{1}^{Q}),\ldots,Q_{I}(\mathbf{s},\mathbf{a}|\theta_{ I}^{Q}))\) is parameterized by weight \(\mathbf{\theta}^{\mathbf{Q}}=(\theta_{1}^{Q},\ldots,\theta_{I}^{Q})\). It updates the weight \(\theta_{i}^{Q}\) of agent \(i\) by minimizing its loss \(Loss(\theta_{i}^{Q})=\mathbb{E}_{s,\mathbf{a},\mathbf{\mu}^{\prime},i\mathbf{s}^{\prime}}[ (Q_{i}(\mathbf{s},\mathbf{a}|\theta_{i}^{Q})-y_{i})^{2}]\) where \(y_{i}=r_{i}+\gamma Q_{i}^{\prime}(\mathbf{s}^{\prime},\mathbf{\mu}^{\prime}(\mathbf{s}^{ \prime}|\mathbf{\theta}^{\mathbf{\mu}^{\prime}})|\theta_{i}^{Q^{\prime}})\). * _Target critic network_\(\mathbf{Q}^{\prime}=(Q_{1}^{\prime}(\mathbf{s},\mathbf{a}|\theta_{1}^{Q^{\prime}}),\ldots, Q_{I}^{\prime}(\mathbf{s},\mathbf{a}|\theta_{I}^{Q^{\prime}}))\) is a copy of the critic network that updates the weight \(\theta_{i}^{Q^{\prime}}\leftarrow\tau\theta_{i}^{Q}+(1-\tau)\theta_{i}^{Q^{ \prime}}\) slowly. Algorithm 5 shows the SADRL with actors and critics approach embedded in the centralized entity. The critic uses the \((s_{i}^{i},a_{t}^{i},r_{t+1}^{i}(s_{t+1}^{i}),s_{t+1}^{i})\) experience from agent \(i\) to calculate the temporal difference \(\delta_{t}^{i}(s_{t}^{i},a_{t}^{i})\) of the state-action pair of agent \(i\). Subsequently, the temporal difference \(\delta_{t}^{i}(s_{t}^{i},a_{t}^{i})\) is used by the actor to update the Q-value \(Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i})\). ```
1:Procedure
2: /*Critic*/
3: Observe current state \(s_{t}^{i}\)
4: Select action \(a_{t}^{i\ast}\)
5: Receive immediate reward \(r_{t+1}^{i}(s_{t+1}^{i})\)
6: Calculate temporal difference \(\delta_{t}^{i}(s_{t}^{i},a_{t}^{i})\)
7: /*Actor*/
8: Update Q-value \(Q_{t+1}^{i}(s_{t}^{i},a_{t}^{i})\)
9:EndProcedure
```  
**Algorithm 5**Actor-Critic RL algorithm. SADRL with actors and critics can be applied to centralized entities (X.3.1) to receive experiences from distributed entities, and coordinate the operations of both centralized and distributed entities. This approach addresses the shortcomings of separate centralized and distributed entities. For increasing scalability (O.5), the centralized agent: (a) gathers data from some distributed agents, rather than all distributed agents, to reduce the signaling overhead (O.4) [35]; and (b) uses less data for learning to reduce the computational load [35]. For increasing stability (O.1), the centralized agent considers the distributed agents to take actions in a sequential manner [35]. For reducing the signaling overhead (O.4), the distributed agents do not exchange local information among themselves [35].