# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### SADRL Applied to Distributed Agents in a Multi-Agent Environment
#### 6.1.4 [PERSON]'s SADRL Approach in a Multi-Agent Environment  
In [73], [PERSON] et al. embedded SADRL in a service function chain (SFC), which is a chain of connected network services, to migrate virtual machines (VMs) to a data center based on the energy consumption and delay requirements of SFC users. The data center possesses virtual network functions (VNFs), including virtualized routers, firewalls, and network address translation services (A.5). The SFCs maximize the global reward that directs an SFC to a common target in a collaborative (X.2.2) manner. The proposed approach addresses the challenge of high dynamicity (C.2) due to the dynamic condition of the traffic.  
Each SFC \(i\) represents state \(s_{t}^{i}\) with a two-tuple information, including the amount of resources required from VNF and the amount of VNF resource remaining. The action \(a_{t}^{i}\) represents whether or not to migrate VMs to VNF, and if yes, which VNF to migrate to  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**MADRL Approaches** & **Description** & **Advantages** \\ \hline Actor–critic & \(\bullet\) & Agent has unique actor–critic networks, in which: (a) the actor network predicts Q-values; and (b) the critic network provides a stable target for learning & \(\bullet\) & Overcomes the shortcomings of separate centralized and distributed entities \\  & \(\bullet\) & Suitable for environments with centralized entity and distributed agents, in which: (a) the centralized entity gathers experiences from distributed agents, stores them in a replay memory, learns, and distributes the weights of the actor networks to distributed agents & \(\bullet\) & Achieving higher scalability (O.5) and stability (O.1), and a lower signaling overhead (O.4) and less complexity (O.3) \\ \hline MADRL with knowledge exchange & \(\bullet\) & Distributed agents exchange knowledge with neighboring agents (X.1.1), estimate Q-values, and select their own actions & \(\bullet\) & Provides a partial observation of the operating environment since the agents collaborate \\  & & \(\bullet\) & Achieving a higher learning speed (O.2) and stability (O.1) \\ \hline Bootstrapped MADRL & \(\bullet\) & Distributed agents perform multiple actions and learn at the end of an episode rather than right after taking an action & \(\bullet\) & Achieving a higher learning speed (O.2) \\  & \(\bullet\) & Suitable for distributed agents to share the critic network for learning and use the main network for action selection & \(\bullet\) & Achieving a higher learning speed (O.2) \\ \hline MADRL with action discovery & \(\bullet\) & Distributed agents explore, evaluate, and store explored actions for training & \(\bullet\) & Achieving a higher learning speed (O.2) \\  & \(\bullet\) & Distributed agents use a cost function to explore more actions when seeking optimal actions, and use potential functions to replace inappropriate actions with newly explored ones & \(\bullet\) & Achieving a higher stability (O.1) \\ \hline MADRL with concurrent learning & \(\bullet\) & Distributed agents with different decision-making capabilities select their own actions to achieve a common global goal, in which they receive joint states and rewards & \(\bullet\) & Achieving a higher stability (O.1) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Summary of MADRL extensions and their advantages.  
The reward (or cost) \(r_{t+1}^{i}(s_{t+1}^{i})\) represents the delay incurred by the nodes. The traditional SADRL approach has been shown to increase energy efficiency (P.4).