# Applications of Multi-Agent Deep Reinforcement Learning: Models and Algorithms
## 6 Application of Multi-Agent Deep Reinforcement Learning: State of the Art
### [PERSON]'s Fully Distributed MADRL with Hierarchical Characteristics  
In [84], [PERSON] et al. embedded MADRL in distributed agents, which are drones, to collect data from sensor nodes within fixed time periods based on the residual energy of drones (or unmanned aerial vehicles, UAVs) (A.1). The proposed approach extends the traditional MADRL approach using options in an extended MDP \((S,A,O,R,P)\). Due to the complexity of the action space of the drones, each drone selects an option (or a task), which represents a sequence of actions based on the option's policy for completing a task. This allows agents to learn at the general level (or the task level) rather than the specific level (or the action level). A multi-agent environment is competitive in nature (X.2.1) due to the goal of maximizing the individual rewards. Using the proposed approach, distributed agents: (a) decompose the complex problem into several sub-problems using hierarchical MADRL; (b) use options (i.e., a generalization of actions) to incorporate sequences of actions in order to ensure stability (O.1); and (c) have prior knowledge about the environment, which reduces complexity (O.3). The proposed approach addresses the challenge of ultra-densification (C.1) caused by a large number of sensor nodes in the multi-agent environment. The proposed model uses a fully distributed MADRL approach with options to obtain an optimal trajectory to sensor nodes. Sensor nodes are clustered, and each UAV agent communicates with neighboring agents via a UAV-to-UAV communication mechanism.  
As an example, the \"opening a door\" option consists of: (a) controlling (or twitch) the body muscles; (b) standing up; and (c) moving to the door. As another example, the \"going to school\" option consists of: (a) controlling the body muscles; (b) getting dressed; and (c) riding the bus. As for the \"collecting data within a fixed time period\" option, it consists of: (a) flying to sensor nodes and collecting data; (b) flying to a charging station and getting charged for a fixed time period; and (c) flying to the charging station when the mission is completed. Since more than a single action can be taken in an option, the state may change several times between two decision epochs. Specifically, each option in the option space \(o^{i}_{t}\in O\) has three components: (a) \(G\) represents the condition that initiates a task; (b) \(\pi\) represents a deterministic policy for an option \(o^{i}_{t}\); and (c) \(B\) represents the condition that terminates the task.  
At time instant \(t\), an agent \(i\) observes state \(s^{i}_{t}\in G\) and selects an option \(o^{i}_{t}\) based on the option learning policy \(\mu\). Then, the agent performs a sequence of actions in the selected option \(a^{i}_{t}\in\sigma^{i}_{t}\), and receives reward \(r^{i}_{t+1}(s^{i}_{t+1})\) when the selected option terminates at \(B\). Each drone \(i\) represents state \(s^{i}_{t}\) with three-tuple information: (a) the amount of data collected by sensor nodes; (b) its position in the terrain; and (c) its residual energy. The action \(a^{i}_{t}\) is either flying to a certain sensor node, getting charged, or collecting data. The reward \(r^{i}_{t+1}(s^{i}_{t+1})\) represents the maximum time that an agent \(i\) needs to finish its task (or for an option to terminate). Each agent \(i\) is punished when it runs out of battery. The proposed scheme has been shown to increase energy efficiency (P.4).