Multi-UAV Path Planning Based on DRL for Data Collection in UAV-Assisted IoT

Authors: lin  li
lei  wang
jiawang  hou
junjie  ma
yang  liu
Venue: IEEE_org_ICIST_2024_Conference
Published: 2024-08-13 16:31:35

Abstract: Due to their flexible mobility and stable network connectivity, unmanned aerial vehicles (UAVs) are increasingly being used as mobile data collectors, greatly expanding the spectrum of data collection. However, safe and effective path planning of multiple UAVs in dynamic environments and complex terrains is always challenging: frequent conflicts arise due to dense flight paths, incomplete observations due to dynamic environments, and risk of local optima from limited exploration. Therefore, we propose a UAV path planning approach based on deep reinforcement learning (DRL). Specifically, we employ the multi-agent proximal policy optimization (MAPPO) algorithm to maximize the data collection rate. We first model the multi-UAV path planning problem as a multi-agent partial observable Markov decision process (MA-POMDP) and integrate the traditional proximal policy optimization (PPO) algorithm into a multi-agent learning framework. Then, to improve the training efficiency of the algorithm and the decision-making capability of the UAVs, the strategy of combining centralized training with decentralized execution is used to enable the effective sharing of information and strategies among UAVs. Furthermore, to mitigate the issue of local optimal convergence during strategy learning due to insufficient exploration of various action plans and strategies in the environment, entropy regularization is introduced into the strategy objective function, enabling the agents to learn more comprehensive and effective path planning strategies. Simulation results validate that the algorithm maximizes total system throughput while adhering to constraints on flight duration, information age, and collision avoidance.

Main paper: 
 i. introduction: 
 The increasing popularity of wireless devices in the Internet of Things (IoT) has added complexity to the integration with future networks [1] , [2] . Unmanned aerial vehicles (UAVs), with their flexible mobility and stable network connectivity, are increasingly being used as mobile data collectors or temporary aerial hubs for IoT endpoints, greatly expanding the spectrum of data collection [3] . With the diversification and increasing complexity of application scenarios, path planning for UAVs has emerged as a critical technological challenge. In dynamic environments and complex terrains, effectively planning paths to avoid obstacles and ensure the successful completion of tasks has become a focal point of current research [4] . Nevertheless, conventional path planning methodologies struggle to accommodate environments characterized by high dynamics and unpredictability [5] . Furthermore, with the increasing number of UAVs, achieving collaborative operations among multiple UAVs to avoid path conflicts remains an urgent issue to be addressed. Therefore, an energy-efficient multi-UAV path planning algorithm is needed to optimize task completion and improve overall system performance.Objective optimization in the planning of UAV flight paths has been highlighted as an area of interest in recent studies. Traditional node-based algorithms are unable to find workable paths in dynamic environments due to the predetermined graph [6] . In contrast, sampling-based approaches provide more flexibility in the case of dynamic obstacles or environmental changes. The Extended Path Rapidly-Exploring Random Tree (EP-RRT) algorithm was introduced in [7] , enhancing the efficiency and convergence of path planning in extended corridor environments using heuristic sampling and the greedy heuristic of RRT-Connect. Although the above algorithms use random sampling of the environment to search for paths, generating feasible paths in narrow passages presents a challenge. To address this issue, [8] introduced a decentralized multi-agent path planning algorithm based on parameterized B-splines. Compared to mathematical-based algorithms, bio-inspired algorithms excel in global search and are better equipped to address the issue of local optima in path planning. A novel particle swarm optimization algorithm was proposed in [9] , which addressed the issue of local optima in complex environments through adaptive parameter adjustment and differential evolution operators, enhancing the quality and efficiency of path planning. Bio-inspired algorithms are developed using the structures and behavior found in natural biological systems, while reinforcement learning algorithms focus on the learning process of UAVs through environmental interactions [10] . A Q-learning-based path planning algorithm was proposed in [11] , introducing a shortest distance prioritization policy and a grid-graph-based method, to address the efficient path planning problem for unmanned aerial vehicles (UAVs). Further, [12] introduced a novel reinforcement learning based approach to handle the problem of collision avoidance and optimal trajectory-planning in the context of UAVs in communication networks. Although the aforementioned algorithms effectively address the problem of UAV path planning, demonstrating adaptability to complex environments and flexible decisionmaking learning, they require a substantial amount of interaction experience for training, resulting in challenges such as lengthy training times, low sample efficiency, and susceptibility to local optima.This paper presents a deep reinforcement learning (DRL)based UAV path planning method. Specifically, we utilize the multi-agent proximal policy optimization (MAPPO) algorithm within an integrated sensing and communication (ISAC)assisted UAV framework. This innovation effectively addresses challenges faced by existing path planning algorithms, such as the complexity of multi-UAV collaboration, the unpredictability of dynamic environmental changes, the limitations of incomplete observations, and the issue of getting stuck in local optima. To overcome the constraints of UAV detection ranges and environmental observation incompleteness, we model the multi-UAV path planning problem as a multi-agent partially observable Markov decision process (MA-POMDP) and integrate the proximal policy optimization (PPO) algorithm into a multi-agent learning framework for multi-UAV scenarios, accommodating the complex dynamics of multi-UAV operations. Then, to enhance information sharing and strategy coordination among UAVs while increasing their operational flexibility and robustness, we use a strategy that combines centralized training and decentralized execution and utilize the spectrum reuse capability of ISAC to enable UAVs to autonomously select radar and communication modes, further improving the efficiency and decision-making capability of path planning. Moreover, to address the problem of converging to local optima during strategy learning due to insufficient exploration of various action plans and strategies in the environment, entropy regularization is introduced into the strategy objective function, enabling UAVs to learn path planning strategies in a more comprehensive and effective manner.The remainder of this study is outlined as follows. Section II delineates the system architecture. Section III introduces a deep reinforcement learning (DRL)-based algorithm for integrated communication-sensing scheduling and UAV trajectory optimization. Section IV details the validation of the proposed algorithm via simulation studies. The paper concludes with Section V.In this paper, we explore a system comprised of multiple UAVs equipped with ISAC, operating within a M × M ∈ UAVs operate at a fixed height H, occupying a single cell within this grid. Their movements are carefully regulated to prevent collisions with any obstacles. The movement actions of the i-th UAV are a uav,i (t) ∈ ãuav (u i (t)),EQUATIONwhere ãuav (u i (t)) is a set of feasible actions that depend on the position of the UAV i . The UAV is limited to moving at a horizontal velocity V uav or standing still, i.e., the velocity of UAV i at time slot t is v i (t) ∈ {0, V uav }, t ∈ [0, T ], T ∈ N is the end time of the UAV's data collection task. The position of the entity changes in accordance with the following motion model:EQUATIONThe operational state ϕ i (t) ∈ {0, 1} of UAV i , either stationary or in motion:EQUATIONThe battery energy level b i (t) ∈ N of UAV is defined as,EQUATIONThe battery charge of the UAV b i (t) is initialized at b(0) ∈ N and diminishes by one unit with each movement step.UAV i facilitates communication with sensor nodes employing a straightforward time division multiple access (TDMA) method. In this method, during each communication slot n ∈ [0, N ], every sensor node k ∈ [1, K] selects the data with the highest SNR i,k (n) for uploading, based on a scheduling algorithm. The TDMA constraint on scheduling variable qEQUATIONThe achievable throughput of the i-th UAV in a task slot t is the sum of the achievable rates of K sensor nodes duringEQUATIONwhere R i,k (n) is the information rate of the k-th device at communication slot n.In the UAV flight environment, we consider two environmental features e = {e weather , e object }, where e weather represents weather conditions and e object indicates the presence of moving objects near the UAV. For simplicity in data representation, both attributes are quantified on a binary scale, where {e weather , e object } ∈ [0, 1]. A value of zero is indicative of optimal safety conditions, whereas higher values reflect increasing risks. This study applies a Bernoulli distribution to characterize the probability of an individual UAV encountering high-risk events X at continuous time intervals. This model includes a probability parameter α exp(eβ), where α is kept fixed and β is a vector of the coefficients that should be estimated to forecast risk properly [13] .We also take into account the minimization of the age of information (AoI) of data packets for the UAV to perform communication and perception mode switches quickly based on observed environmental features and also change operational states. Since the maximum size of the packet data queue is L, the age of the packets l in the queue is given by A l , such that together they form the age vector A ∈ R L . Furthermore, the corresponding emergency category of each data packet isC l ∈ {1, 2, ..., M }, such that C ∈ R L . The vector Λ ∈ R Mrepresents the AoI for each emergency category.The primary goal of the UAV path planning challenge is to optimize data collection efficiency by maximizing throughput, while ensuring compliance with several key constraints: compliance with a prespecified maximum flight duration, obstacle avoidance along the entire flight path and the safe landing within a certain designated area. This optimization problem is defined through the equation outlined below:EQUATIONEQUATION(7a) ensures that active UAVs avoid collisions with each other. (7b) mandates that UAVs steer clear of collisions with high structures. (7c) restricts the operating time of the UAV, forcing it to safely land within the landing area before the battery is depleted. (7d) stipulates that the starting position of the UAV is at the takeoff/landing area, with a takeoff height of h, while (7e) ensures that the UAV starts in a valid operational state. (7f) is TDMA constraint.Since the actions of the agents are based solely on their relative positions to features, such as their distance to devices, this paper employs UAV-centric global map mapping processing, which significantly improves learning efficiency. Then the map data can be directly provided to UAVs, and the input space is defined as:EQUATIONwherep(t) = [x(t), y(t)] T ∈ R 2 is the UAV's ground- projected position, I ∈ B M ×M ×3 is the physical environment map in the Boolean field B ∈ {0, 1}, D ∈ R K×2 is the two-dimensional coordinates of K IoT devices, D ∈ R K×2is the available data for each device, and b t ∈ N represents the remaining flight time of UAVs. For the purpose of UAVcentric global map processing, the map is enlarged to a dimension of (2M -1) × (2M -1) to ensure UAVs can independently observe the entire map regardless of its position. This expansion centers the map around the UAV's current location, creating the defined input space,EQUATIONIn an ISAC-assisted multi-UAV framework, we define the combined challenge of communication-sensing scheduling and UAVs path optimization as a partial observable Markov decision process (POMDP). To tackle this, we utilize DRL techniques that approximate the optimal control strategies for UAVs, despite the lack of initial knowledge about the complex characteristics of wireless channels in densely populated urban settings.1) State space: To enhance carrier sensing capabilities, the local state information for each agent has been expanded to include t last and t idle . Here, t last denotes the number of time steps since the joint radio communication (JRC) system is last utilized to maintain the channel in an idle state, and t idle indicates the duration in time steps that the channel has remained unoccupied. Another approach to carrier sensing involves the use of channel state c, which, in a multi-agent environment, signifies whether a device user is granted priority access to the channel. If a device user successfully transmits data at time step t -1, this user alone is awarded priority (c = 1) for the subsequent time step t. Each agent is restricted to observing only its immediate state and the local environmental conditions, and the state space available to each agent is structured as follows:EQUATIONThe system's state space is composed by aggregating the observation spaces of each of the I agents: 2) UAV action space: Agents are allowed to cooperate by foregoing their access to the communication channel, thus broadening the range of possible actions to include a nonoperational action, A i = {nop,a uav , a r , a (1) , a (2) , ..., a (M ) }, (12) where the subscript i is the index of the given agent.3) Reward: At each time step i, the reward structure for the agent is designed as a weighted sum that promotes efforts to minimize the age of data packets in the queue while also aiming to enhance throughput. r i (t) = w i,age r i,age (t) + w i,over r i,over (t) + w i,rad r i,rad (t) + w i,data r i,data (t) + w i,sc r i,sc (t) + w i,mov r i,mov (t) + w i,crash r i,crash (t), (13) where w i,age , w i,over , w i,rad , w i,data , w i,sc , w i,mov and w i,crash are weights. r i,age encourages agents to minimize AoI as much as possible, encourages agents to send packets before queue overflow occurs, r i,rad encourages agents to perform radar scans when environmental conditions are unfavorable, r i,data is the reward for data collection based on the throughput achieved during the current period, r i,mov is the penalty for UAVs continuing to move without completing tasks, and r i,crash is the penalty for UAVs failing to safely land in the landing zone when the remaining flight time is zero.Upon evaluating the data transmission and expiry within the current time step, the penalty for exceeding the queue's capacity is calculated based on the number of new data packets that cannot be accommodated. This penalty scales in end for 13: end for direct proportion to the excess packets, reflecting the system's inability to manage the overflow effectively, ri,over(t) = min(0, (EQUATION)Whenever a high-risk event X occurs, the number of adverse environmental characteristics in r i,rad (t) and e is proportional:EQUATIONIn multi-agent scenarios characterized by complexity, traditional Q-learning applied individually to agents typically results in inadequate performance [14] . To address the challenges posed by partially observable Markov decision processes, we propose a MAPPO algorithm. This algorithm leverages the strengths of policy gradient techniques, which are notably advantageous because they do not require knowledge of the state transition probabilities P (s t+1 | s t , a t ). This freedom from the Markov assumptions of the preceding states is a significant methodological advantage. Although as a rule agent-based systems utilize inter-agent communication or centralized frameworks, our method introduces a decentralized multi-agent PPO algorithm. This innovation is aimed at improving the effectiveness without the necessity to have a centralized control or direct interaction among the agents.In a multi-agent setting with I agents, we describe π = {π 1 , π 2 , . . . , π I } as the set of policies, parameterized by Θ = {Θ 1 , Θ 2 , . . . , Θ I }. For each iteration of the algorithm, the policy of each agent can be trained independently. To improve the knowledge sharing among the agents, we suggest the construction of sequence of policy models, represented by I -with I -< I. Each policy π i -is assigned to a subset of agents in the environment. Therefore, the objective function of the policy function is as follows:EQUATIONwhere (at|ot) is the ratio of the policy π i - change between the previous and current iterations.δ t (Θ i -) = π i -(at|ot) π i -,oldÂπ i - i (t) represents the estimated advantage value:EQUATION)The objective function of the parameterized value function Θ i -is:EQUATIONwherey i (t) = r i (t)+γ V π i - i (s(t+1))is the target value. While this framework necessitates the sharing of policy parameters among the agents during the training phase, it supports autonomous decision-making by each agent when operational.In policy-gradient learning frameworks, the exploration of new policies is performed by sampling actions from the recent modifications of the stochastic policy. The smell of diminishing randomness of policy selection in the training stage usually results in poor exploration that confines the system to local optimal solutions. To overcome this limitation and ensure a more comprehensive exploration, this paper introduces an entropy regularization component into the objective function. This modification increases the exploration power of the agents that prevents the local optima and enhances the overall system performance,EQUATIONwhere k 1 and k 2 are coefficients, and S[π Θ i -](o i ) represents the policy entropy of π Θ i -given input observation o i . The MAPPO-based algorithm for ISAC-assisted UAVs path planning is illustrated in Algorithm 1.IV. SIMULATION ANALYSIS In this experiment, the binary exponential backoff (BEB) algorithm, dueling double deep Q-network (D3QN) algorithm, and advantage actor-critic (A2C) algorithm are used as comparative algorithms. The system parameters are set as shown in Table 1 . The experimental setup involves UAVs tasked with data collection within a 320×320 grid, initiated with a total available flight duration of T steps. For each UAV action, either movement or hovering, there is a decrement of one step in the remaining flight time. Regulatory compliance mandates that the UAV cannot operate over tall buildings or beyond the confines of the designated grid. Each operational slot t ∈ [0, T ] encompasses κ = 4 communication slots n ∈ [0, N ]. To enhance data freshness, the system's threshold for maximum age has been lowered to A max = 2, preventing the aging of data packets in the queue. The hyperparameter settings for the MAPPO network are shown in Table 2 . The training regimen is structured around 400 time steps per episode, with a total of 2500 episodes conducted. To validate the stability and reproducibility of the experimental outcomes, each parameter configuration is rigorously tested using multiple independent trials, each initiated with a unique random seed. Fig. 2 demonstrates the impact of the number of UAVs on the performance of different algorithms. We can observe that the performance of each algorithm decreases as the number of UAVs increases. First, as the number of UAVs increases, the frequency of radar mode switching increases accordingly, which raises the risk of being penalized. As shown in Fig. 2(b) , this leads to a decrease in radar-related rewards. Additionally, more UAVs in the system increase the competition for a limited time step to transmit data or operate the radar. And the agents frequently switch radar modes, this leads to a decrease in communication performance and ultimately to a decrease in the overall reward, as shown in Fig. 2 (a). However, as the increasing number of UAVs, the MAPPO method maintains optimal performance and better balances the increase in radar operations with the need for normal transmission of data streams. Therefore, the MAPPO approach enables agents to learn effective action strategies for the current environmental conditions with minimal prior knowledge of the environment. Fig. 3 analyzes the impact of the number of IoT devices on the average data collection rate for successful UAV landings. Due to the random placement of IoT devices in unoccupied areas of the map, an increase in the number of devices results in more complex trajectory requirements and performance degradation. From Fig. 3 , it is observed that when devices are less than 5, the proposed algorithm does not achieve the optimal data collection rate. However, when devices reach or exceed 5, the performance of the proposed algorithm remains optimal. This indicates that the proposed algorithm has good adaptability to an increasing number of devices.Fig. 4 shows the effect of the initial amount of data per device on the overall data collection rate. We find that as the number of UAVs increases, the data collection rate increases for all algorithms. However, beyond 10-12 data units, increasing the initial amount of data per device leads to a decrease in the data collection rate. This is because the UAVs are limited by a maximum flight time and are prioritized to land before exceeding this limit, thus forgoing some data collection. In addition, we note that the data collection rate of the proposed algorithm decreases at a slower rate as the initial amount of data per device increases. This indicates that the proposed algorithm can collect more device data while satisfying the need to land before the end of the maximum flight time. This is attributed to the fact that the algorithm employs entropy regularization, which enhances exploration and allows for a more effective action selection strategy in response to an increase in the amount of initial data per device.Fig. 5 illustrates the relationship between the maximum allowable flight time and the data collection rate. It is evident that an increase in the maximum flight time correlates with an improvement in the rate of data collection. Among the four different algorithms, the data collection rate of the MAPPO algorithm is the highest. This is because the MAPPO algorithm can more effectively utilize each data sample, meaning that within the same timeframe, MAPPO can learn more environmental information, thus collecting more data. However, as the bulk of the data is secured, this effect diminishes, leading UAVs to focus more on minimizing total flight time and ensuring a safe landing rather than gathering the residual data.Fig. 3, 4 , and 5 illustrate that as scenario parameters increase, particularly at higher values, the MAPPO algorithm we propose consistently achieves a higher data collection rate compared to other algorithms. The BEB algorithm's random action selection fails to adopt effective strategies in response to environmental parameter changes. Additionally, an expansion of the UAV's action space with increasing scenario parameters leads to a decline in the performance of value-based D3QN algorithms. Although A2C is a policy gradient method, its high variance and sensitivity to non-linear gradients limit its effectiveness. Our proposed MAPPO algorithm constrains the magnitude of policy updates by estimating the policy gradient and incorporates an entropy regularization term into the objective function to enhance the agent's exploration capabilities. Consequently, the algorithm enables the agent to adapt well to changes in scenario parameters, thereby making effective action selection strategies in response to current parameter variations.The range of values for random scenario parameters is as follows: the number of deployed UAVs I ∈ {2, 3}., the number of IoT sensors K ∈ [310], the amount of data to be collected by each device D k ∈ [5.020.0], and the maximum flight time steps. The positions of IoT devices are randomly distributed throughout the entire unoccupied map space. In Fig. 6 , other scenario parameters remain constant, and two scenario instances, labeled 6(a), and 6(b), were selected to illustrate how path planning adapts to the increase in the number of UAVs. In Fig. 6 (a), two UAVs are deployed. Due to the maximum flight time limit, they ignore the red, purple, and brown devices, collect data from other devices, and land in the bottom-right landing area, achieving a collection rate of 77.3%. In Fig. 6 (b), three UAVs are deployed, two from the top-left and one from the bottom-right, collecting data from all devices with a collection rate of 100%. The three UAVs evenly distribute data collection tasks, achieve comprehensive data collection, avoid tall buildings, and land along efficient trajectories promptly.In this paper, in order to address the challenges of limited UAV detection range and incomplete environmental observations, we model the multi-UAV path planning problem as an MA-POMDP and integrate the PPO algorithm into a multi-agent learning framework, proposing a MAPPO-based algorithm to enhance data collection rates. To improve the training efficiency of the algorithm and the decision-making capabilities of the UAVs, the strategy of combining centralized training with decentralized execution is used to enable the effective sharing of information and strategies among UAVs. Additionally, to mitigate the problem of converging on local optima due to insufficient exploration during strategy learning, entropy regularization is introduced to the strategy objective function. Simulation results show that compared to competitive algorithms, the proposed algorithm can significantly improve the throughput of the system, thereby enhancing data collection rates.This study is designed to maximize system throughput, and while UAV energy consumption is considered under constraints, it has not been quantitatively analyzed. Considering the limited battery life and payload capacity of UAVs, it is crucial to tackle the issue of reducing overall energy consumption in UAV-assisted IoT systems to extend their operational lifespan. Therefore, in future research, multi-objective optimization schemes for throughput and energy efficiency need to be designed to ensure that the system is better adapted to practical applications. 
 
 Appendix:  



Reviews: review: title: title: This paper proposed a UAV path planning approach based on deep reinforcement learning (DRL). Specifically, adopting the multi-agent proximal policy optimization (MAPPO) algorithm to ensure efficient UAV path planning, thereby maximizing the data collection rate. The topic of this paper is interesting. Below is a list of comments that should be taken into account further when revising the paper.
main_review: main_review: 1. The contribution of this article should be compared with previous literature, and the basic technical difficulties of this article should be listed? And what methods should be used to solve this problem, emphasizing novelty and technological contribution.
2. In the experimental results section, after separately explaining the four different algorithms, they should be compared uniformly to draw an overall conclusion.
3. In the conclusion section, all strategies should be compared. Finally, a MAPPO-based algorithm is the most effective feature extraction component to improve the model’s expressive ability to make the best model performance. Meanwhile, please elaborate on the future plans.
title: title: This article is well written and acceptable.
main_review: main_review: 1.Authors are required to review manuscript templates and make corrections. 
2.There are many grammatical and typographical errors in the manuscript. Please check the full text carefully and correct them. 
3.The formula in the manuscript is too small, please change it.
4.Based on the proposed idea and obtained results in this paper, the authors should be able to present some more descriptions in conclusion part, for example, further study direction.
title: title: This article is quite fascinating and of high quality.
main_review: main_review: This article, " Multi-UAV Path Planning Based on DRL for Data Collection in UAV-Assisted IoT," proposes a UAV path planning method based on deep reinforcement learning (DRL). Firstly, the multi-agent near-end Strategy Optimization (MAPPO) algorithm is used to ensure effective path planning for UAVs. Finally, a strategy combining centralized training and decentralized execution is adopted to enable UAVs to share information and strategies for path planning collaboration. The article has clear logic and organization, but there are still some problems. My specific feedback is as follows :1) In the abstract part, the author should briefly discuss the problem in the first two sentences. 2) In the introduction, the author has insufficient background content for the study of UAV path planning. 3) What are the advantages of combining multiple algorithms in the research?