# Network Resource Allocation Strategy Based on UAV Cooperative Edge Computing

Authors: Shuo Wang; Ning Kong
Journal: Journal of Robotics
 (2022)

---

## Abstract

Aiming at the problem that fixed mobile edge computing (MEC) server is difficult to meet the needs of mobile users and temporary computing services, this study proposes a network resource allocation strategy based on unmanned aerial vehicle (UAV) cooperative edge computing. First, a UAV-aided MEC scene is designed, and a single UAV with an MEC server is used to provide auxiliary computing services for ground multiusers. Then, an optimization model aiming at total system delay is constructed by considering the system communication model and calculation model. Finally, Deep Q-Network is used to solve the optimization problem to obtain the best resource allocation scheme. Based on the experimental platform, the proposed strategy is demonstrated and analyzed. The results show that when the number of user equipment is 40, the total delay is about 33s, which is 35.29%, 31.25%, and 15.38% lower than other comparison strategies and effectively reduces the computing delay of users.


---

## Body

## 1. Introduction

In recent years, with the development of 5G mobile communication technology and the Internet of Things (IoT), smart mobile equipment have shown an explosive growth [1–3]. In the context of cloud computing, MEC is considered to be a technical key to improving the computing efficiency of mobile edge equipment [4]. In MEC systems, task offloading is the key for mobile equipment to support resource-intensive applications by executing on edge cloud resources [5, 6]. However, computing servers are usually deployed in fixed base stations. In practical scenarios, fixed base stations cannot meet dynamic services such as user mobility, base station damage, and temporary hotspot areas. It is particularly significant to meet the dynamic communication and multiequipment access requirements.According to the existing research, computing servers can be deployed on an unmanned aerial vehicle (UAV) to meet the communication requirements. UAV themselves have the advantages of low cost and high mobility. In UAV-assisted MEC networks, mobile equipment can offload tasks to UAV with high computing power and flexible connectivity at the network edge [7, 8]. This method utilizes the flexibility of UAV and better channel gain to offload computing tasks for users, which can not only save computing delay and user energy but also reduce the traffic load on fixed cloud servers [9]. Reference [10] proposed a two-stage joint hovering altitude and power control solution for the resource allocation problem in UAV networks considering the inevitable cross-tier interference from space-air-ground heterogeneous networks.At present, significant progress has been made on the research of computing task offloading. For example, reference [11] proposed a collaborative service deployment and application allocation algorithm to achieve the final edge service policy deployment. The minimum energy consumption was obtained through the minimum resource ratio increasing algorithm, and computing tasks are redistributed in combination with the load balancing algorithm to balance the computing load. However, its overall computational efficiency needs to be improved. Reference [12] proposed a computing framework for coordinating terminals, edge nodes, and cloud centers based on the pipeline offloading scheme. According to the computing and communication capabilities of the entire network, they reasonably allocated computing-intensive tasks to specific terminals or clouds, effectively improving computing efficiency. Reference [13] proposed a vehicle-assisted computational offloading architecture for UAVs. The proposed framework used vehicle-assisted computing offload for UAV computing tasks and network resource optimization, which has commonality with the research topic. Reference [14] introduced agents into computing task offloading and proposed a UAV-MEC (UMEC) agent-enabled computing task offloading framework to help users, UAV, and edge clouds perform computing task offloading. Reference [15] proposed a computational offloading scheme to minimize the time and energy consumption of computational task costs. The scheme reduced the execution cost by coordinating the allocation of computing resources between mobile equipment and edge servers. However, many iterations were needed to find the optimal solution. Reference [16] designed a multiround iterative auction algorithm based on auction theory. But the overall performance of the algorithm needs to be further optimized. Reference [17] proposed an optimized auction-based incentive mechanism. The mechanism can optimize long-term system welfare by operating in an online fashion. However, it does not have good scalability for resources that cannot be covered by network hardware.With the continuous development of computer technology, intelligent algorithms such as machine learning are continuously applied to edge computing. As in reference [18], a heuristic-based algorithm with a low time cost is proposed. Compared with the existing centralized resource allocation and decision-making algorithms, this scheme acquired a higher number of successfully offloaded tasks in different scenarios, but the offloading of computing tasks in complex situations cannot achieve collaborative optimization. Reference [19] proposed a blockchain-driven collaborative framework for MEC. Reference [20] proposed a caching mechanism based on Q-learning to reduce the backhaul traffic load and transmission delay from the cloud. Reference [21] proposed a collaborative computing framework based on deep neural networks. The experimental results indicate that the proposed method has batter effectiveness than the traditional methods. The above methods based on machine learning not only consider the efficiency and timeliness of MEC network resource allocation to a certain extent but also do not have good scalability. Therefore, the offloading strategy based on UAV has been further researched and developed. Reference [22] proposed a deep reinforcement learning-based MEC UAV-assisted computing offloading. The total cost minimization was taken as the objective function. Although this method expands the wireless network to a certain extent, it needs to be further optimized for efficient offloading of computing tasks and reasonable allocation of network resources in complex IoT.Aiming at the problem that it is difficult for fixed MEC servers to meet the computing needs of mobile users and computing efficiency of existing UMEC, a network resource allocation strategy based on UAV collaborative edge computing was proposed. Compared with the traditional network resource optimization allocation strategies, the innovations of this study are as follows:(1)
For the low-latency service guarantee for users, the UAV carrying MEC server is designed to improve the auxiliary edge computing system model for users. Taking the total system delay as the optimization goal, the transmission delay of user calculation is greatly shortened.(2)
Due to the huge amount of data and high real-time requirements of the 5G system, the proposed strategy uses Deep Q-Network (DQN) to solve the optimal resource allocation scheme, which improves the analysis efficiency of the network state.


## 2. System Model and Optimization Goal

### 2.1. System Model

The scenario of human-machine-assisted MEC system uplink communication is studied, as shown in Figure1, that is, UAVs provide auxiliary computing services for multiple ground user equipment. Only the scenario where a single rotor UAV provides services to N user equipment on the ground is considered.Figure 1
System model.For the convenience of theoretical analysis, a three-dimensional Cartesian coordinate system is considered, whereLn=xn,yn represents the position of n user equipment on the ground. The drone flies in a circle within a specified range and provides auxiliary computing services to ground user equipment. Among them, UAV is at a fixed height and the initial position of UAV that is denoted as d1, and the final position is denoted as dK. Each ground user equipment offloads some of computing tasks to UAV and the rest is locally computed.It is assumed that the total execution time is denotedT, and it can be equally divided into K slots and denoted t=1,2,…,K, where τ=T/K, where τ refers to the length of each slot. In the t slot, the position of drone is dt=xt,yt. Assuming that the maximum flight speed of UAV is Vmax, the trajectory constraints of UAV are as follows:(1)d1=dK,dt+1−dtτ≤Vmax,where formula (1) indicates that the flight speed of the UAV cannot exceed its maximum flight speed. Since UAV flight energy consumption is related to UAV flight speed, UAV flight energy consumption expression is as follows:(2)Eflyt=ηvt2,where vt=dt+1−dt/τ represents the flight speed of UAV in the t time slot; η represents the proportional factor of energy consumption and flight speed.


### 2.2. Communication Model

Assuming that the wireless channel between UAV and each user equipment is a line-of-sight (LOS) channel, the channel power gain between UAV and user equipmentn can be expressed as follows:(3)hnt=β0dn−2t=β0H2+dt−Ln2,where β0 represents the channel power gain per unit distance; dn−2t represents the horizontal distance between UAV and user equipment n in the t gap; and ⋅ represents Euclidean norm.


### 2.3. Computational Model

#### 2.3.1. Local Computing

When user equipmentn selects the local mode, all computing tasks will be locally performed, and the computing delay can be expressed by the following formula:(4)Tnloc=CnDnfn,n∈N,where fn is the computing capability of user equipment n (number of cycles per second); Cn represents the number of cycles required to calculate 1 bit; and Dn is the size of input data.


#### 2.3.2. Offload to UAV for Calculation

The UAV carrying MEC server periodically flies at a fixed altitude above the ground user equipment. The user equipment that selects MEC mode can offload tasks to UAV through time-division multiple access (TDMA) [23, 24]. The set of ground users is set to be offloaded to MEC as N′, and UAV can choose to relay some tasks to access point (AP) for calculation. Therefore, it can be assumed that the ratio of user equipment n that selects the relay to the ground base station in the t time slot is ζnt,n∈N′,t∈K; then, the task that needs to be calculated by UAV in the t time slot is 1−ζntDnt.(1)
The transmission speed of user equipmentn in time slot t:(5)vnut=B1log21+Pnhntσ2,n∈N′,whereB1 is the bandwidth between UAV and user n; Pn is the maximum transmission power of the user n, and the user equipment uses the maximum power transmission; and hnt is t time slot channel gain between the user n and UAV communication.(2)
Transmission delay of user equipmentn: assuming that the bits offloaded by the user n in time slot t are Dnt, so the user n needs to offload all bits in K time slots, and the constraints are expressed as follows:(6)∑n=1NDnt=Dn.The total transmission delay of the user equipment is expressed as follows:(7)Tnmec=∑t=1KDntvnt.(3)
The calculation delay of UAV: according to the relay ratio, the task calculated by UAV for user equipmentn in the t time slot and then the calculation delay of user equipment in one time slot in MEC mode can be obtained, and finally, the calculation task of UAV for a single user equipment can be obtained. The total delay is as follows:(8)Tku=∑t=1K1−ζntDntCnfc,wherefc represents the computing power of MEC server.


#### 2.3.3. Offload to Base Station on the Ground for Calculation

UAVs can choose to relay part of computing tasks to AP computing. The proportion of UAVs choosing relays in time slott is ζnt∈0,1,n∈N′. Since the ground base station can have multiple MEC servers built in, the calculation delay of ground base station is considered to be ignored. At the same time, the result of the calculation task is usually very small, and it is considered to ignore the transmission delay of calculation result. Thus, the delay of UAV relaying tasks to the computing part of the ground base station only includes the transmission delay of the UAV.The flying height of the UAV is relatively high from the ground base station and has a good line-of-sight link. It is considered that the communication between the UAV and ground base station adopts LOS channel. The transmission rate of UAV relaying the task to AP in the time slot is calculated as follows:(9)vnU2Et=B2log21+Puhutσ2,where B2 is the bandwidth between UAV and AP; Pu is the transmission power of UAV; and hut is the communication channel between UAV and AP.The transmission delay from UAV relay to ground AP can be expressed as follows:(10)TnU2E=∑t=1KζntDntvnU2Et.Assuming that UAV can calculate and transmit part of computing tasks to the ground AP at the same time, the total calculation delay offloaded to UAV is as follows:(11)Tnmax=maxTnU2E,Tnu.When the user equipment selects MEC mode, the total delay of offloading computing tasks to UAV is as follows:(12)TnΩ=Tnmec+Tnmax.


### 2.4. Model Constraint Description

The system goal is to minimize the total delay of all requesting user equipment, so the system goal can be defined as follows:(13)T∑=∑n=1NTn,where Tn represents the calculation delay of the user n.Considering different computation and offloading modes,Tn is defined as follows:(14)Tn=1−αnTnloc+αnTnΩ,where αk represents the mode selected by the user equipment. When the variable is 1, it means that the offloading to the drone is selected, and when the variable is 0, it means that the local calculation is selected. Tnloc represents the total delay locally calculated by the user equipment; TnΩ represents the total delay calculated by the user to select all offload to the drone.Optimization of user equipment offloading strategy, UAV relay ratio, UAV trajectory, and user bit allocation are combined to minimize the total delay of user equipment. The mathematical expression is as follows:(15)P1:min∑n=1N1−αnTnloc+αnTnΩ,C1:αn∈0,1,n∈N,C2:ζnt∈0,1,n∈N′,C3:Tn≤Tnmax,n∈N,C4:∑t=1KDnt=Dn,n∈N′,C5:d1=dK,C6:dt+1−dt2≤vmaxΔ2,t∈τ,C7:∑n=1N′Dnt≤Cu,t∈K,where C1 represents that the user equipment n can only choose one mode; C2 represents that the relay ratio of UAV is a variable between 0 and 1. C3 represents that the calculation delay for each user equipment n must be less than the maximum tolerated delay. C4 represents that the user equipment n that selects the MEC mode needs to complete the offloading of tasks within K time slots. C5 means that the drone flies in cycles; C6 means that the maximum horizontal distance of the drone in one time slot cannot exceed the threshold. C7 represents that the bits offloaded by all user equipment to UAV in a time slot cannot exceed the computational threshold of UAV.


## 2.1. System Model

The scenario of human-machine-assisted MEC system uplink communication is studied, as shown in Figure1, that is, UAVs provide auxiliary computing services for multiple ground user equipment. Only the scenario where a single rotor UAV provides services to N user equipment on the ground is considered.Figure 1
System model.For the convenience of theoretical analysis, a three-dimensional Cartesian coordinate system is considered, whereLn=xn,yn represents the position of n user equipment on the ground. The drone flies in a circle within a specified range and provides auxiliary computing services to ground user equipment. Among them, UAV is at a fixed height and the initial position of UAV that is denoted as d1, and the final position is denoted as dK. Each ground user equipment offloads some of computing tasks to UAV and the rest is locally computed.It is assumed that the total execution time is denotedT, and it can be equally divided into K slots and denoted t=1,2,…,K, where τ=T/K, where τ refers to the length of each slot. In the t slot, the position of drone is dt=xt,yt. Assuming that the maximum flight speed of UAV is Vmax, the trajectory constraints of UAV are as follows:(1)d1=dK,dt+1−dtτ≤Vmax,where formula (1) indicates that the flight speed of the UAV cannot exceed its maximum flight speed. Since UAV flight energy consumption is related to UAV flight speed, UAV flight energy consumption expression is as follows:(2)Eflyt=ηvt2,where vt=dt+1−dt/τ represents the flight speed of UAV in the t time slot; η represents the proportional factor of energy consumption and flight speed.


## 2.2. Communication Model

Assuming that the wireless channel between UAV and each user equipment is a line-of-sight (LOS) channel, the channel power gain between UAV and user equipmentn can be expressed as follows:(3)hnt=β0dn−2t=β0H2+dt−Ln2,where β0 represents the channel power gain per unit distance; dn−2t represents the horizontal distance between UAV and user equipment n in the t gap; and ⋅ represents Euclidean norm.


## 2.3. Computational Model

### 2.3.1. Local Computing

When user equipmentn selects the local mode, all computing tasks will be locally performed, and the computing delay can be expressed by the following formula:(4)Tnloc=CnDnfn,n∈N,where fn is the computing capability of user equipment n (number of cycles per second); Cn represents the number of cycles required to calculate 1 bit; and Dn is the size of input data.


### 2.3.2. Offload to UAV for Calculation

The UAV carrying MEC server periodically flies at a fixed altitude above the ground user equipment. The user equipment that selects MEC mode can offload tasks to UAV through time-division multiple access (TDMA) [23, 24]. The set of ground users is set to be offloaded to MEC as N′, and UAV can choose to relay some tasks to access point (AP) for calculation. Therefore, it can be assumed that the ratio of user equipment n that selects the relay to the ground base station in the t time slot is ζnt,n∈N′,t∈K; then, the task that needs to be calculated by UAV in the t time slot is 1−ζntDnt.(1)
The transmission speed of user equipmentn in time slot t:(5)vnut=B1log21+Pnhntσ2,n∈N′,whereB1 is the bandwidth between UAV and user n; Pn is the maximum transmission power of the user n, and the user equipment uses the maximum power transmission; and hnt is t time slot channel gain between the user n and UAV communication.(2)
Transmission delay of user equipmentn: assuming that the bits offloaded by the user n in time slot t are Dnt, so the user n needs to offload all bits in K time slots, and the constraints are expressed as follows:(6)∑n=1NDnt=Dn.The total transmission delay of the user equipment is expressed as follows:(7)Tnmec=∑t=1KDntvnt.(3)
The calculation delay of UAV: according to the relay ratio, the task calculated by UAV for user equipmentn in the t time slot and then the calculation delay of user equipment in one time slot in MEC mode can be obtained, and finally, the calculation task of UAV for a single user equipment can be obtained. The total delay is as follows:(8)Tku=∑t=1K1−ζntDntCnfc,wherefc represents the computing power of MEC server.


### 2.3.3. Offload to Base Station on the Ground for Calculation

UAVs can choose to relay part of computing tasks to AP computing. The proportion of UAVs choosing relays in time slott is ζnt∈0,1,n∈N′. Since the ground base station can have multiple MEC servers built in, the calculation delay of ground base station is considered to be ignored. At the same time, the result of the calculation task is usually very small, and it is considered to ignore the transmission delay of calculation result. Thus, the delay of UAV relaying tasks to the computing part of the ground base station only includes the transmission delay of the UAV.The flying height of the UAV is relatively high from the ground base station and has a good line-of-sight link. It is considered that the communication between the UAV and ground base station adopts LOS channel. The transmission rate of UAV relaying the task to AP in the time slot is calculated as follows:(9)vnU2Et=B2log21+Puhutσ2,where B2 is the bandwidth between UAV and AP; Pu is the transmission power of UAV; and hut is the communication channel between UAV and AP.The transmission delay from UAV relay to ground AP can be expressed as follows:(10)TnU2E=∑t=1KζntDntvnU2Et.Assuming that UAV can calculate and transmit part of computing tasks to the ground AP at the same time, the total calculation delay offloaded to UAV is as follows:(11)Tnmax=maxTnU2E,Tnu.When the user equipment selects MEC mode, the total delay of offloading computing tasks to UAV is as follows:(12)TnΩ=Tnmec+Tnmax.


## 2.3.1. Local Computing

When user equipmentn selects the local mode, all computing tasks will be locally performed, and the computing delay can be expressed by the following formula:(4)Tnloc=CnDnfn,n∈N,where fn is the computing capability of user equipment n (number of cycles per second); Cn represents the number of cycles required to calculate 1 bit; and Dn is the size of input data.


## 2.3.2. Offload to UAV for Calculation

The UAV carrying MEC server periodically flies at a fixed altitude above the ground user equipment. The user equipment that selects MEC mode can offload tasks to UAV through time-division multiple access (TDMA) [23, 24]. The set of ground users is set to be offloaded to MEC as N′, and UAV can choose to relay some tasks to access point (AP) for calculation. Therefore, it can be assumed that the ratio of user equipment n that selects the relay to the ground base station in the t time slot is ζnt,n∈N′,t∈K; then, the task that needs to be calculated by UAV in the t time slot is 1−ζntDnt.(1)
The transmission speed of user equipmentn in time slot t:(5)vnut=B1log21+Pnhntσ2,n∈N′,whereB1 is the bandwidth between UAV and user n; Pn is the maximum transmission power of the user n, and the user equipment uses the maximum power transmission; and hnt is t time slot channel gain between the user n and UAV communication.(2)
Transmission delay of user equipmentn: assuming that the bits offloaded by the user n in time slot t are Dnt, so the user n needs to offload all bits in K time slots, and the constraints are expressed as follows:(6)∑n=1NDnt=Dn.The total transmission delay of the user equipment is expressed as follows:(7)Tnmec=∑t=1KDntvnt.(3)
The calculation delay of UAV: according to the relay ratio, the task calculated by UAV for user equipmentn in the t time slot and then the calculation delay of user equipment in one time slot in MEC mode can be obtained, and finally, the calculation task of UAV for a single user equipment can be obtained. The total delay is as follows:(8)Tku=∑t=1K1−ζntDntCnfc,wherefc represents the computing power of MEC server.


## 2.3.3. Offload to Base Station on the Ground for Calculation

UAVs can choose to relay part of computing tasks to AP computing. The proportion of UAVs choosing relays in time slott is ζnt∈0,1,n∈N′. Since the ground base station can have multiple MEC servers built in, the calculation delay of ground base station is considered to be ignored. At the same time, the result of the calculation task is usually very small, and it is considered to ignore the transmission delay of calculation result. Thus, the delay of UAV relaying tasks to the computing part of the ground base station only includes the transmission delay of the UAV.The flying height of the UAV is relatively high from the ground base station and has a good line-of-sight link. It is considered that the communication between the UAV and ground base station adopts LOS channel. The transmission rate of UAV relaying the task to AP in the time slot is calculated as follows:(9)vnU2Et=B2log21+Puhutσ2,where B2 is the bandwidth between UAV and AP; Pu is the transmission power of UAV; and hut is the communication channel between UAV and AP.The transmission delay from UAV relay to ground AP can be expressed as follows:(10)TnU2E=∑t=1KζntDntvnU2Et.Assuming that UAV can calculate and transmit part of computing tasks to the ground AP at the same time, the total calculation delay offloaded to UAV is as follows:(11)Tnmax=maxTnU2E,Tnu.When the user equipment selects MEC mode, the total delay of offloading computing tasks to UAV is as follows:(12)TnΩ=Tnmec+Tnmax.


## 2.4. Model Constraint Description

The system goal is to minimize the total delay of all requesting user equipment, so the system goal can be defined as follows:(13)T∑=∑n=1NTn,where Tn represents the calculation delay of the user n.Considering different computation and offloading modes,Tn is defined as follows:(14)Tn=1−αnTnloc+αnTnΩ,where αk represents the mode selected by the user equipment. When the variable is 1, it means that the offloading to the drone is selected, and when the variable is 0, it means that the local calculation is selected. Tnloc represents the total delay locally calculated by the user equipment; TnΩ represents the total delay calculated by the user to select all offload to the drone.Optimization of user equipment offloading strategy, UAV relay ratio, UAV trajectory, and user bit allocation are combined to minimize the total delay of user equipment. The mathematical expression is as follows:(15)P1:min∑n=1N1−αnTnloc+αnTnΩ,C1:αn∈0,1,n∈N,C2:ζnt∈0,1,n∈N′,C3:Tn≤Tnmax,n∈N,C4:∑t=1KDnt=Dn,n∈N′,C5:d1=dK,C6:dt+1−dt2≤vmaxΔ2,t∈τ,C7:∑n=1N′Dnt≤Cu,t∈K,where C1 represents that the user equipment n can only choose one mode; C2 represents that the relay ratio of UAV is a variable between 0 and 1. C3 represents that the calculation delay for each user equipment n must be less than the maximum tolerated delay. C4 represents that the user equipment n that selects the MEC mode needs to complete the offloading of tasks within K time slots. C5 means that the drone flies in cycles; C6 means that the maximum horizontal distance of the drone in one time slot cannot exceed the threshold. C7 represents that the bits offloaded by all user equipment to UAV in a time slot cannot exceed the computational threshold of UAV.


## 3. Solutions Based on Deep Reinforcement Learning

### 3.1. Reinforcement Learning Modeling

The optimization problem in equation (15) is solved by a single-agent Q-learning algorithm based on reinforcement learning. The algorithm model consists of four parts, namely, agent, state, action, and reward [25].(1)
Agent: Using UAV as the agent of the algorithm, UAV will be responsible for collecting the information of each equipment in the system and making scheduling decisions.(2)
Action:a=λ,ϖn is defined to represent each action decision variable of UAV. Among them, λ=n indicates that the drone will provide on-demand services for the n user equipment in the current state, and λ∈N is satisfied. ϖn represents the computing mode of required computing task of n user equipment.(3)
State: The state variables=ω,z,tserve,tfly is defined to represent the service state in the system. The state quantity consists of the following four parts: the user equipment serves state quantity ω=ω1,…,ωN and satisfies ωn∈0,1. ωn represents whether the n user equipment in the system has been serviced by drone and the service status has been completed, ωn=1 represents that the drone has completed the on-demand service for the first user equipment, and otherwise, ωn=0. The maximum tolerant delay amount z=z1,…,zN and satisfies zn=Tnmax, that is, zn represents the maximum tolerant delay of n user equipment in the system. Waiting service delay amount tserve=t1serve,…,tNserve; flight delay amount tfly=t1fly,…,tNfly.(4)
Rewards: The proposed optimization problem aims at maximizing the number of network user equipment services under UAV-MEC system architecture, while the proposed Q-learning algorithm based on reinforcement learning aims to obtain maximum reward feedback. Combining the above two points, and based on the current system state variable,s=ω,z,tserve,tfly and the selected action variable a=λ,ϖn. The system reward feedback function is defined as rs,a=∑n=1Nωn, that is, when UAV selects the action decision variable as a in the state s, the total number of network equipment in the system can meet the service requirements [26, 27].The training iterative process of the Q-learning algorithm based on reinforcement learning satisfies the Bellman equation. The selection principle of action decision performed by UAV under different state variables is based onε-greedy mechanism.


### 3.2. DQN-Based Offloading Strategy Optimization Algorithm

The pseudocode of the DQN-based offloading strategy optimization algorithm is shown in Algorithm1.Algorithm 1: Pseudocode of offloading strategy optimization algorithm based on DQN.
Initialization Experience pool capacityC1, Invalid action pool capacity C2, Parameters θ of action value function Q, Parameters θ^=θ of target action value function Q^, Initial state s.Begin(1)
Whilet≤tmax(2)
Forn=1:NIfrand0,1<εUsern randomly selects action a from action space A;ElseUsern selection action a=argmaxa∈AQs,a;θEnd ifEnd for(3)
Allocate resources according to user actions(4)
Forn=1:N(5)
Obtain a rewardr and a new status s′, and store the new sample s,a,r,s′ in the experience pool.(6)
Update statuss=s′(7)
Random samplingsi,ai,ri,si+1 is conducted from the experience pool to form small batch samples.(8)
Perform gradient descent onyi−Q(si,αi;θ2 with respect to parameter θ.(9)
Reset the target action value functionQ^=Q every e steps.(10)
End for(11)
t=t+1(12)
End While(13)
Determine the final user association matrix according to all action sets.EndFirst, the network parameters are initialized, including the capacityC1 of replay experience pool, the parameter θ of action value function Qs,a;θ, the parameter θ^=θ of target action value function Q^s,a;θ^, and the initial state s. Then, at each episode t, each user equipment randomly picks an action from the feasible action space according to ε-greedy policy. A random action with probability ε or an action satisfying the formula a=argmaxQs,a;θ.Then, resource allocation is performed according to the action selected by user equipment. If the local processing task is selected, let the local CPU cycle frequency be the maximum computing power of equipment, namely,Fn=fnloc. If the user equipment chooses to upload tasks to UAV for processing, let its uplink transmission power be the maximum available power, i.e., pn=Pn.After the resource allocation is completed, rewardr can be calculated according to the designed reward function, a new state s′ can be obtained, and a new sample s,a,r,s′ can be stored in the experience pool. Finally, a batch of samples is randomly sampled from the experience pool, and these samples are used to update the parameters θ of Q-Network, synchronizing the parameters θ^=θ of target Q-Network every e steps. Note that although each user equipment independently selects actions, since the resources of UAVs are shared, the results of their action selections influence each other [28, 29]. Assuming that all users have selected the same drone for association, the drone may be overloaded, resulting in the user's equipment mission not being able to complete within the specified latency limit, or the drone running out of energy. In order to balance the load as much as possible to avoid this extreme situation, an invalid action pool is added. After all user equipment complete the action selection, if the action set composed of all the user equipment cannot meet the energy consumption constraints of the base station or the load is too concentrated so that most of the user equipment cannot meet the delay limit Tnmax, the action set is added to invalid action pool [30]. In this way, when the user equipment performs action selection again, if the current action combination has already appeared in an invalid action pool, it is ignored and the selection is made again. This avoids invalid calculations to improve efficiency.


## 3.1. Reinforcement Learning Modeling

The optimization problem in equation (15) is solved by a single-agent Q-learning algorithm based on reinforcement learning. The algorithm model consists of four parts, namely, agent, state, action, and reward [25].(1)
Agent: Using UAV as the agent of the algorithm, UAV will be responsible for collecting the information of each equipment in the system and making scheduling decisions.(2)
Action:a=λ,ϖn is defined to represent each action decision variable of UAV. Among them, λ=n indicates that the drone will provide on-demand services for the n user equipment in the current state, and λ∈N is satisfied. ϖn represents the computing mode of required computing task of n user equipment.(3)
State: The state variables=ω,z,tserve,tfly is defined to represent the service state in the system. The state quantity consists of the following four parts: the user equipment serves state quantity ω=ω1,…,ωN and satisfies ωn∈0,1. ωn represents whether the n user equipment in the system has been serviced by drone and the service status has been completed, ωn=1 represents that the drone has completed the on-demand service for the first user equipment, and otherwise, ωn=0. The maximum tolerant delay amount z=z1,…,zN and satisfies zn=Tnmax, that is, zn represents the maximum tolerant delay of n user equipment in the system. Waiting service delay amount tserve=t1serve,…,tNserve; flight delay amount tfly=t1fly,…,tNfly.(4)
Rewards: The proposed optimization problem aims at maximizing the number of network user equipment services under UAV-MEC system architecture, while the proposed Q-learning algorithm based on reinforcement learning aims to obtain maximum reward feedback. Combining the above two points, and based on the current system state variable,s=ω,z,tserve,tfly and the selected action variable a=λ,ϖn. The system reward feedback function is defined as rs,a=∑n=1Nωn, that is, when UAV selects the action decision variable as a in the state s, the total number of network equipment in the system can meet the service requirements [26, 27].The training iterative process of the Q-learning algorithm based on reinforcement learning satisfies the Bellman equation. The selection principle of action decision performed by UAV under different state variables is based onε-greedy mechanism.


## 3.2. DQN-Based Offloading Strategy Optimization Algorithm

The pseudocode of the DQN-based offloading strategy optimization algorithm is shown in Algorithm1.Algorithm 1: Pseudocode of offloading strategy optimization algorithm based on DQN.
Initialization Experience pool capacityC1, Invalid action pool capacity C2, Parameters θ of action value function Q, Parameters θ^=θ of target action value function Q^, Initial state s.Begin(1)
Whilet≤tmax(2)
Forn=1:NIfrand0,1<εUsern randomly selects action a from action space A;ElseUsern selection action a=argmaxa∈AQs,a;θEnd ifEnd for(3)
Allocate resources according to user actions(4)
Forn=1:N(5)
Obtain a rewardr and a new status s′, and store the new sample s,a,r,s′ in the experience pool.(6)
Update statuss=s′(7)
Random samplingsi,ai,ri,si+1 is conducted from the experience pool to form small batch samples.(8)
Perform gradient descent onyi−Q(si,αi;θ2 with respect to parameter θ.(9)
Reset the target action value functionQ^=Q every e steps.(10)
End for(11)
t=t+1(12)
End While(13)
Determine the final user association matrix according to all action sets.EndFirst, the network parameters are initialized, including the capacityC1 of replay experience pool, the parameter θ of action value function Qs,a;θ, the parameter θ^=θ of target action value function Q^s,a;θ^, and the initial state s. Then, at each episode t, each user equipment randomly picks an action from the feasible action space according to ε-greedy policy. A random action with probability ε or an action satisfying the formula a=argmaxQs,a;θ.Then, resource allocation is performed according to the action selected by user equipment. If the local processing task is selected, let the local CPU cycle frequency be the maximum computing power of equipment, namely,Fn=fnloc. If the user equipment chooses to upload tasks to UAV for processing, let its uplink transmission power be the maximum available power, i.e., pn=Pn.After the resource allocation is completed, rewardr can be calculated according to the designed reward function, a new state s′ can be obtained, and a new sample s,a,r,s′ can be stored in the experience pool. Finally, a batch of samples is randomly sampled from the experience pool, and these samples are used to update the parameters θ of Q-Network, synchronizing the parameters θ^=θ of target Q-Network every e steps. Note that although each user equipment independently selects actions, since the resources of UAVs are shared, the results of their action selections influence each other [28, 29]. Assuming that all users have selected the same drone for association, the drone may be overloaded, resulting in the user's equipment mission not being able to complete within the specified latency limit, or the drone running out of energy. In order to balance the load as much as possible to avoid this extreme situation, an invalid action pool is added. After all user equipment complete the action selection, if the action set composed of all the user equipment cannot meet the energy consumption constraints of the base station or the load is too concentrated so that most of the user equipment cannot meet the delay limit Tnmax, the action set is added to invalid action pool [30]. In this way, when the user equipment performs action selection again, if the current action combination has already appeared in an invalid action pool, it is ignored and the selection is made again. This avoids invalid calculations to improve efficiency.


## 4. Experiments and Analysis

Simulation results are used to evaluate the performance and efficiency of the proposed strategy. It is assumed that the user equipment is randomly distributed in a two-dimensional area of 200 m × 200 m, and the AP is located in the upper right corner of the two-dimensional area. Each user equipment has different computing tasks, time delay tolerance, and the number of cycles required for computing a 1 bit task. The specific parameters are shown in Table1.Table 1
System simulation parameters.
System parametersValueBandwidth between UAV and user/MHz10Bandwidth between UAV and AP/GHz1UAV computing power/MHz1200User task size/Mbits[20, 30, 50]User transmission power/W3Maximum user delay/s[10, 15, 25]Flight cycle time slot of UAV32Fixed altitude of UAV/m60Maximum available speed of UAV/m·s−145Gauss white noise/W10–10


### 4.1. Average Reward of Algorithm Training Process

The average reward value of multiple episodes in the training phase of the DQN-based offloading strategy optimization algorithm is shown in Figure2. In order to increase the exploratory nature, a certain amount of noise is added to the action during the experiment, which makes the original image have many glitches and become unsmooth. Figure 2 shows the reward curve after taking the moving average.Figure 2
The average reward in the training process of the proposed algorithm.It can be seen from Figure2 that the proposed algorithm has converged at 2,000 time slots, and the average reward value fluctuates around 220. It can be seen that the algorithm has a fast convergence speed and an ideal average reward value.


### 4.2. Relationship between the Number of Convergence Iterations and the Number of IoT Equipment

As the number of equipment in the system increases, the number of times that the proposed algorithm training reaches convergence also increases, as shown in Figure3.Figure 3
Relationship between the number of convergence iterations and the number of system equipment.As can be seen from Figure3, when the number of system equipment increases, more iterations are required to achieve convergence. But when the number of equipment exceeds 35, the increase in the number of iterations increases. This is mainly because the Q-table size of the algorithm is closely related to the number of equipment present in the system. When the number of equipment exceeds the threshold, the processing pressure of the algorithm sharply increases, resulting in a rapid increase in the number of iterations required for convergence. However, the algorithm can achieve convergence regardless of the number of equipment, thus demonstrating the effectiveness of the proposed algorithm.


### 4.3. Relationship between the Calculated Energy Efficiency and Maximum Energy Consumption of Different Loading Models

The relationship between the computing energy efficiency of user equipment and the maximum energy consumption under different loading models is shown in Figure4.Figure 4
Relationship between calculated energy efficiency and maximum energy consumption of different loading models.As can be seen from Figure4, the local computing model is that the user equipment only performs local computing, while the global offloading model is that the user equipment completely offloads computing tasks to UAV for computing. Both schemes optimize the flight trajectory of UAV. Using the DQN-based offloading optimization algorithm for partial offloading can achieve higher computational energy efficiency. When the maximum energy consumption is 4J, the computing energy efficiency of its user equipment exceeds 200 bits/J. This is because the user equipment can flexibly allocate resources according to the quality of channel state information, so as to select offload computation or local computation under the partial offload model. Furthermore, the global offload model outperforms the local computing model, and the computing energy efficiency of user equipment increases with the increase in maximum consumed energy. The reason is that as the energy of user equipment increases, the user equipment has more energy to perform local computations or offload computations. In addition, in the local computing mode, the computing energy efficiency does not change as the energy increases. It can be considered that when the maximum energy consumption value is 1J, the computing energy efficiency of user equipment has reached the maximum value. Subsequently, the user equipment does not need to consume more energy to improve the computing energy efficiency.


### 4.4. Relationship between Total Delay and the Number of IoT Equipment

As the number of equipment in the system changes, the total latency of the IoT system will also accordingly change. In order to demonstrate the performance of the proposed strategy, it is compared with reference [13], reference [14], and reference [22], and the results are shown in Figure 5.Figure 5
Relationship between time delay and the number of user equipment under different strategies.As can be seen from Figure5, as the number of equipment increases, the total system delay also increases as expected. The main reason is that when the number of equipment in the system increases, UAV flight delay, UAV edge computing delay, equipment upload delay, and equipment local computing delay corresponding to the newly added equipment will be added to the system. will be added to the total delay of system. Furthermore, the proposed strategy significantly outperforms other comparative strategies in reducing the total system latency. When the number of user equipment is 40, the total delay is about 33s, which is 35.29%, 31.25%, and 15.38% lower than reference [13], reference [14], and reference [22], respectively. Since the proposed strategy utilizes mobile UAVs for computational offloading and optimizes resource allocation with DQN, the delay can be minimized. Reference [13] proposed a vehicle-assisted computing offloading architecture for UAVs. Since the optimization algorithm has a weak ability to seek optimization, it takes a long time, more than 50s. Reference [14] proposed a UAV-assisted agent-enabled computing task offloading framework to help users, UAVs, and edge clouds perform computing task offloading. However, this algorithm requires a large amount of computation and takes a long time, resulting in increased delay. Reference [22] proposed a deep reinforcement learning-based MEC UAV-assisted computing offloading scheme. When the number of user equipment is small, the total delay is close to the proposed strategy. However, when the number of user equipment is large, the processing timeliness is not strong, and the delay is relatively large.


## 4.1. Average Reward of Algorithm Training Process

The average reward value of multiple episodes in the training phase of the DQN-based offloading strategy optimization algorithm is shown in Figure2. In order to increase the exploratory nature, a certain amount of noise is added to the action during the experiment, which makes the original image have many glitches and become unsmooth. Figure 2 shows the reward curve after taking the moving average.Figure 2
The average reward in the training process of the proposed algorithm.It can be seen from Figure2 that the proposed algorithm has converged at 2,000 time slots, and the average reward value fluctuates around 220. It can be seen that the algorithm has a fast convergence speed and an ideal average reward value.


## 4.2. Relationship between the Number of Convergence Iterations and the Number of IoT Equipment

As the number of equipment in the system increases, the number of times that the proposed algorithm training reaches convergence also increases, as shown in Figure3.Figure 3
Relationship between the number of convergence iterations and the number of system equipment.As can be seen from Figure3, when the number of system equipment increases, more iterations are required to achieve convergence. But when the number of equipment exceeds 35, the increase in the number of iterations increases. This is mainly because the Q-table size of the algorithm is closely related to the number of equipment present in the system. When the number of equipment exceeds the threshold, the processing pressure of the algorithm sharply increases, resulting in a rapid increase in the number of iterations required for convergence. However, the algorithm can achieve convergence regardless of the number of equipment, thus demonstrating the effectiveness of the proposed algorithm.


## 4.3. Relationship between the Calculated Energy Efficiency and Maximum Energy Consumption of Different Loading Models

The relationship between the computing energy efficiency of user equipment and the maximum energy consumption under different loading models is shown in Figure4.Figure 4
Relationship between calculated energy efficiency and maximum energy consumption of different loading models.As can be seen from Figure4, the local computing model is that the user equipment only performs local computing, while the global offloading model is that the user equipment completely offloads computing tasks to UAV for computing. Both schemes optimize the flight trajectory of UAV. Using the DQN-based offloading optimization algorithm for partial offloading can achieve higher computational energy efficiency. When the maximum energy consumption is 4J, the computing energy efficiency of its user equipment exceeds 200 bits/J. This is because the user equipment can flexibly allocate resources according to the quality of channel state information, so as to select offload computation or local computation under the partial offload model. Furthermore, the global offload model outperforms the local computing model, and the computing energy efficiency of user equipment increases with the increase in maximum consumed energy. The reason is that as the energy of user equipment increases, the user equipment has more energy to perform local computations or offload computations. In addition, in the local computing mode, the computing energy efficiency does not change as the energy increases. It can be considered that when the maximum energy consumption value is 1J, the computing energy efficiency of user equipment has reached the maximum value. Subsequently, the user equipment does not need to consume more energy to improve the computing energy efficiency.


## 4.4. Relationship between Total Delay and the Number of IoT Equipment

As the number of equipment in the system changes, the total latency of the IoT system will also accordingly change. In order to demonstrate the performance of the proposed strategy, it is compared with reference [13], reference [14], and reference [22], and the results are shown in Figure 5.Figure 5
Relationship between time delay and the number of user equipment under different strategies.As can be seen from Figure5, as the number of equipment increases, the total system delay also increases as expected. The main reason is that when the number of equipment in the system increases, UAV flight delay, UAV edge computing delay, equipment upload delay, and equipment local computing delay corresponding to the newly added equipment will be added to the system. will be added to the total delay of system. Furthermore, the proposed strategy significantly outperforms other comparative strategies in reducing the total system latency. When the number of user equipment is 40, the total delay is about 33s, which is 35.29%, 31.25%, and 15.38% lower than reference [13], reference [14], and reference [22], respectively. Since the proposed strategy utilizes mobile UAVs for computational offloading and optimizes resource allocation with DQN, the delay can be minimized. Reference [13] proposed a vehicle-assisted computing offloading architecture for UAVs. Since the optimization algorithm has a weak ability to seek optimization, it takes a long time, more than 50s. Reference [14] proposed a UAV-assisted agent-enabled computing task offloading framework to help users, UAVs, and edge clouds perform computing task offloading. However, this algorithm requires a large amount of computation and takes a long time, resulting in increased delay. Reference [22] proposed a deep reinforcement learning-based MEC UAV-assisted computing offloading scheme. When the number of user equipment is small, the total delay is close to the proposed strategy. However, when the number of user equipment is large, the processing timeliness is not strong, and the delay is relatively large.


## 5. Conclusions

In order to give full play to the advantages of a UAV-assisted MEC network, it is necessary to further formulate a reasonable user offloading strategy and UAV flight trajectory. To this end, this study proposes a network resource allocation strategy based on UAV collaborative edge computing. Based on the system scenario of collaborative computing between UAVs and ground users, an optimization model is constructed to minimize the total system delay and use DQN to optimize, so as to obtain the best resource allocation scheme. The experimental results show the following:(1)
With the fast mobility of UAVs, users in the system can offload computing tasks in real time, which greatly reduces the transmission delay. Especially when the computing power of UAV reaches 2000 MHz, the total system delay is about 20s.(2)
The proposed strategy uses DQN to optimize the offloading strategy, which not only has a fast optimization speed but also has an efficient optimization result, and realizes a reasonable allocation of resources while reducing the delay. Especially when the amount of user data is large, the optimization effect is more obvious. When the number of user equipment is 40, the total delay is about 33s, which is more than 10% lower than the total delay of other comparison strategies.As the network scale becomes larger, the network scenarios also become more complex. In order to be suitable for large-scale complex network scenarios and meet the real-time requirements of MEC systems, we will focus on researching and designing a simpler and more effective online scheduling strategy in the future. This is crucial for the large-scale application of MEC.



---

*Source: 8514235-2022-03-29.xml*