# Deep Reinforcement Learning-Based UAV Data Collection and Offloading in NOMA-Enabled Marine IoT Systems

Authors: Yanpeng Dai; Ziyi Liang; Ling Lyu; Bin Lin
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

The rapid growth of maritime wireless communication demand and the complex offshore wireless communication environment have brought challenges to ensure the real-time and reliability of data transmission in the marine Internet of Things (MIoT). Unmanned aerial vehicles (UAVs) have great advantages in enhancing coverage and channel quality. Hence, we investigate a UAV-assisted data collection and data offloading system based on nonorthogonal multiple access (NOMA) technology in this paper. We jointly optimize the buoy-UAV association relationship, transmit powers, and the UAV trajectory to minimize the total mission completion time while ensuring data transmission requirements. We first propose a UAV trajectory optimization algorithm based on deep reinforcement learning (DRL). Then, we design a heuristic algorithm to effectively solve the subproblem of power control and the association relationship. Finally, we propose a joint optimization scheme to solve the minimization problem. Simulation results show the effectiveness of the proposed scheme.


---

## Body

## 1. Introduction

Marine environmental monitoring is indispensable with the continuous increase of human marine activities. A large amount of meteorological and hydrological data leads to an increase in the demand for maritime wireless communication [1, 2]. Buoys are widely deployed in the ocean due to their low cost and flexible deployment. With the development of technology, buoys can be used for marine environment monitoring with a variety of sensors and communication equipment and can also be powered by power supply methods such as lithium-ion batteries and solar energy [3, 4]. However, the transmit power of the buoy is limited. Traditional maritime wireless communication methods, such as land base stations and satellites, have disadvantages such as limited coverage and long transmission distance, which seriously affect the real-time and reliability of information transmission [5]. For the current five-generation (5G) and the upcoming six-generation (6G) era, it is of great significance to build an efficient and dynamic maritime communication network [6]. Therefore, the UAV-assisted wireless communication system (UWCS) has received widespread attention.Unmanned aerial vehicles (UAVs) have the advantages of maneuverability and easy manipulation, which can be deployed on demand and enlarge coverage [7]. It is easier to establish a line-of-sight (LoS) channel and a stronger communication link with the target device, which can better deal with the variable ocean environment [8, 9]. In the marine Internet of Things (MIoT), aiming at the problem of the large number and wide distribution of buoys, UAV can act as a mobile base station, collecting data collected by buoys from the target area and offloading the data to the OBS [10, 11]. Furthermore, the limited spectrum resources of MIoT also pose a challenge to the reliability and efficiency of data transmission. Nonorthogonal multiple access (NOMA) technology is considered a promising technology in the 5G era [12]. Compared with orthogonal multiple access (OMA) technology, NOMA greatly improves the spectrum efficiency in the presence of limited spectrum resources by allowing multiple users to access simultaneously in the same channel and relying on power domain multiplexing and successive interference cancellation (SIC) decoding technology [13–15].Recently, much research has applied NOMA technology to UAV-assisted wireless communication system. Zhao et al. in [13] investigated a NOMA-assisted UAV large-scale IoT data collection system and proposed a data collection optimization algorithm. The results show that, compared with the traditional UWCS, the NOMA-based UWCS has better performance in data collection. W. Chen et al. in [16] maximized the sum rate of the UAV-assisted uplink NOMA system by jointly optimizing the UAV location, buoy sensor grouping, and power control. The simulation results show the performance gain of NOMA in the sum rate of the system. Tang et al. in [17] investigated the scenario of a UAV-assisted marine wireless communication downlink in which the UAV hovers continuously to provide services to multiple groups of ships. Obviously, the NOMA-based UWCS has great advantages in enhancing coverage and strengthening communication links.In the above work, the optimization problem is usually formulated as a mixed-integer nonconvex problem. They can usually be divided into several subproblems, which can be solved by traditional optimization techniques and iterative algorithms [18]. However, the above solutions may have high computational complexity. Furthermore, the buoys associated with the UAV and NOMA cochannel interference vary with UAV position. The complex dynamic changes bring great challenges to the traditional convex optimization technology as well. With the development of machine learning technology, reinforcement learning (RL) is considered to be an effective solution to the high-dynamic environment [19–21]. Deep reinforcement learning (DRL) solves the continuous state space problem that RL cannot solve by introducing deep neural network (DNN), such as deep Q-learning (DQN) and deep deterministic policy gradient (DDPG) [22]. At present, a lot of work has focused on the research of UWCS based on DRL. L. Wang et al. in [23] minimized the energy consumption of all user equipment by jointly optimizing UAV trajectories, user associations, and resource allocation. Two algorithms are proposed to effectively solve the minimization problem based on convex optimization and DRL technology, respectively. The results show that the DRL-based method is better than the convex optimization method. Zhang et al. in [24] studied the UAV lineup and user distribution change scenarios and developed a DDPG-based proactive self-regulation method for UAV networks, which is based on the proposed asynchronous parallel computing architecture. Wang et al. in [25] studied a UAV-assisted mobile edge computing system. They minimized the maximum processing delay and proposed a DDPG-based algorithm to solve the high-dimensional state space and continuous action space. However, the flight action space of actual UAV is continuous and high dimensional, which may bring dimensional disaster to traditional reinforcement learning methods (such as DQN) [23, 26]. DDPG exists the problem of overestimation. To solve the above problems, Fujimoto et al. in [27] proposed the twin-delayed deep deterministic (TD3) algorithm based on DDPG. In [28], Sun et al. considered the age of information (AoI) and energy consumption and proposed an AoI-energy-aware UAV trajectory optimization algorithm based on TD3.In this paper, we investigate a UAV-assisted data collection and data offloading system in MIoT. Specifically, buoys are used to collect marine environment information in the sensing layer. The UAV collects the sensing information from buoys based on NOMA technology and offloads the collected data to the OBS. Our goal is to minimize the total mission completion time of the UAV by jointly optimizing the UAV trajectory, the buoy-UAV association relationship, the UAV transmit power, and the buoys transmit power. The main contributions of our paper are listed as follows.(i)
We jointly consider the UAV trajectory, buoy-UAV association relationship, and transmit powers to investigate the UAV’s total mission completion time minimization problem. The above minimization problem is a mixed-integer nonconvex problem. Accordingly, we divide the total mission process of UAV into data collection stage and data offloading stage for analysis(ii)
We propose a UAV trajectory optimization algorithm based on TD3 to solve the UAV trajectory coupling of data collection and data offloading since the minimization problem is a mixed-integer nonconvex problem. Furthermore, we design a heuristic algorithm to effectively solve the above problem due to the coupling between the buoy-UAV association relationship and the buoys transmit power(iii)
We propose a joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme that effectively solves the mixed-integer nonconvex problem. The simulation results show that the proposed scheme can effectively shorten the UAV’s total mission completion time while ensuring that the data transmission requirements are metThe remainder of this paper is organized as follows. Section2 presents the system model and the problem formulation. Section 3 briefly introduces the TD3 algorithm and proposes the TD3-based UAV trajectory optimization algorithm. Then, we design a heuristic algorithm to solve the subproblem of power control and buoy-UAV association relationship. Finally, we propose a joint optimization scheme for the minimization problem. Simulation results and conclusion are given in Sections 4 and 5.


## 2. System Model and Problem Formulation

### 2.1. Network Model

We consider a UAV-assisted MIoT system as shown in Figure1, which includes a UAV base station, M buoys, and an OBS. Each buoy senses and stores hydrometeorological data and is powered by a lithium-ion battery to ensure that it has sufficient energy to transmit data. The total mission time of the UAV is denoted as Ttotal and divided into K time slots. The time slot length is δ. The total mission process of UAV consists of two stages. The first stage is that the UAV utilizes NOMA to collect data from M buoys, and the number of time slots of this stage is Kco. The UAV is allowed to collect data from at most U buoys in each time slot where U≤M. The second stage is that the UAV offloads all the collected data to the OBS after completing the first stage, and the number of time slots of this stage is Kof. Therefore, Ttotal can be expressed as
(1)Ttotal=Kδ=Kco+Kofδ.Figure 1
UAV-assisted marine IoT system.LetM=1,2,⋯,M denote the set of all buoys. We denote that the horizontal coordinate of the UAV in the k-th time slot is qk=xk,yk, where k∈K=1,⋯,Kco,Kco+1,⋯,K. Let Kco=1,2,⋯,Kco denote the set of the data collection time, and Kof=Kco+1,⋯,K denote the set of the data offloading time.The fixed flight height of UAV isH, and the flight velocity of UAV in the k-th time slot is Vk. Then, the UAV should follow the maximum flight velocity constraints, which are expressed as
(2)Vk+1=Vk+Δkδ,∀k∈K,Vmin≤Vk≤Vmax,∀k∈K,Δk≤Δmax,∀k∈K,qk+1=qk+Vkδ+12Δkδ2,∀k∈K,whereVmax and Δmax are the maximum flight velocity and acceleration, respectively, and Vmin is the minimum flight velocity.The horizontal coordinate of them-th buoy is Dm=xm,ym. The horizontal coordinate of OBS is D0=x0,y0. If the time slot δ is small enough, the motion of the UAV in each time slot can be regarded as static approximately. Hence, the distance between UAV and the m-th buoy at the k-th time slot is dm,k=qk−Dm2+H−Hm2, ∀k∈Kco, ∀m∈M. In the stage of data offloading, the distance between UAV and OBS at the k-th time slot is d0,k=∥qk−D0∥2+H−H02,∀k∈Kof. Hm and H0 denote the antenna heights of the m-th buoy and OBS, respectively.


### 2.2. Transmission Model

#### 2.2.1. Channel Model

We adopt the model of the air-to-ground channel and the two-ray path loss model [29] and give the LoS and NLoS path loss models of the buoy-UAV and UAV-OBS links, respectively.Specifically, the channel gain of buoy-UAV and UAV-OBS links is expressed ashi,k=1/Li,k,∀k∈K,∀i∈0,1,⋯,M, where Li,k denotes the average path loss in the k-th time slot, expressed as
(3)Li,k=Pi,kLoSLi,kLoS+1−Pi,kLoSLi,kNLoS,Li,kLoS=4πdi,kλ2μi,kξLoS,Li,kNLoS=4πdi,kλ2μi,kξNLoS,whereLi,kLoS and Li,kNLoS are the average path loss for LoS and NLoS, μi,k=1. ξLoS and ξNLoS are the excessive path loss for LoS and NLoS paths, respectively, λ is the wavelength, and Pi,kLoS denotes the probability of LoS link which is expressed as
(4)Pi,kLoS=11+aexp−bψi,k−a,where a and b are two constant values depending on the environment and ψi,k denotes the elevation angle between the m-th buoy (or OBS) and UAV, which is given by ψi,k=180∘/π×arcsinH−Hi/qk−Di,k2.


#### 2.2.2. UAV Data Collection from Buoys

In this stage that∀k∈Kco, let αm,k denote the association indicator between m-th buoy and UAV. αm,k=1 means that the m-th buoy is associated with the UAV in the k-th time slot. Otherwise, αm,k=0.In uplink NOMA system, the UAV is regarded as a receiver to receive signals from multiple buoys at the same time and allows multiple buoys to share the same channel. The SIC decoding technique is used to demodulate the received signals with different received power levels. The successfully demodulated signal is deleted from all received signals, and the later decoded signal receives less cochannel interference. Therefore, the buoy with high channel gain is usually demodulated first, and its interference comes from the buoys with worse channel gain [30, 31]. The cochannel interference of uplink transmission between the m-th buoy and UAV in the k-th time slot can be given by
(5)Im,k=∑i∈Mkαi,kPi,khi,k,∀k∈Kco,where Mk=ii∈M,hm,k>hi,k is the set of the buoy whose channel gain is worse than the m-th buoy in the k-th time slot.Hence, the signal-to-interference-noise-ratio (SINR) between them-th buoy and UAV at the k-th time slot is expressed as
(6)gm,k=Pm,khm,kσ2+Im,k,∀m∈M,∀k∈Kco,where Pm,k is the transmit power of the m-th buoy in the k-th time slot and σ2 is the noise power.The transmission rate ofm-th buoy in the k-th time slot is expressed as
(7)Rm,k=αm,kBlog21+gm,k,∀m∈M,∀k∈Kco,where B is the spectrum. In order for the received signal to be demodulated successfully, the SIC demodulation condition that SINR needs to meet is as [32]
(8)αm,kPm,khm,k∑i∈Mkαi,kPi,khi,k+σ2≥ηSIC,∀m∈M,where ηSIC denotes the SIC threshold.


#### 2.2.3. UAV Data Offloading to OBS

In this stage that∀k∈Kof, let βk denote the association indicator between UAV and OBS. βk=1 means that the UAV can offload the data to the OBS at the k-th time slot. Otherwise, βk=0.The signal-to-noise-ratio (SNR) between the UAV and OBS at thek-th time slot needs to satisfy the following condition:
(9)g0,k=Pkh0,kσ2≥g¯0,∀k∈Kof,where Pk is the transmit power of the UAV and g¯0 is the SNR threshold.The transmission rate between the UAV and OBS at thek-th time slot is given by
(10)R0,k=βkBlog21+g0,k,∀k∈Kof.


### 2.3. Problem Formulation

Our goal is to minimize the UAV’s total mission completion time by jointing optimization of the buoy-UAV association relationship, UAV transmit power, buoys transmit power, and UAV trajectory. LetAm=αm,k,∀m∈M,∀k∈Kco denote the buoy-UAV-associated variables, P=Pk,∀k∈Kof denote the UAV transmit power during data offloading, Pm=Pm,k,∀m∈M,∀k∈Kco denote the buoys transmit power, and Q=qk,∀k∈K denote the UAV trajectory. The total mission completion time minimization problem can be formulated as
(11)P1:minAm,P,Pm,QTtotal,s.t.C1:0≤Pm,k≤Pmmax,C2:0≤Pk≤Pmax,C3:αm,k∈0,1,∀m∈M,∀k∈Kco,C4:βk∈0,1,∀k∈Kof,C5:∑m=1Mαm,k≤U,∀k∈Kco,C6:∑k=Kco+1KR0,kδ≥∑m=1MCm,C7:∑k=1KcoRm,kδ≥Cm,∀m∈M,C8−C11:3,4,13,14In problemP1, C1, and C2 restrict the maximum transmit power of UAV and buoy, respectively. C5 limits the maximum number of buoys that can be associated with the UAV in each time slot. Let Cm denote the data size that needs to be collected in m-th buoy. C6 ensures that all data collected by the UAV is offloaded to OBS. C7 ensures that the data collection requirements of each buoy are met. C8 and C9 are the UAV maximum velocity and maximum acceleration constraints, respectively. C10 is the SIC demodulation constraint that SINR needs to meet in the data collection stage. C11 is the SNR constraint in the data offloading. Problem P1 is a mixed-integer nonconvex problem since it contains binary relational variables, which makes it difficult to be solved effectively.


## 2.1. Network Model

We consider a UAV-assisted MIoT system as shown in Figure1, which includes a UAV base station, M buoys, and an OBS. Each buoy senses and stores hydrometeorological data and is powered by a lithium-ion battery to ensure that it has sufficient energy to transmit data. The total mission time of the UAV is denoted as Ttotal and divided into K time slots. The time slot length is δ. The total mission process of UAV consists of two stages. The first stage is that the UAV utilizes NOMA to collect data from M buoys, and the number of time slots of this stage is Kco. The UAV is allowed to collect data from at most U buoys in each time slot where U≤M. The second stage is that the UAV offloads all the collected data to the OBS after completing the first stage, and the number of time slots of this stage is Kof. Therefore, Ttotal can be expressed as
(1)Ttotal=Kδ=Kco+Kofδ.Figure 1
UAV-assisted marine IoT system.LetM=1,2,⋯,M denote the set of all buoys. We denote that the horizontal coordinate of the UAV in the k-th time slot is qk=xk,yk, where k∈K=1,⋯,Kco,Kco+1,⋯,K. Let Kco=1,2,⋯,Kco denote the set of the data collection time, and Kof=Kco+1,⋯,K denote the set of the data offloading time.The fixed flight height of UAV isH, and the flight velocity of UAV in the k-th time slot is Vk. Then, the UAV should follow the maximum flight velocity constraints, which are expressed as
(2)Vk+1=Vk+Δkδ,∀k∈K,Vmin≤Vk≤Vmax,∀k∈K,Δk≤Δmax,∀k∈K,qk+1=qk+Vkδ+12Δkδ2,∀k∈K,whereVmax and Δmax are the maximum flight velocity and acceleration, respectively, and Vmin is the minimum flight velocity.The horizontal coordinate of them-th buoy is Dm=xm,ym. The horizontal coordinate of OBS is D0=x0,y0. If the time slot δ is small enough, the motion of the UAV in each time slot can be regarded as static approximately. Hence, the distance between UAV and the m-th buoy at the k-th time slot is dm,k=qk−Dm2+H−Hm2, ∀k∈Kco, ∀m∈M. In the stage of data offloading, the distance between UAV and OBS at the k-th time slot is d0,k=∥qk−D0∥2+H−H02,∀k∈Kof. Hm and H0 denote the antenna heights of the m-th buoy and OBS, respectively.


## 2.2. Transmission Model

### 2.2.1. Channel Model

We adopt the model of the air-to-ground channel and the two-ray path loss model [29] and give the LoS and NLoS path loss models of the buoy-UAV and UAV-OBS links, respectively.Specifically, the channel gain of buoy-UAV and UAV-OBS links is expressed ashi,k=1/Li,k,∀k∈K,∀i∈0,1,⋯,M, where Li,k denotes the average path loss in the k-th time slot, expressed as
(3)Li,k=Pi,kLoSLi,kLoS+1−Pi,kLoSLi,kNLoS,Li,kLoS=4πdi,kλ2μi,kξLoS,Li,kNLoS=4πdi,kλ2μi,kξNLoS,whereLi,kLoS and Li,kNLoS are the average path loss for LoS and NLoS, μi,k=1. ξLoS and ξNLoS are the excessive path loss for LoS and NLoS paths, respectively, λ is the wavelength, and Pi,kLoS denotes the probability of LoS link which is expressed as
(4)Pi,kLoS=11+aexp−bψi,k−a,where a and b are two constant values depending on the environment and ψi,k denotes the elevation angle between the m-th buoy (or OBS) and UAV, which is given by ψi,k=180∘/π×arcsinH−Hi/qk−Di,k2.


### 2.2.2. UAV Data Collection from Buoys

In this stage that∀k∈Kco, let αm,k denote the association indicator between m-th buoy and UAV. αm,k=1 means that the m-th buoy is associated with the UAV in the k-th time slot. Otherwise, αm,k=0.In uplink NOMA system, the UAV is regarded as a receiver to receive signals from multiple buoys at the same time and allows multiple buoys to share the same channel. The SIC decoding technique is used to demodulate the received signals with different received power levels. The successfully demodulated signal is deleted from all received signals, and the later decoded signal receives less cochannel interference. Therefore, the buoy with high channel gain is usually demodulated first, and its interference comes from the buoys with worse channel gain [30, 31]. The cochannel interference of uplink transmission between the m-th buoy and UAV in the k-th time slot can be given by
(5)Im,k=∑i∈Mkαi,kPi,khi,k,∀k∈Kco,where Mk=ii∈M,hm,k>hi,k is the set of the buoy whose channel gain is worse than the m-th buoy in the k-th time slot.Hence, the signal-to-interference-noise-ratio (SINR) between them-th buoy and UAV at the k-th time slot is expressed as
(6)gm,k=Pm,khm,kσ2+Im,k,∀m∈M,∀k∈Kco,where Pm,k is the transmit power of the m-th buoy in the k-th time slot and σ2 is the noise power.The transmission rate ofm-th buoy in the k-th time slot is expressed as
(7)Rm,k=αm,kBlog21+gm,k,∀m∈M,∀k∈Kco,where B is the spectrum. In order for the received signal to be demodulated successfully, the SIC demodulation condition that SINR needs to meet is as [32]
(8)αm,kPm,khm,k∑i∈Mkαi,kPi,khi,k+σ2≥ηSIC,∀m∈M,where ηSIC denotes the SIC threshold.


### 2.2.3. UAV Data Offloading to OBS

In this stage that∀k∈Kof, let βk denote the association indicator between UAV and OBS. βk=1 means that the UAV can offload the data to the OBS at the k-th time slot. Otherwise, βk=0.The signal-to-noise-ratio (SNR) between the UAV and OBS at thek-th time slot needs to satisfy the following condition:
(9)g0,k=Pkh0,kσ2≥g¯0,∀k∈Kof,where Pk is the transmit power of the UAV and g¯0 is the SNR threshold.The transmission rate between the UAV and OBS at thek-th time slot is given by
(10)R0,k=βkBlog21+g0,k,∀k∈Kof.


## 2.2.1. Channel Model

We adopt the model of the air-to-ground channel and the two-ray path loss model [29] and give the LoS and NLoS path loss models of the buoy-UAV and UAV-OBS links, respectively.Specifically, the channel gain of buoy-UAV and UAV-OBS links is expressed ashi,k=1/Li,k,∀k∈K,∀i∈0,1,⋯,M, where Li,k denotes the average path loss in the k-th time slot, expressed as
(3)Li,k=Pi,kLoSLi,kLoS+1−Pi,kLoSLi,kNLoS,Li,kLoS=4πdi,kλ2μi,kξLoS,Li,kNLoS=4πdi,kλ2μi,kξNLoS,whereLi,kLoS and Li,kNLoS are the average path loss for LoS and NLoS, μi,k=1. ξLoS and ξNLoS are the excessive path loss for LoS and NLoS paths, respectively, λ is the wavelength, and Pi,kLoS denotes the probability of LoS link which is expressed as
(4)Pi,kLoS=11+aexp−bψi,k−a,where a and b are two constant values depending on the environment and ψi,k denotes the elevation angle between the m-th buoy (or OBS) and UAV, which is given by ψi,k=180∘/π×arcsinH−Hi/qk−Di,k2.


## 2.2.2. UAV Data Collection from Buoys

In this stage that∀k∈Kco, let αm,k denote the association indicator between m-th buoy and UAV. αm,k=1 means that the m-th buoy is associated with the UAV in the k-th time slot. Otherwise, αm,k=0.In uplink NOMA system, the UAV is regarded as a receiver to receive signals from multiple buoys at the same time and allows multiple buoys to share the same channel. The SIC decoding technique is used to demodulate the received signals with different received power levels. The successfully demodulated signal is deleted from all received signals, and the later decoded signal receives less cochannel interference. Therefore, the buoy with high channel gain is usually demodulated first, and its interference comes from the buoys with worse channel gain [30, 31]. The cochannel interference of uplink transmission between the m-th buoy and UAV in the k-th time slot can be given by
(5)Im,k=∑i∈Mkαi,kPi,khi,k,∀k∈Kco,where Mk=ii∈M,hm,k>hi,k is the set of the buoy whose channel gain is worse than the m-th buoy in the k-th time slot.Hence, the signal-to-interference-noise-ratio (SINR) between them-th buoy and UAV at the k-th time slot is expressed as
(6)gm,k=Pm,khm,kσ2+Im,k,∀m∈M,∀k∈Kco,where Pm,k is the transmit power of the m-th buoy in the k-th time slot and σ2 is the noise power.The transmission rate ofm-th buoy in the k-th time slot is expressed as
(7)Rm,k=αm,kBlog21+gm,k,∀m∈M,∀k∈Kco,where B is the spectrum. In order for the received signal to be demodulated successfully, the SIC demodulation condition that SINR needs to meet is as [32]
(8)αm,kPm,khm,k∑i∈Mkαi,kPi,khi,k+σ2≥ηSIC,∀m∈M,where ηSIC denotes the SIC threshold.


## 2.2.3. UAV Data Offloading to OBS

In this stage that∀k∈Kof, let βk denote the association indicator between UAV and OBS. βk=1 means that the UAV can offload the data to the OBS at the k-th time slot. Otherwise, βk=0.The signal-to-noise-ratio (SNR) between the UAV and OBS at thek-th time slot needs to satisfy the following condition:
(9)g0,k=Pkh0,kσ2≥g¯0,∀k∈Kof,where Pk is the transmit power of the UAV and g¯0 is the SNR threshold.The transmission rate between the UAV and OBS at thek-th time slot is given by
(10)R0,k=βkBlog21+g0,k,∀k∈Kof.


## 2.3. Problem Formulation

Our goal is to minimize the UAV’s total mission completion time by jointing optimization of the buoy-UAV association relationship, UAV transmit power, buoys transmit power, and UAV trajectory. LetAm=αm,k,∀m∈M,∀k∈Kco denote the buoy-UAV-associated variables, P=Pk,∀k∈Kof denote the UAV transmit power during data offloading, Pm=Pm,k,∀m∈M,∀k∈Kco denote the buoys transmit power, and Q=qk,∀k∈K denote the UAV trajectory. The total mission completion time minimization problem can be formulated as
(11)P1:minAm,P,Pm,QTtotal,s.t.C1:0≤Pm,k≤Pmmax,C2:0≤Pk≤Pmax,C3:αm,k∈0,1,∀m∈M,∀k∈Kco,C4:βk∈0,1,∀k∈Kof,C5:∑m=1Mαm,k≤U,∀k∈Kco,C6:∑k=Kco+1KR0,kδ≥∑m=1MCm,C7:∑k=1KcoRm,kδ≥Cm,∀m∈M,C8−C11:3,4,13,14In problemP1, C1, and C2 restrict the maximum transmit power of UAV and buoy, respectively. C5 limits the maximum number of buoys that can be associated with the UAV in each time slot. Let Cm denote the data size that needs to be collected in m-th buoy. C6 ensures that all data collected by the UAV is offloaded to OBS. C7 ensures that the data collection requirements of each buoy are met. C8 and C9 are the UAV maximum velocity and maximum acceleration constraints, respectively. C10 is the SIC demodulation constraint that SINR needs to meet in the data collection stage. C11 is the SNR constraint in the data offloading. Problem P1 is a mixed-integer nonconvex problem since it contains binary relational variables, which makes it difficult to be solved effectively.


## 3. Proposed Scheme

In order to solve problemP1, we first propose a TD3-based UAV trajectory optimization algorithm (TTO). Then, we design a heuristic algorithm to solve the power control problem while determining the buoy-UAV association relationship (PCAR). Finally, the above two algorithms are combined to effectively solve the problem P1.


### 3.1. TD3-Based UAV Trajectory Optimization

In our system, the start position of data offloading stage is similar to the end position of data collection, which is named as the transition position (TP) between two stages. Therefore, the UAV trajectories of these two stages are coupled, and the TP cannot be determined in advance. Furthermore, the UAV trajectory changes dynamically according to the requirements of data collection and data offloading. The traditional deterministic optimization method is difficult to solve the above problems [19]. Therefore, in this paper, we use an advanced DRL method, TD3, to solve the UAV trajectory optimization subproblem with the given transmit powers and association relationships. In the following, we first give the definitions of state, action, and reward and then briefly introduce the TD3.


#### 3.1.1. State Definition

In the data collection stage, the UAV’s action is closely related to the remaining data size of buoys and the buoy-UAV association relationship. Similarly, in the data offloading stage, the UAV’s action is related to the remaining data size of UAV and the UAV-OBS association relationship. Furthermore, the UAV only collects data from buoys in the target area. Hence, we define the state space as(12)Sk=αk,ck,xk,yk,βk,ckuav,ρk,where the variables contained in the above expression are defined as
(i)
αk=α1,k,α2,k,⋯,αM,k denotes the set of the buoy-UAV association relationship in the k-th time slot(ii)
ck=c1,k,c2,k,⋯,cM,k denotes the remaining data size of each buoy in the k-th time slot(13)cm,k=Cm−∑j=1k−1Rm,jδ(iii)
ckuav denotes the remaining data size of UAV(14)ckuav=∑k=1Kco∑m=1MRm,kδ−∑j=Kco+1k−1R0,jδ(iv)
ρk denotes the boundary penalty information of the UAV to judge whether the UAV position exceeds the target area in the k-th time slot


#### 3.1.2. Action Definition

Based on the above state and environment information, the UAV’s action is defined as(15)Ak=φk,Vk,where φk∈0,2π denotes the flight angle of UAV in the k-th time slot and Vk∈0,Vmax.


#### 3.1.3. Reward Definition

LetKmax denote the number of maximum total mission completion time slots. K∗=Kmax−K if the UAV completes the mission in K time slots. Then, we design the reward function as
(16)rk=Rkδ+ρk+K∗,ifC6andC7aresatisfied,CkorRkδ+ρk,otherwise.It can be seen from (16) that the shorter the time it takes for the UAV to complete the mission, the greater the reward it will eventually obtain.Note that for better performance, we reduce the order of magnitude ofB and Cm by n1 orders of magnitude to be less than or equal to the order of magnitude of Kmax when calculating the state and reward.


#### 3.1.4. TD3

The TD3 has the following advantages [27]:
(i)
Clipped doubleQ-learning for actor-critic: TD3 contains two critic networks. For the two target Q-values generated by the two critic target networks, the minimum of them is selected to suppress the overestimation problem caused by high variance, expressed as(17)y=rk+γmini=1,2Qθi′S′,A~where y is the target value which is used to update the two critic networks, r is reward, γ is the reward discount factor, A~ is the target policy, θi′ is the target network parameter, and S′ is the next state
(ii)
Target networks and delayed policy updates: the TD3 algorithm updates the actor and its target network after a fixed number of updates to the critic network, expressed as(18)θi′⟵τθi+1−τθi′,i=1,2where τ is the update parameter
(iii)
Target policy smoothing regularization: TD3 smoothes the estimate and reduces the error by adding a small amount of random noise to the target actor network and averaging over minibatch, expressed as(19)A~⟵πϕ′Sk+ε,ε~clipN0,ω~,−c,cIt can be seen from (12) that most of the state dimensions are related to buoys. Only two dimensions are related to the UAV’s position, two dimensions are related to OBS, and one dimension is related to UAV boundary penalty information. Therefore, there is a problem of dimension imbalance. Dimension spread technology can effectively solve this problem. We spread the above state dimensions. For example, we connect the position state dimension of UAV to a spread network composed of M neurons and spread its dimension to M [33, 34]. Furthermore, we set a termination flag l to indicate whether UAV has completed its mission. l is applied to the target value function. Hence, the Q-value of the target value function is 0 after the UAV completes the mission, so as to make the critic learning performance more stable [34].In summary, our proposed algorithm TTO is shown in Algorithm1. The update process of TD3 is shown in Figure 2. In lines 1-3, we first initialize the network parameters and the experience replay buffer B. In lines 5-17, we initialize the environment, obtain the initial state information, and set a time variable k to represent the time spent by UAV to perform the mission. Moreover, a termination mark l is set. l=0 indicates that UAV has not completed the mission. Then, UAV makes an action selection according to the observed state and environmental information. Specifically, UAV constantly interacts with the environment and updates the actor and critic networks. The actor network outputs the action Ak to be executed by the UAV according to the state information Sk. Then, the transition information Sk,Ak,rk,S′,l is stored in B. If the UAV flies beyond the target area, the action will be canceled and the boundary penalty information will be given. In the data collection stage, we remain βk=0. The data offloading stage begins when UAV completes the data collection mission. Then, we remain αk=0. l=1 if the UAV completes the data offloading mission. The episode is terminated when l=1 or k=Kmax.Figure 2
TD3-based UAV trajectory optimization.In lines 18-28,N transition information is randomly selected from B to form a minibatch, which is input into the actor and critic network. The actor network calculates the corresponding A~ according to S′. After selecting the target Q-value and smoothing the target policy according to S′ and A~, the critic network minimizes the loss function and updates the critical network by the following way:
(20)θi⟵argminθiN−1∑y−QθiSk,Ak2.Then, the actor network is updated in the way of delayed update by the deterministic policy gradient, expressed as(21)∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.Finally, the optimal trajectory of UAV is obtained by cyclic iteration until the maximum number of episodesEmax.Algorithm 1: TD3-based trajectory optimization algorithm (TTO).
1. Initialize critic networksQθ1, Qθ2, and actor network πϕ with random parameters θ1, θ2, and ϕ.2. Initialize target networksθ1′⟵θ1, θ2′⟵θ2, and ϕ′⟵ϕ.3. Initialize experience replay bufferB.4.for episode =0 toEmaxdo5.  Initialize the environment and stateS0, and the terminated flag l=0.6.for epoch k=1toKmaxdo7.   Select actionAk=πϕSk+ε, ε~N0,ω, and observe reward rk and next state S′.8.if the UAV flies beyond the target area then9.ρk=1. Then cancel the UAV’s action and updaterk, S′ based on the current state.10.end if11.if∑k=1KcoRm,kδ≥Cm,∀m∈Mthen12.    letαk=0, and start the data offloading.13.else14.    Letβk=0, and continue the data collection.15.end if16.   if∑k=k∗KRkδ≥∑m=1MCmthen17.rk=Rkδ+ρk+K∗, and let l=1.18.end if19.   Store transition tupleSk,Ak,rk,S′,l in B.20.ifB>2,000then21.    Sample mini-batch ofN transitions Sk,Ak,rk,S′,l from B.22.A~⟵πϕ′S′+ε, ε~ clipN0,ω~,−c,c.23.y⟵rk+1−l·γmini=1,2Qθi′S′,A~.24.    Update critics:25.θi⟵argminθiN−1∑y−QθiSk,Ak2.26.    Update the actor policyϕ by the deterministic policy gradient:27.∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.28.    Update target networks:29.θi′⟵τθi+1−τθi′.30.ϕ′⟵τϕ+1−τϕ′.31.end if32.end for33.end for34.return The UAV trajectory Q


### 3.2. Power Control and Buoy-UAV Association Relationship

Given the UAV trajectory, the problemP1 can be written as
(22)P2:minAm,P,Pm,QTtotal,s.t.C1−C7,C10,C11Obviously, the problemP2 is still a mixed-integer nonconvex problem. Given the mission completion time and the association relationships of buoy-UAV and UAV-OBS, P2 can be transformed into a problem of maximizing the total transmission data size in the k-th time slot, which can be divided into two parts.First, in the data collection stage, it can be seen from problemP2 that the SINR between UAV and buoys is related not only to the transmit power of buoys but also to the buoy-UAV association relationship. Therefore, in order to determine the buoy-UAV association relationship, we first introduce Lemmas 1 and 2.Lemma 1.
The UAV must be associated with the firstU∗U∗≤U buoys with the larger channel gain in the k-th time slot.Proof.
As can be seen fromP2a, the total transmission data size Ck depends on the summation term ∑m=1UPmhm. Therefore, we might as well assume that the transmit power of all buoys is the maximum transmission power Pmmax. The channel gain between M buoys and UAV is expressed in descending order as h1⟶h2⟶⋯⟶hM. Obviously, selecting the first U∗ buoys with the largest channel gain can maximize the total transmission data size.Lemma 2.
The transmit power of the buoy with the largest channel gain among the buoys associated with the UAV in thek-th time slot must be Pmmax.Proof.
Except for the buoys transmit power, other assumptions are the same as Proof. If the UAV is associated withU buoys, for the buoy with the largest channel gain, the SINR between it and the UAV is expressed as
(23)g1=P1h1∑i=2UPihi+σ2.
In order to meet the constraintsC10 and C12 and maximize the total transmission data size, the value of denominator term P1h1 should be as large as possible, so as to maximize the value of molecular term and make more buoys connected to UAV. Therefore, P1 should be Pmmax.
The total transmission data size ofU∗ buoys in the k-th time slot is expressed as
(24)Ck=∑m=1U∗Blog21+Pm,khm,k∑i∈MkPi,khi,k+σ2δ.
Therefore, problemP2 can be transformed into
(25)P2a:maxPmCk,s.t.C1:0≤Pm≤Pmmax,C10:Pm,khm,k∑i∈MkPi,khi,k+σ2≥ηSIC
Due to the existence of cochannel interference between buoys,P2a is still nonconvex. Therefore, we convert Ck into the following form:
(26)Ck=Bδlog21+∑m=1U∗Pm,khm,kσ2.
Therefore,P2a is a convex problem that can be solved by a standard convex optimization solver (such as cvxpy).Algorithm 2: Power control and buoy-UAV association relationship algorithm (PCAR).
1. Input the UAV’s current positionqk, ηSIC and U.2. Input current channel gaingm,k, and sort it in descending order to obtain g′m,k.3. According toU, get initial NOMA group and the number of initial associated buoys U∗=U.4.repeat5.  Solve P2a to obtain the optimal solutionPm∗ and the optimal value Ck.6.if the solution state is not optimal then7.   Remove the one with the worst channel gain in the currently associated buoy.8.U∗⟵U∗−1.9.end if10.untilU∗=1.11.returnPm∗ and Am;The algorithm PCAR is shown in Algorithm2. In Algorithm 2, the channel gains are first sorted in descending order. It is stipulated that the UAV is associated with U buoys at most, and the first U buoys with the largest channel gain are selected to form the initial NOMA group. Then, we solve P2a. If P2a has an optimal solution, the optimal solution Pm and the optimal value Ck are obtained. If P2a has no optimal solution, the buoy with the worst channel gain is removed from the current NOMA group to form a new NOMA group. The above process is repeated until P2a has an optimal solution. Note that the UAV is associated with at least one buoy in each time slot.Algorithm 3: Joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme (TTO-PCAR).
Input: The UAV’s initial position q1, the buoys’ position Dm, the OBS’s position D0;Output: Am, P, Pm, Q;1.for episode =0 toEmaxdo2.for epoch k=1toKmaxdo3.   /∗Lines 11-15 of Algorithm 1∗/4.if∑k=1KcoRm,kδ≥Cm,∀m∈Mthen5.    Letαk=0, and obtain current channel gain g0,k.6.    SetP=Pmax.7.ifg0,k≥g¯0then8.     Let UAV-OBS association relationshipβk=1.9.end if10.else11.    Letβk=0, and obtain current channel gain gm,k.12.    UpdatePm,k,∀m∈M and αk with given qk by performing Algorithm 2.13./∗Lines 7-10 of Algorithm 1∗/14.    Updateqk with given tranmist power and association relationship.15.end if16.end for17.end forSecond, in the data offloading stage, the total transmission data size of UAV in thek-th time slot is expressed as
(27)Ckuav=Bδlog21+Pkh0,kσ2.Then,P2 can be transformed into the following form:
(28)P2b:maxPCkuav,s.t.C2:0≤Pk≤Pmax,C11:Pkh0,kσ2≥g¯0ProblemP2b is a standard convex problem, the optimal solution of which is Pmax.In summary, we propose a joint TTO and PCAR scheme (TTO-PCAR) to solve the problemP1. The TD3 agent is deployed in the OBS, and OBS maintains the communication with the UAV. During training, UAV collects data from buoys through traffic channel. Meanwhile, the UAV receives the states information of buoys through the control channel and feeds back the states information of itself and buoys to OBS. The OBS operates the proposed scheme with the above states information and sends the operation results to UAV in each time slot. Then, the UAV forwards relevant signalling (such as transmit power of buoys and buoy-UAV association relationship) to the buoy through the control channel. The specific process is shown in Figure 3 and Algorithm 3. Specifically, the UAV initial position is first given. In lines 3-12, P and Am are obtained by Algorithm 2 according to the UAV current position qk in the data collection stage. βk is obtained according to qk and g¯0 in the data offloading stage. Then, the UAV next position information is updated by lines 7-10 of Algorithm 1. Finally, the above process is repeated until Emax.Figure 3
Joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme.


### 3.3. Complexity Analysis

TD3 contains two actor networks and four critic networks. Hence, the computational complexity of Algorithm1 is O2∑e=1Eanaenae−1+4∑e=1Ecncence−1. Ea is the number of the fully connected layers of actor network. Ec is the number of the fully connected layers of critic network. na and nc are the unit numbers in the e-th layer of the actor network and the critic network, respectively. Since the UAV is associated with at most U buoys in each time slot, the computational complexity of Algorithm 2 is OU. Hence, the computational complexity of Algorithm 3 is OEmaxKmax∑e=1Eanaenae−1+∑e=1Ecncence−1+U.


## 3.1. TD3-Based UAV Trajectory Optimization

In our system, the start position of data offloading stage is similar to the end position of data collection, which is named as the transition position (TP) between two stages. Therefore, the UAV trajectories of these two stages are coupled, and the TP cannot be determined in advance. Furthermore, the UAV trajectory changes dynamically according to the requirements of data collection and data offloading. The traditional deterministic optimization method is difficult to solve the above problems [19]. Therefore, in this paper, we use an advanced DRL method, TD3, to solve the UAV trajectory optimization subproblem with the given transmit powers and association relationships. In the following, we first give the definitions of state, action, and reward and then briefly introduce the TD3.


### 3.1.1. State Definition

In the data collection stage, the UAV’s action is closely related to the remaining data size of buoys and the buoy-UAV association relationship. Similarly, in the data offloading stage, the UAV’s action is related to the remaining data size of UAV and the UAV-OBS association relationship. Furthermore, the UAV only collects data from buoys in the target area. Hence, we define the state space as(12)Sk=αk,ck,xk,yk,βk,ckuav,ρk,where the variables contained in the above expression are defined as
(i)
αk=α1,k,α2,k,⋯,αM,k denotes the set of the buoy-UAV association relationship in the k-th time slot(ii)
ck=c1,k,c2,k,⋯,cM,k denotes the remaining data size of each buoy in the k-th time slot(13)cm,k=Cm−∑j=1k−1Rm,jδ(iii)
ckuav denotes the remaining data size of UAV(14)ckuav=∑k=1Kco∑m=1MRm,kδ−∑j=Kco+1k−1R0,jδ(iv)
ρk denotes the boundary penalty information of the UAV to judge whether the UAV position exceeds the target area in the k-th time slot


### 3.1.2. Action Definition

Based on the above state and environment information, the UAV’s action is defined as(15)Ak=φk,Vk,where φk∈0,2π denotes the flight angle of UAV in the k-th time slot and Vk∈0,Vmax.


### 3.1.3. Reward Definition

LetKmax denote the number of maximum total mission completion time slots. K∗=Kmax−K if the UAV completes the mission in K time slots. Then, we design the reward function as
(16)rk=Rkδ+ρk+K∗,ifC6andC7aresatisfied,CkorRkδ+ρk,otherwise.It can be seen from (16) that the shorter the time it takes for the UAV to complete the mission, the greater the reward it will eventually obtain.Note that for better performance, we reduce the order of magnitude ofB and Cm by n1 orders of magnitude to be less than or equal to the order of magnitude of Kmax when calculating the state and reward.


### 3.1.4. TD3

The TD3 has the following advantages [27]:
(i)
Clipped doubleQ-learning for actor-critic: TD3 contains two critic networks. For the two target Q-values generated by the two critic target networks, the minimum of them is selected to suppress the overestimation problem caused by high variance, expressed as(17)y=rk+γmini=1,2Qθi′S′,A~where y is the target value which is used to update the two critic networks, r is reward, γ is the reward discount factor, A~ is the target policy, θi′ is the target network parameter, and S′ is the next state
(ii)
Target networks and delayed policy updates: the TD3 algorithm updates the actor and its target network after a fixed number of updates to the critic network, expressed as(18)θi′⟵τθi+1−τθi′,i=1,2where τ is the update parameter
(iii)
Target policy smoothing regularization: TD3 smoothes the estimate and reduces the error by adding a small amount of random noise to the target actor network and averaging over minibatch, expressed as(19)A~⟵πϕ′Sk+ε,ε~clipN0,ω~,−c,cIt can be seen from (12) that most of the state dimensions are related to buoys. Only two dimensions are related to the UAV’s position, two dimensions are related to OBS, and one dimension is related to UAV boundary penalty information. Therefore, there is a problem of dimension imbalance. Dimension spread technology can effectively solve this problem. We spread the above state dimensions. For example, we connect the position state dimension of UAV to a spread network composed of M neurons and spread its dimension to M [33, 34]. Furthermore, we set a termination flag l to indicate whether UAV has completed its mission. l is applied to the target value function. Hence, the Q-value of the target value function is 0 after the UAV completes the mission, so as to make the critic learning performance more stable [34].In summary, our proposed algorithm TTO is shown in Algorithm1. The update process of TD3 is shown in Figure 2. In lines 1-3, we first initialize the network parameters and the experience replay buffer B. In lines 5-17, we initialize the environment, obtain the initial state information, and set a time variable k to represent the time spent by UAV to perform the mission. Moreover, a termination mark l is set. l=0 indicates that UAV has not completed the mission. Then, UAV makes an action selection according to the observed state and environmental information. Specifically, UAV constantly interacts with the environment and updates the actor and critic networks. The actor network outputs the action Ak to be executed by the UAV according to the state information Sk. Then, the transition information Sk,Ak,rk,S′,l is stored in B. If the UAV flies beyond the target area, the action will be canceled and the boundary penalty information will be given. In the data collection stage, we remain βk=0. The data offloading stage begins when UAV completes the data collection mission. Then, we remain αk=0. l=1 if the UAV completes the data offloading mission. The episode is terminated when l=1 or k=Kmax.Figure 2
TD3-based UAV trajectory optimization.In lines 18-28,N transition information is randomly selected from B to form a minibatch, which is input into the actor and critic network. The actor network calculates the corresponding A~ according to S′. After selecting the target Q-value and smoothing the target policy according to S′ and A~, the critic network minimizes the loss function and updates the critical network by the following way:
(20)θi⟵argminθiN−1∑y−QθiSk,Ak2.Then, the actor network is updated in the way of delayed update by the deterministic policy gradient, expressed as(21)∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.Finally, the optimal trajectory of UAV is obtained by cyclic iteration until the maximum number of episodesEmax.Algorithm 1: TD3-based trajectory optimization algorithm (TTO).
1. Initialize critic networksQθ1, Qθ2, and actor network πϕ with random parameters θ1, θ2, and ϕ.2. Initialize target networksθ1′⟵θ1, θ2′⟵θ2, and ϕ′⟵ϕ.3. Initialize experience replay bufferB.4.for episode =0 toEmaxdo5.  Initialize the environment and stateS0, and the terminated flag l=0.6.for epoch k=1toKmaxdo7.   Select actionAk=πϕSk+ε, ε~N0,ω, and observe reward rk and next state S′.8.if the UAV flies beyond the target area then9.ρk=1. Then cancel the UAV’s action and updaterk, S′ based on the current state.10.end if11.if∑k=1KcoRm,kδ≥Cm,∀m∈Mthen12.    letαk=0, and start the data offloading.13.else14.    Letβk=0, and continue the data collection.15.end if16.   if∑k=k∗KRkδ≥∑m=1MCmthen17.rk=Rkδ+ρk+K∗, and let l=1.18.end if19.   Store transition tupleSk,Ak,rk,S′,l in B.20.ifB>2,000then21.    Sample mini-batch ofN transitions Sk,Ak,rk,S′,l from B.22.A~⟵πϕ′S′+ε, ε~ clipN0,ω~,−c,c.23.y⟵rk+1−l·γmini=1,2Qθi′S′,A~.24.    Update critics:25.θi⟵argminθiN−1∑y−QθiSk,Ak2.26.    Update the actor policyϕ by the deterministic policy gradient:27.∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.28.    Update target networks:29.θi′⟵τθi+1−τθi′.30.ϕ′⟵τϕ+1−τϕ′.31.end if32.end for33.end for34.return The UAV trajectory Q


## 3.1.1. State Definition

In the data collection stage, the UAV’s action is closely related to the remaining data size of buoys and the buoy-UAV association relationship. Similarly, in the data offloading stage, the UAV’s action is related to the remaining data size of UAV and the UAV-OBS association relationship. Furthermore, the UAV only collects data from buoys in the target area. Hence, we define the state space as(12)Sk=αk,ck,xk,yk,βk,ckuav,ρk,where the variables contained in the above expression are defined as
(i)
αk=α1,k,α2,k,⋯,αM,k denotes the set of the buoy-UAV association relationship in the k-th time slot(ii)
ck=c1,k,c2,k,⋯,cM,k denotes the remaining data size of each buoy in the k-th time slot(13)cm,k=Cm−∑j=1k−1Rm,jδ(iii)
ckuav denotes the remaining data size of UAV(14)ckuav=∑k=1Kco∑m=1MRm,kδ−∑j=Kco+1k−1R0,jδ(iv)
ρk denotes the boundary penalty information of the UAV to judge whether the UAV position exceeds the target area in the k-th time slot


## 3.1.2. Action Definition

Based on the above state and environment information, the UAV’s action is defined as(15)Ak=φk,Vk,where φk∈0,2π denotes the flight angle of UAV in the k-th time slot and Vk∈0,Vmax.


## 3.1.3. Reward Definition

LetKmax denote the number of maximum total mission completion time slots. K∗=Kmax−K if the UAV completes the mission in K time slots. Then, we design the reward function as
(16)rk=Rkδ+ρk+K∗,ifC6andC7aresatisfied,CkorRkδ+ρk,otherwise.It can be seen from (16) that the shorter the time it takes for the UAV to complete the mission, the greater the reward it will eventually obtain.Note that for better performance, we reduce the order of magnitude ofB and Cm by n1 orders of magnitude to be less than or equal to the order of magnitude of Kmax when calculating the state and reward.


## 3.1.4. TD3

The TD3 has the following advantages [27]:
(i)
Clipped doubleQ-learning for actor-critic: TD3 contains two critic networks. For the two target Q-values generated by the two critic target networks, the minimum of them is selected to suppress the overestimation problem caused by high variance, expressed as(17)y=rk+γmini=1,2Qθi′S′,A~where y is the target value which is used to update the two critic networks, r is reward, γ is the reward discount factor, A~ is the target policy, θi′ is the target network parameter, and S′ is the next state
(ii)
Target networks and delayed policy updates: the TD3 algorithm updates the actor and its target network after a fixed number of updates to the critic network, expressed as(18)θi′⟵τθi+1−τθi′,i=1,2where τ is the update parameter
(iii)
Target policy smoothing regularization: TD3 smoothes the estimate and reduces the error by adding a small amount of random noise to the target actor network and averaging over minibatch, expressed as(19)A~⟵πϕ′Sk+ε,ε~clipN0,ω~,−c,cIt can be seen from (12) that most of the state dimensions are related to buoys. Only two dimensions are related to the UAV’s position, two dimensions are related to OBS, and one dimension is related to UAV boundary penalty information. Therefore, there is a problem of dimension imbalance. Dimension spread technology can effectively solve this problem. We spread the above state dimensions. For example, we connect the position state dimension of UAV to a spread network composed of M neurons and spread its dimension to M [33, 34]. Furthermore, we set a termination flag l to indicate whether UAV has completed its mission. l is applied to the target value function. Hence, the Q-value of the target value function is 0 after the UAV completes the mission, so as to make the critic learning performance more stable [34].In summary, our proposed algorithm TTO is shown in Algorithm1. The update process of TD3 is shown in Figure 2. In lines 1-3, we first initialize the network parameters and the experience replay buffer B. In lines 5-17, we initialize the environment, obtain the initial state information, and set a time variable k to represent the time spent by UAV to perform the mission. Moreover, a termination mark l is set. l=0 indicates that UAV has not completed the mission. Then, UAV makes an action selection according to the observed state and environmental information. Specifically, UAV constantly interacts with the environment and updates the actor and critic networks. The actor network outputs the action Ak to be executed by the UAV according to the state information Sk. Then, the transition information Sk,Ak,rk,S′,l is stored in B. If the UAV flies beyond the target area, the action will be canceled and the boundary penalty information will be given. In the data collection stage, we remain βk=0. The data offloading stage begins when UAV completes the data collection mission. Then, we remain αk=0. l=1 if the UAV completes the data offloading mission. The episode is terminated when l=1 or k=Kmax.Figure 2
TD3-based UAV trajectory optimization.In lines 18-28,N transition information is randomly selected from B to form a minibatch, which is input into the actor and critic network. The actor network calculates the corresponding A~ according to S′. After selecting the target Q-value and smoothing the target policy according to S′ and A~, the critic network minimizes the loss function and updates the critical network by the following way:
(20)θi⟵argminθiN−1∑y−QθiSk,Ak2.Then, the actor network is updated in the way of delayed update by the deterministic policy gradient, expressed as(21)∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.Finally, the optimal trajectory of UAV is obtained by cyclic iteration until the maximum number of episodesEmax.Algorithm 1: TD3-based trajectory optimization algorithm (TTO).
1. Initialize critic networksQθ1, Qθ2, and actor network πϕ with random parameters θ1, θ2, and ϕ.2. Initialize target networksθ1′⟵θ1, θ2′⟵θ2, and ϕ′⟵ϕ.3. Initialize experience replay bufferB.4.for episode =0 toEmaxdo5.  Initialize the environment and stateS0, and the terminated flag l=0.6.for epoch k=1toKmaxdo7.   Select actionAk=πϕSk+ε, ε~N0,ω, and observe reward rk and next state S′.8.if the UAV flies beyond the target area then9.ρk=1. Then cancel the UAV’s action and updaterk, S′ based on the current state.10.end if11.if∑k=1KcoRm,kδ≥Cm,∀m∈Mthen12.    letαk=0, and start the data offloading.13.else14.    Letβk=0, and continue the data collection.15.end if16.   if∑k=k∗KRkδ≥∑m=1MCmthen17.rk=Rkδ+ρk+K∗, and let l=1.18.end if19.   Store transition tupleSk,Ak,rk,S′,l in B.20.ifB>2,000then21.    Sample mini-batch ofN transitions Sk,Ak,rk,S′,l from B.22.A~⟵πϕ′S′+ε, ε~ clipN0,ω~,−c,c.23.y⟵rk+1−l·γmini=1,2Qθi′S′,A~.24.    Update critics:25.θi⟵argminθiN−1∑y−QθiSk,Ak2.26.    Update the actor policyϕ by the deterministic policy gradient:27.∇ϕJϕ=N−1∑∇AkQθ1Sk,AkAk=πϕSk∇ϕπϕSk.28.    Update target networks:29.θi′⟵τθi+1−τθi′.30.ϕ′⟵τϕ+1−τϕ′.31.end if32.end for33.end for34.return The UAV trajectory Q


## 3.2. Power Control and Buoy-UAV Association Relationship

Given the UAV trajectory, the problemP1 can be written as
(22)P2:minAm,P,Pm,QTtotal,s.t.C1−C7,C10,C11Obviously, the problemP2 is still a mixed-integer nonconvex problem. Given the mission completion time and the association relationships of buoy-UAV and UAV-OBS, P2 can be transformed into a problem of maximizing the total transmission data size in the k-th time slot, which can be divided into two parts.First, in the data collection stage, it can be seen from problemP2 that the SINR between UAV and buoys is related not only to the transmit power of buoys but also to the buoy-UAV association relationship. Therefore, in order to determine the buoy-UAV association relationship, we first introduce Lemmas 1 and 2.Lemma 1.
The UAV must be associated with the firstU∗U∗≤U buoys with the larger channel gain in the k-th time slot.Proof.
As can be seen fromP2a, the total transmission data size Ck depends on the summation term ∑m=1UPmhm. Therefore, we might as well assume that the transmit power of all buoys is the maximum transmission power Pmmax. The channel gain between M buoys and UAV is expressed in descending order as h1⟶h2⟶⋯⟶hM. Obviously, selecting the first U∗ buoys with the largest channel gain can maximize the total transmission data size.Lemma 2.
The transmit power of the buoy with the largest channel gain among the buoys associated with the UAV in thek-th time slot must be Pmmax.Proof.
Except for the buoys transmit power, other assumptions are the same as Proof. If the UAV is associated withU buoys, for the buoy with the largest channel gain, the SINR between it and the UAV is expressed as
(23)g1=P1h1∑i=2UPihi+σ2.
In order to meet the constraintsC10 and C12 and maximize the total transmission data size, the value of denominator term P1h1 should be as large as possible, so as to maximize the value of molecular term and make more buoys connected to UAV. Therefore, P1 should be Pmmax.
The total transmission data size ofU∗ buoys in the k-th time slot is expressed as
(24)Ck=∑m=1U∗Blog21+Pm,khm,k∑i∈MkPi,khi,k+σ2δ.
Therefore, problemP2 can be transformed into
(25)P2a:maxPmCk,s.t.C1:0≤Pm≤Pmmax,C10:Pm,khm,k∑i∈MkPi,khi,k+σ2≥ηSIC
Due to the existence of cochannel interference between buoys,P2a is still nonconvex. Therefore, we convert Ck into the following form:
(26)Ck=Bδlog21+∑m=1U∗Pm,khm,kσ2.
Therefore,P2a is a convex problem that can be solved by a standard convex optimization solver (such as cvxpy).Algorithm 2: Power control and buoy-UAV association relationship algorithm (PCAR).
1. Input the UAV’s current positionqk, ηSIC and U.2. Input current channel gaingm,k, and sort it in descending order to obtain g′m,k.3. According toU, get initial NOMA group and the number of initial associated buoys U∗=U.4.repeat5.  Solve P2a to obtain the optimal solutionPm∗ and the optimal value Ck.6.if the solution state is not optimal then7.   Remove the one with the worst channel gain in the currently associated buoy.8.U∗⟵U∗−1.9.end if10.untilU∗=1.11.returnPm∗ and Am;The algorithm PCAR is shown in Algorithm2. In Algorithm 2, the channel gains are first sorted in descending order. It is stipulated that the UAV is associated with U buoys at most, and the first U buoys with the largest channel gain are selected to form the initial NOMA group. Then, we solve P2a. If P2a has an optimal solution, the optimal solution Pm and the optimal value Ck are obtained. If P2a has no optimal solution, the buoy with the worst channel gain is removed from the current NOMA group to form a new NOMA group. The above process is repeated until P2a has an optimal solution. Note that the UAV is associated with at least one buoy in each time slot.Algorithm 3: Joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme (TTO-PCAR).
Input: The UAV’s initial position q1, the buoys’ position Dm, the OBS’s position D0;Output: Am, P, Pm, Q;1.for episode =0 toEmaxdo2.for epoch k=1toKmaxdo3.   /∗Lines 11-15 of Algorithm 1∗/4.if∑k=1KcoRm,kδ≥Cm,∀m∈Mthen5.    Letαk=0, and obtain current channel gain g0,k.6.    SetP=Pmax.7.ifg0,k≥g¯0then8.     Let UAV-OBS association relationshipβk=1.9.end if10.else11.    Letβk=0, and obtain current channel gain gm,k.12.    UpdatePm,k,∀m∈M and αk with given qk by performing Algorithm 2.13./∗Lines 7-10 of Algorithm 1∗/14.    Updateqk with given tranmist power and association relationship.15.end if16.end for17.end forSecond, in the data offloading stage, the total transmission data size of UAV in thek-th time slot is expressed as
(27)Ckuav=Bδlog21+Pkh0,kσ2.Then,P2 can be transformed into the following form:
(28)P2b:maxPCkuav,s.t.C2:0≤Pk≤Pmax,C11:Pkh0,kσ2≥g¯0ProblemP2b is a standard convex problem, the optimal solution of which is Pmax.In summary, we propose a joint TTO and PCAR scheme (TTO-PCAR) to solve the problemP1. The TD3 agent is deployed in the OBS, and OBS maintains the communication with the UAV. During training, UAV collects data from buoys through traffic channel. Meanwhile, the UAV receives the states information of buoys through the control channel and feeds back the states information of itself and buoys to OBS. The OBS operates the proposed scheme with the above states information and sends the operation results to UAV in each time slot. Then, the UAV forwards relevant signalling (such as transmit power of buoys and buoy-UAV association relationship) to the buoy through the control channel. The specific process is shown in Figure 3 and Algorithm 3. Specifically, the UAV initial position is first given. In lines 3-12, P and Am are obtained by Algorithm 2 according to the UAV current position qk in the data collection stage. βk is obtained according to qk and g¯0 in the data offloading stage. Then, the UAV next position information is updated by lines 7-10 of Algorithm 1. Finally, the above process is repeated until Emax.Figure 3
Joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme.


## 3.3. Complexity Analysis

TD3 contains two actor networks and four critic networks. Hence, the computational complexity of Algorithm1 is O2∑e=1Eanaenae−1+4∑e=1Ecncence−1. Ea is the number of the fully connected layers of actor network. Ec is the number of the fully connected layers of critic network. na and nc are the unit numbers in the e-th layer of the actor network and the critic network, respectively. Since the UAV is associated with at most U buoys in each time slot, the computational complexity of Algorithm 2 is OU. Hence, the computational complexity of Algorithm 3 is OEmaxKmax∑e=1Eanaenae−1+∑e=1Ecncence−1+U.


## 4. Simulation Results

In simulation, the considered target area is1000m×1000m where M=10 buoys are randomly distributed. A UAV is used to collect data with a fixed height H=100 m from the target area. The flight velocity of UAV is Vmax=50 m/s and Vmin=0 m/s. The maximum transmit power of UAV and buoys is Pmax=0.1 W and Pmmax=24 dBm, respectively. The position of OBS is D0=0,0 m. The UAV is allowed to associate up to 3 buoys in each time slot, i.e., U=3. The time slot length is δ=1 s. The data size range of each buoy is Cm∈10,20 Mbits. The spectrum is B=1 MHz. Furthermore, our proposed algorithm TTO is based on Pytorch. For actor and critic networks, we use a fully connected DNN with two hidden layers of 400 neurons. The learning rate is 0.0001. The experience memory buffer size is 100000. The minibatch size N is 256. The discount factor γ is 0.99. ω~=0.36. τ=0.005. Kmax=300. Other simulation parameters are shown in Table 1.Table 1
Simulation parameters.
ParameterValueMaximum acceleration,Δmax25m/s2Environment parameter,a, b9.61, 0.16Excessive path loss,ξLoS, ξNLoS1, 20Channel parameter,μm,k1The antenna heights of buoys and OBS,Hm, H00 m, 0 mWavelength,λ0.15 mNoise power,σ2-94 dBmSINR and SNR threshold,g¯co, g¯of10 dB, 3 dBSIC threshold,ηSIC10 dBOrder of magnitude parameter,n6In order to compare performance, we use the following scheme as the comparison algorithm.(i)
UAV trajectory based on Fermat point (FTP) [35]: this scheme first regards each user as the vertex of a triangle to form multiple triangles. Then, the Fermat points of each triangle are taken as the hovering points of the UAV. The UAV hovers at the points in turn to collect data(ii)
UAV trajectory based on circle scheme (CTS): this scheme first finds the geometric center of all users as the center of the circle and then averages the distance from all users to the center of the circle to determine the radius of the UAV trajectory(iii)
UAV data collection based on OMA (TTO-OMA): this scheme refers to the UAV using OMA technology for data collection. The proposed TTO algorithm is still used to determine the UAV trajectory(iv)
DRL scheme based on DQN (DTO-PCAR): this scheme uses DQN instead of TD3 in our proposed algorithmFigure4 shows the comparison of accumulative reward for different schemes. For the convenience of observation, we smoothed the curves. It can be seen that the proposed TTO-PCAR scheme could be convergent after 1000 episodes, while the compared TTO-OMA scheme needs 3000 episodes to be convergent. Moreover, the compared DTO-PCAR scheme cannot be convergent after 6000 episodes. Therefore, the performance of our proposed scheme is significantly better than the other two schemes. Figure 5 shows the UAV trajectory comparison with TTO-PCAR scheme under different SIC thresholds. The SIC thresholds are 10 dB, 12 dB, and 15 dB, respectively. We find that the average total mission completion time of UAV is basically the same, which is 33 s, 36 s, and 37 s, respectively. However, the UAV trajectory is closer to the farther buoy with the increase of SIC threshold. This is because when the UAV uses NOMA technology for data collection, the channel gain of the farther buoy is poor. Therefore, in order to meet the SIC constraint, the UAV will gradually fly to the farther buoys whose data has not yet been collected.Figure 4
Accumulative reward.Figure 5
UAV trajectory with different SIC thresholds.Figure6 shows the comparison of the total mission completion time of under different spectrums. Figure 7 shows the UAV trajectory obtained by our proposed scheme with M=10 buoys and compares it with the other three schemes. It can be seen that the total mission completion time of TTO-PCAR is significantly lower than that of other schemes. In particular, the data collection time of our proposed scheme is 20 s with B=1 MHz and that of TTO-OMA is 33 s; thus, NOMA is more efficient in data collection than OMA. This is because the designed reward (shown in Equation (16)) is related to the total transmission rate in each time slot. Second, the trajectory of UAV data collection process is fixed with the FTP and CTS scheme, resulting in the UAV trajectory of data offloading process longer. TTO-PCAR scheme takes the coupling of two stages into the consideration of UAV trajectory optimization; thus, the time of data offloading process is less. The total flight distance based on TTO-PCAR is also significantly lower than that of the other two schemes.Figure 6
Total mission completion time with different schemes.Figure 7
UAV trajectory of different schemes.Figure8 shows the UAV trajectory based on TTO-PCAR with different spectrums. It can be seen from Figure 8 that the flight distance of UAV decreases with the increase of spectrum. This is because the transmission rate of buoys is reduced with the reduction of spectrum. If the data of the buoy far from the OBS has not been collected, the agent chooses to make the UAV closer to the buoy in order to increase the transmission rate and obtain greater reward according to (16).Figure 8
UAV trajectory with different spectrums.Figure9 shows the total mission completion time of different schemes with different buoy numbers. FTP scheme is to find the hover points to collect data and classify the problem as a travelling salesman problem, so as to traverse the hover points. Hence, FTP takes a lot of time on UAV flight. Although CTS scheme can collect data in each time slot, it does not consider the data collection requirements of different buoys, because the UAV just flies based on circle. The proposed scheme TTO-PCAR dynamically adjusts the UAV trajectory according to the data collection requirements of different buoys. Therefore, the total mission completion time of TTO-PCAR is significantly lower than that of FTP and CTS.Figure 9
Total mission completion time with different buoy numbers.


## 5. Conclusion

This paper has investigated the joint optimization problem of the buoy-UAV association relationship, transmit powers, and the UAV trajectory for NOMA-enabled UAV data collection and offloading in MIoT. First, we propose a TD3-based UAV trajectory optimization algorithm to solve the UAV trajectory subproblem. Second, we design a heuristic algorithm to solve the subproblem of power control and buoy-UAV association relationship. Finally, we propose a joint TD3-based trajectory optimization, power control, and buoy-UAV association relationship scheme. The proposed scheme can effectively solve the mixed-integer nonconvex problem. Simulation results show that the proposed scheme significantly shortens the total mission completion time of UAV. In future work, we will investigate the problem of UAV trajectory optimization based on NOMA to shorten the time for UAV to perform mission in the MIoT.



---

*Source: 8805416-2022-05-12.xml*