# CSO-DRL: A Collaborative Service Offloading Approach with Deep Reinforcement Learning in Vehicular Edge Computing

Authors: Yuze Huang; Yuhui Cao; Miao Zhang; Beipeng Feng; Zhenzhen Guo
Journal: Scientific Programming
 (2022)

---

## Abstract

In vehicular edge computing, vehicles move along the road and request the services from the nearest edge servers with low latency. Due to the limitation of computation capacity of vehicular devices, the services should be offloaded on RSUs equipped with edge servers to provide service with low latency. Noticed that the location of service offloading may affect the service requesting delay directly, and it may exist some interrelationship between interacting services﻿; all of these are rarely considered in recent studies. To address such problems, we propose a collaborative service offloading approach with deep reinforcement learning in vehicular edge computing named CSO-DRL. Our approach first divides the road segments by k-means-based algorithm through analyzing the trajectory data of vehicles, and then the offloading location is determined by observing the vehicle running status. Secondly, the interacting services are discovered by a parallel frequent pattern-based algorithm efficiently. Furthermore, the collaborative service offloading algorithm is presented by the DDPG model for offloading the interacting services, which can minimize the service requesting delay and data communication delay between interacting services. Finally, the efficiency of the algorithm is evaluated by real-world data-based simulation experimental evaluations. The results show our algorithm can obtain a lower delay than other baseline algorithms in searching for the optimal service offloading strategy.


---

## Body

## 1. Introduction

With the development of intelligent transportation technologies, the internet of vehicles (IoV) has raised public attention and applications in the urban transportation system, which can be regarded as the evolution of vehicular ad hoc networks. In IoV, the vehicles equipped with intelligent devices are capable of vehicles to everything communication (V2X) through the wireless network [1]. Through the V2X communication, the vehicles can realize the information interaction among vehicles, roadside units (RSUs), and humans to assist the decision of auto-driving and avoid the traffic accidents [2]. In the IoV-based intelligent transportation system, the data including vehicles status information and traffic information are collected by numerous sensors for analysis and computation [3]. Since the vehicles have limited computation capability to execute complex computation tasks, the most direct approach is transmitting the traffic data to a cloud data center for computing. Generally speaking, a cloud data center is far away from the vehicles, which may bring some problems such as network congestion, time-consuming, and data privacy leaking [4]. With the complexity of urban traffic, the response time of cloud service will be increasingly difficult to satisfy the real-time decision for auto-driving. To solve such problems, edge computing regarded as the supplement of traditional cloud computing is used in IoV, which can efficiently provide service to users with low latency [5, 6].In edge computing, the services are offloaded on edge servers (ESs) in close proximity to users, which mainly undertake the capacity of data computing and most services executions. While the intelligent devices are only responsible for the services of raw data preprocessing and few noncomputation-intensive services executions, and then only a small amount of intelligent computation services are delivered to remote cloud servers for execution [7–9]. As an efficient approach, service offloading (also called computation offloading) can address the problems of insufficient devices and the high latency for cloud services, which has raised public attention in edge computing [10–12]. During the service offloading, the users’ experience may be greatly affected by service requesting delays and the energy consumption of intelligent devices [13, 14]. In such conditions, some offloading approaches are produced to minimize the latency and energy, but some defections are revealed in IoV service offloading.In vehicular edge computing, when the vehicles move along the roads, the vehicles request the services from the nearest edge server to gain low-latency services. Due to the coverage limitation of the RSUs equipped with edge servers, when the vehicles reach the boundaries of edge servers, the edge servers must be switched to guarantee the continuity of service, so the selection of edge servers may influence the delay of service requesting to a large extent [15]. Thus, the speed and location of vehicles must be considered in service offloading. Moreover, the interrelationship between services is another factor that must be considered in service offloading. Many services collaborate with different services to complete a certain complex business goal, which may produce lots of transmission data among the interacting services. If the interacting services are offloaded separately, the data communication delay between them will be increased. As the number of data increases exponentially, the communication delay between interacting services becomes a nonignored problem for service offloading. Throughout the studies for services offloading [16, 17], although some works have a focus on the collaborative problem of cloud and edge servers [13, 18], a few studies considered the collaboration of services. In summary, the mobility of vehicles and the interrelationships between services bring new challenges for service offloading.To address the mentioned challenges, a collaborative service offloading approach named CSO-DRL is proposed to search optimal offloading strategy for interacting services. In our approach, an algorithm for offloading location selection is presented to consider the impact of speed and location of vehicles. This algorithm divides the road segments by analyzing the trajectory of vehicles, and then the location of offloading is determined by the status of vehicles. Furthermore, the interacting services are discovered based on a parallel frequent pattern mining algorithm. Finally, we constructed the system models and put forward a collaborative offloading algorithm based on deep reinforcement learning to offload the interacting services while considering the optimization problem for minimizing the service requesting delay and data communication delay between interacting services. Specifically, the contribution of this paper is threefold as follows:(i)
An offloading location decision algorithm is presented by analyzing the trajectory of vehicles, which divides the road segments with the k-means clustering algorithm first, and then the offloading location is determined via the status of vehicles.(ii)
In order to reduce the service requesting delay and data communication delay, the interacting services are discovered by a parallel frequent pattern-based algorithm first, and then the offloading models are constructed by analyzing the communication environments and the computation models.(iii)
The collaborative service offloading algorithm with deep deterministic policy gradient is put forward to obtain the optimal service offloading strategy. This algorithm reveals the interrelationship of interacting services by minimizing the service requesting delay and data communication delay in service offloading.The rest of this paper is organized as follows. Section2 introduces the related work of this research. Section 3 presents an overall framework of the collaborative service offloading, and then an offloading location decision algorithm and the system model based on the interacting services discovered are produced in this section. Furthermore, a collaborative service offloading algorithm with deep deterministic policy gradient (DDPG) model is proposed to offload the interacting services in Section 4, which can realize the optimization of minimizing the service requesting delay and data communication delay between interacting services. Finally, the experimental evaluations are conducted in Section 5, and then we conclude this paper in Section 6.


## 2. Related Work

With the advent of the internet of everything, the data produced and collected by the various sensors in IoT are experiencing a steep rise [19–21]. Though the traditional cloud computing paradigm can solve the inefficient computation capacity of intelligent devices, it can also bring some problems, such as high latency, network congestion, and data privacy leaking. In order to address such problems, a new computing paradigm named edge computing is introduced [22]. In edge computing, the data are preprocessed by intelligent devices, and the edge servers nearby users undertake the majority of tasks for computing data and executing the services, while the remote cloud servers are only responsible for the training of deep neural network models [23, 24]. Due to the resource-constrained of end devices, the services should be offloaded on edge servers or cloud servers [25].Service offloading (also called computation offloading) aims to address the inefficient storage capacity, computing resources, and energy of numerous devices at the user level [26]. In the depth of offloading researches, the service offloading mainly contains offloading decisions and offloading optimization. Offloading decision is mainly studies on whether the service should be offloaded or not, and which service should be offloaded. The main studies can be divided into two types, which are 0/1 offloading and partial offloading [27, 28]. 0/1 offloading means the service can be executed either locally or on the other devices fully, which cannot archive the optimal delay. In partial offloading, the service can be split into a quantity of subservices. The offloading strategy is responsible for determining which subservices should be executed locally and which part should be executed on edge or cloud. During the service offloading, latency and energy are the main factors considered by researchers. For time-sensitive services, for example, VR gaming and multimedia services, the researchers mainly focus on how to reduce the latency [16]. Some studies have examined the optimization of latency and energy together, which is difficult to search for the balance optimal result in latency and energy consumption [29]. Some studies noticed that deep reinforcement learning (DRL) is a better way to search for the optimal offloading strategy than the traditional optimization algorithm. Since then, the task offloading approaches with deep reinforcement learning are increasingly, and some efficient approaches based on various DRL algorithms are introduced to get the optimal latency or energy [30–32]. Besides these studies, the load balance brings new challenges for resource allocation; some efficiency approaches are presented to address these challenges [33, 34].Internet of vehicles (IoV) is a typical IoT, which connects vehicles and RSUs through wireless communication technologies. As the number of vehicles increases exponentially, as a complex system, the cloud computing-enabled IoV cannot meet the real-time demands for vehicular service, while vehicular edge computing (VEC) can provide lower data process delay and a more stable network transmission rate. Some studies have introduced for VEC framework and task scheduling in VEC [35, 36]. In IoV, since the vehicles move along the roads, the task offloading in VEC is a dynamic process. The new challenges for task offloading in VEC have been produced, and some efficient approaches have been introduced. Wang et al. [37] put forward a solution for task offloading in the fog-based IoV system, which decomposes the offloading optimization problem into two subproblems and schedules the traffic queue between different edge servers, the experimental evaluation results show that the algorithm can obtain the lower latency than other algorithms. In [38], the privacy conflicts of offloading are considered, and the NSGA-II algorithm is designed to solve the multiobjective optimization for reducing the service computation delay and the energy consumption of devices while guarding against privacy conflicts. He et al. [39] considered the QoE of services, and then a DRL-based algorithm is introduced to save energy consumption and improve the QoE value.In the sight of these studies, few works have examined the location selection and the data communication delay between interacting services, which are the nonignored factors for service offloading. In view of the importance of location decisions for offloading and the interrelationships between services, this paper proposes a collaborative service offloading approach with deep reinforcement learning to address the above challenges.


## 3. System Model and Problem Formulation

In this section, we present the framework of the collaborative service offloading approach first, and then the algorithm for offloading location selection is produced. Finally, the system model, contained communication model, and computation model for our approach are put forward to formulate the offloading problem, which should be solved in the following sections.


### 3.1. Framework of Collaborative Service Offloading

In vehicular edge computing, the vehicular devices collect the data from numerous sensors and then transmit the preprocessed raw data to RSUs equipped with edge servers. The RSUs receive the data and execute the service to send back the decisions to vehicle users. Due to the limitation of computation capacity of vehicular devices, the computing-intensive services should be offloaded to the edge servers, and the vehicular devices only undertake the task of preprocessing the raw data and execute the noncomputing-intensive services. Thus, how to offload the services with low latency is important for vehicular edge computing.As Figure1 shows, the vehicles move along the road and communicate with RSUs through a wireless network. Due to the coverage limitation of RSU, the vehicle cannot request the service from a fixed RSU. When the vehicle moves at the boundary of RSU, the network should be handoff and the vehicle accesses another RSU nearby the vehicle. Therefore, the mobility of the vehicles may affect the service requesting a delay. In the sight of the studies for service offloading, most of them focus on the optimization of service requesting a delay, and few of them study the location selection for service offloading. In order to obtain a lower delay, during the process of service offloading, the location selection for offloading is an important problem, which must be considered. Besides offloading location selection, the interrelationship of services is another nonignored problem for service offloading. We noticed some services may interact with each other to complete a complex business goal, which may produce lots of transmission data among them. If the interacting services are offloaded separately, the data communication delay between them will be increased. Thus, the data communication between these interacting services is a nonignored factor.Figure 1
Vehicular edge computing scenario.In order to address the problems mentioned above, we propose a collaborative service offloading approach with deep reinforcement learning named CSO-DRL to offload the interacting services. As Figure2 shows, the k-means-based road segments dived algorithm is presented first, and then the suitable RSU for offloading is selected based on the mobility of vehicles. Furthermore, the interacting services are discovered by analyzing the service requesting log to reveal the relationship between interacting services. Finally, a DDPG-based collaborative service offloading algorithm is presented for offloading the interacting services, which can optimize the service requesting delay and data communication delay between interacting services, the detailed process can be expressed as follows.Figure 2
Framework of CSO-DRL in VEC.Step 1.
The trajectory data of vehicles are collected, and then a k-means-based road segments dividing algorithm is put forward to obtain the road segments information. For each segment, an RSU is deployed as the location of service offloading in this segment.Step 2.
A vehicle is randomly selected to observe the running status. When the vehicle reaches the boundary of this segment, the RSU should not be switched to reduce the time consumption produced by network handoff; otherwise, another RSU is selected for offloading.Step 3.
The service requesting log is exacted to discover the interacting relationship for services. A parallel frequent pattern-based algorithm is presented to mine the frequent service 2-itemsets for interacting services, which can be found in our previous works [40, 41].Step 4.
A DDPG-based collaborative service offloading algorithm is proposed to offload the interacting services, which can optimize the service requesting delay and data communication delay between interacting services. In this algorithm, the training model is deployed on the cloud servers to receive the vehicle running environment state, and then the optimal offloading strategy is obtained and performed after numerous interaction computing.


### 3.2. Offloading Location Decision

In VEC, the mobility of vehicles may affect the service requesting a delay. Due to the coverage limitation of each RSU, the vehicles access the different RSUs when the vehicles reached different locations. The selection of offloading location is an important problem, which should not be ignored during the service offloading. In this section, we put forward a service offloading location selection approach to assist service offloading action decisions.In reality, it is difficult to deploy the edge servers on all the RSUs, as it is a time-consuming and resource-consuming process. Hence, a k-means-based algorithm is introduced to divide the road segment and select the RSU to deploy the edge server as the offloading location. In this algorithm, the trajectory data of vehicles are collected as the input of our proposed algorithm, and then the segments of the road are divided by a k-means clustering-based algorithm. In our algorithm, the distance of timeti and tj can be expressed as dij, which is computed by the Euclidean distance as follows:(1)dij=lonti−lontj2+lati−latj2,where lonti denotes the longitude of the vehicles in time episode ti and the latitude of the vehicle in time episode ti can be denoted as lati.The mobility of vehicles may produce the dynamic of service requesting delay. During the services offloading, the main purpose is deciding the strategy of offloading and optimizing the service requesting delays, but the location decision is also a nonignore problem, which may affect the time delay. Thus, we put forward an algorithm for dividing road segments based on the k-means clustering algorithm [42]. Details of this algorithm can be found as Algorithm 1 shows.Algorithm 1: Algorithm for dividing road segments.
Input. Data set of vehicles trajectory Vt; number of road segments kOutput. Road segments R=r1,r2,…,rk(1)
Select thek samples as the initial vector u1,u2,…,uk.(2)
while True do(3)
num=0(4)
fori=0,1,…,kdo(5)
Ci=Φ(6)
end for(7)
forj=1,2,…,ndo(8)
Compute the distance ofvj from each vector ui1≤i≤k according to equation (1).(9)
Determine the clusters’ label according to the nearest vector,τj=argmini∈1,2,…,kdji(10)
rτi=rτi∪vj(11)
end for(12)
fori=1,2,…,kdo(13)
ui′=1/Ci∑x∈Cix(14)
ifui′≠uithen(15)
ui=ui′(16)
else(17)
num++(18)
end if(19)
end for(20)
if num = kthen(21)
break(22)
end if(23)
end while(24)
returnR=ri,r2,…,rkWith the road segments divided, we select an RSU in each segment to equip with the edge server, which mainly undertakes the task of receiving the data transmitted from vehicles and executing the computing-intensive services requested by the vehicles running in the area of this road segment. For each segment, a vehicle is randomly selected as the observed object. When the vehicle moves in the segment and the corresponding edge server is selected as the offloading location. With the running of the vehicle, the vehicle may reach the boundary of the segment; the edge server should not be switched to reduce the time consumption produced by the network handoff. In each road segment, the vehicle is observed, and the environment states are transferred to the collaborative offloading model, which is deployed on the cloud server. After the iteration computing, the offloading actions are obtained and sent to the vehicle for performing. If the vehicle moves out of the coverage of the current segment, the edge server must be switched and another RSU will be selected as the offloading location, and then the training steps are adopted as previous actions.


### 3.3. System Model

In this section, the system model, contained communication model, and computation model are presented to formulate the problem of our approach.Compared with previous studies, our approach offloads the interacting services to optimize the service requesting delay and data communication delay. In our previous works [40, 41], a parallel frequent pattern-based algorithm is presented to discover the interacting services. In this algorithm, the service requesting log denoted as EL≜cid,Ts,∏ is adapted as the input of the interacting services discovery algorithm, where cid denotes the nonempty set of service requesting case ID and the Ts denotes the finite serial of time stamps. The ∏ is the finite serial of services. We noticed that a composition service is composed of a series subservices to complete a certain complex business goal, which can be denoted as Si=s1,s2,…,sn. In these subservices, it may exist the interrelationships between subservices. In our algorithm, the fine-grained frequent 2-itemsets of services (also called interacting services pairs) are discovered by a parallel frequent pattern-based algorithm directly, which can reveal the interrelationships between them. The details can be found in our previous works [40, 41]. Finally, the interacting services are offloaded by our CSO-DRL algorithm, which can be found in Section 4.


#### 3.3.1. Communication Model

In order to make this paper clear, we give the important notations in our paper first, which can be found in Table1.Table 1
List of important notations.
NotationsDescriptionHChannel power gainPTransmission powerσ2Noise powerWTransmission bandwidthCiComputation capacity for executing servicesifdevComputation capacity of devicesfedgeComputation capacity of edge serversDioffData transmission size from devices to edge serversCioffComputation capacity for executing the offloaded serviceCilocalComputation capacity for executing the service deployed on the devicesDjreData communication size between interacting servicesSince the signal will be affected Gaussian white noise during the transmission process, we construct the communication model with Gaussian white noise to reflect the real communication channel. Here, we assume the transmission power is fixed, which can be denoted asP. The standard path loss propagation index can be denoted as θ. In IoV, the vehicles move along the road and access the service from the nearest edge server at RSU; thus, the distance between the vehicle and edge server in a time slot t can be denoted as dt. The signal-to-noise ration SNR of the communication channel can be expressed as follows:(2)SNR=PHdt−θσ2,where σ2 is the power of Gaussian white noise and H denotes the channel gain. So the transmission rate can be given as follows:(3)R=Wlog1+SNR,where W denotes the channel bandwidth.


#### 3.3.2. Computation Model

In this paper, we considered the interrelationships between interacting services. If some services are offloaded to remote cloud servers, which may produce lots of data communications between them. In the VEC system, due to the computation limitation of vehicular devices, the vehicular devices offload the portion services to RSUs equipped with edge servers, while the remaining services are locally executed at devices. The RSUs will receive the preprocessed data and execute the computing-intensive services, and then the decision will be sent back to vehicular users. Hence, the partial service offloading manner is adapted in each time slotT. The offloading manner contains three parts, which are locally computing, partial offloading on edge servers, and fully edge computing. Next, we will give the service delay model for these three parts:(A)
Locally computing. In our approach, the services without high computation capacity can be computed on the devices directly. So the computation delay can be given as follows:(4)Tilocal=Cifdev,wherefdev denotes the computation capacity of the vehicular device and the computation capacity for executing the service si can be denoted as Ci.(B)
Partial offloading. Due to the computation capacity limitation of the vehicular devices, the services are offloaded by a partial offloading strategy. In these composition services, some subservices are executed on vehicular devices, but some computation-intensive subservices should be offloaded on RSUs equipped with the edge servers; therefore, the data produced by these services should be transmitted to the edge servers. The time delay can be expressed as follows:(5)Tipar=maxTilocal,2Titran+Tiedge,whereTitran denotes the transmission delay for data transmitting and Tiedge denotes the computation delay for the edge computing part. These can be found as follows:(6)Titran=DioffR,Tiedge=Ciofffedge,whereDioff denotes the transmission data to edge, Cioff denotes the computation capacity for executing the offloaded services, and fedge denotes the computation capacity of the edge server. Thus, the delay for partial offloading can be expressed as follows:(8)Tipar=maxCilocalfdev,2DioffR+Ciofffedge,whereCilocal denotes the computation capacity for executing the services deployed on devices.(C)
Fully offloading. For the computation-intensive services, all of these should be offloaded on edge servers; thus, the delay can be expressed as follows:(9)Tifull=2DiR+Cifedge,whereDi is the data for executing these services and the Ci denotes the computation capacity for executing the service.


### 3.4. Problem Formulation

Based on the description of the system problem in the above section, the delay can be expressed as follows:(10)Ti=Cifdev,maxCilocalfdev,2DioffR+Ciofffedge,2DiR+Cifedge.Besides the system computation model produced in the above section, we noticed that the data communication delay between interacting services is the nonignore problem. In this system, we assume the number of interacting service pairs isn, and the number of interacting service pairs executed on vehicular devices is a; thus, the latency of the system can be described as the following equation:(11)Ti=Cifdev,maxCilocalfdev,2DioffR+∑j=1n−aDjreR+Ciofffedge,2DiR+∑j=1nDjreR+Cifedge.where Djre denotes the data communication size between the interacting services.In our approach, we letλ∈0,1 denotes the ratio of service offloading. So Dioff=λ∗Di. Finally, the latency of our system can be given as follows:(12)Tisum=max1−λCilocalfdev,λ2DiR+Ciofffedge+∑j=1n−aDjreR.In this paper, the purpose is to optimize the latency, which combines the service requesting delay and data communication delay between interacting services. Thus, the optimization function can be given as follows:(13)MinTisum.Based on the analysis of the environments, the constraint conditions can be expressed as follows:(14)Cilocal≤Ci≤fdev,Cioff≤Ci≤fedge.


## 3.1. Framework of Collaborative Service Offloading

In vehicular edge computing, the vehicular devices collect the data from numerous sensors and then transmit the preprocessed raw data to RSUs equipped with edge servers. The RSUs receive the data and execute the service to send back the decisions to vehicle users. Due to the limitation of computation capacity of vehicular devices, the computing-intensive services should be offloaded to the edge servers, and the vehicular devices only undertake the task of preprocessing the raw data and execute the noncomputing-intensive services. Thus, how to offload the services with low latency is important for vehicular edge computing.As Figure1 shows, the vehicles move along the road and communicate with RSUs through a wireless network. Due to the coverage limitation of RSU, the vehicle cannot request the service from a fixed RSU. When the vehicle moves at the boundary of RSU, the network should be handoff and the vehicle accesses another RSU nearby the vehicle. Therefore, the mobility of the vehicles may affect the service requesting a delay. In the sight of the studies for service offloading, most of them focus on the optimization of service requesting a delay, and few of them study the location selection for service offloading. In order to obtain a lower delay, during the process of service offloading, the location selection for offloading is an important problem, which must be considered. Besides offloading location selection, the interrelationship of services is another nonignored problem for service offloading. We noticed some services may interact with each other to complete a complex business goal, which may produce lots of transmission data among them. If the interacting services are offloaded separately, the data communication delay between them will be increased. Thus, the data communication between these interacting services is a nonignored factor.Figure 1
Vehicular edge computing scenario.In order to address the problems mentioned above, we propose a collaborative service offloading approach with deep reinforcement learning named CSO-DRL to offload the interacting services. As Figure2 shows, the k-means-based road segments dived algorithm is presented first, and then the suitable RSU for offloading is selected based on the mobility of vehicles. Furthermore, the interacting services are discovered by analyzing the service requesting log to reveal the relationship between interacting services. Finally, a DDPG-based collaborative service offloading algorithm is presented for offloading the interacting services, which can optimize the service requesting delay and data communication delay between interacting services, the detailed process can be expressed as follows.Figure 2
Framework of CSO-DRL in VEC.Step 1.
The trajectory data of vehicles are collected, and then a k-means-based road segments dividing algorithm is put forward to obtain the road segments information. For each segment, an RSU is deployed as the location of service offloading in this segment.Step 2.
A vehicle is randomly selected to observe the running status. When the vehicle reaches the boundary of this segment, the RSU should not be switched to reduce the time consumption produced by network handoff; otherwise, another RSU is selected for offloading.Step 3.
The service requesting log is exacted to discover the interacting relationship for services. A parallel frequent pattern-based algorithm is presented to mine the frequent service 2-itemsets for interacting services, which can be found in our previous works [40, 41].Step 4.
A DDPG-based collaborative service offloading algorithm is proposed to offload the interacting services, which can optimize the service requesting delay and data communication delay between interacting services. In this algorithm, the training model is deployed on the cloud servers to receive the vehicle running environment state, and then the optimal offloading strategy is obtained and performed after numerous interaction computing.


## 3.2. Offloading Location Decision

In VEC, the mobility of vehicles may affect the service requesting a delay. Due to the coverage limitation of each RSU, the vehicles access the different RSUs when the vehicles reached different locations. The selection of offloading location is an important problem, which should not be ignored during the service offloading. In this section, we put forward a service offloading location selection approach to assist service offloading action decisions.In reality, it is difficult to deploy the edge servers on all the RSUs, as it is a time-consuming and resource-consuming process. Hence, a k-means-based algorithm is introduced to divide the road segment and select the RSU to deploy the edge server as the offloading location. In this algorithm, the trajectory data of vehicles are collected as the input of our proposed algorithm, and then the segments of the road are divided by a k-means clustering-based algorithm. In our algorithm, the distance of timeti and tj can be expressed as dij, which is computed by the Euclidean distance as follows:(1)dij=lonti−lontj2+lati−latj2,where lonti denotes the longitude of the vehicles in time episode ti and the latitude of the vehicle in time episode ti can be denoted as lati.The mobility of vehicles may produce the dynamic of service requesting delay. During the services offloading, the main purpose is deciding the strategy of offloading and optimizing the service requesting delays, but the location decision is also a nonignore problem, which may affect the time delay. Thus, we put forward an algorithm for dividing road segments based on the k-means clustering algorithm [42]. Details of this algorithm can be found as Algorithm 1 shows.Algorithm 1: Algorithm for dividing road segments.
Input. Data set of vehicles trajectory Vt; number of road segments kOutput. Road segments R=r1,r2,…,rk(1)
Select thek samples as the initial vector u1,u2,…,uk.(2)
while True do(3)
num=0(4)
fori=0,1,…,kdo(5)
Ci=Φ(6)
end for(7)
forj=1,2,…,ndo(8)
Compute the distance ofvj from each vector ui1≤i≤k according to equation (1).(9)
Determine the clusters’ label according to the nearest vector,τj=argmini∈1,2,…,kdji(10)
rτi=rτi∪vj(11)
end for(12)
fori=1,2,…,kdo(13)
ui′=1/Ci∑x∈Cix(14)
ifui′≠uithen(15)
ui=ui′(16)
else(17)
num++(18)
end if(19)
end for(20)
if num = kthen(21)
break(22)
end if(23)
end while(24)
returnR=ri,r2,…,rkWith the road segments divided, we select an RSU in each segment to equip with the edge server, which mainly undertakes the task of receiving the data transmitted from vehicles and executing the computing-intensive services requested by the vehicles running in the area of this road segment. For each segment, a vehicle is randomly selected as the observed object. When the vehicle moves in the segment and the corresponding edge server is selected as the offloading location. With the running of the vehicle, the vehicle may reach the boundary of the segment; the edge server should not be switched to reduce the time consumption produced by the network handoff. In each road segment, the vehicle is observed, and the environment states are transferred to the collaborative offloading model, which is deployed on the cloud server. After the iteration computing, the offloading actions are obtained and sent to the vehicle for performing. If the vehicle moves out of the coverage of the current segment, the edge server must be switched and another RSU will be selected as the offloading location, and then the training steps are adopted as previous actions.


## 3.3. System Model

In this section, the system model, contained communication model, and computation model are presented to formulate the problem of our approach.Compared with previous studies, our approach offloads the interacting services to optimize the service requesting delay and data communication delay. In our previous works [40, 41], a parallel frequent pattern-based algorithm is presented to discover the interacting services. In this algorithm, the service requesting log denoted as EL≜cid,Ts,∏ is adapted as the input of the interacting services discovery algorithm, where cid denotes the nonempty set of service requesting case ID and the Ts denotes the finite serial of time stamps. The ∏ is the finite serial of services. We noticed that a composition service is composed of a series subservices to complete a certain complex business goal, which can be denoted as Si=s1,s2,…,sn. In these subservices, it may exist the interrelationships between subservices. In our algorithm, the fine-grained frequent 2-itemsets of services (also called interacting services pairs) are discovered by a parallel frequent pattern-based algorithm directly, which can reveal the interrelationships between them. The details can be found in our previous works [40, 41]. Finally, the interacting services are offloaded by our CSO-DRL algorithm, which can be found in Section 4.


### 3.3.1. Communication Model

In order to make this paper clear, we give the important notations in our paper first, which can be found in Table1.Table 1
List of important notations.
NotationsDescriptionHChannel power gainPTransmission powerσ2Noise powerWTransmission bandwidthCiComputation capacity for executing servicesifdevComputation capacity of devicesfedgeComputation capacity of edge serversDioffData transmission size from devices to edge serversCioffComputation capacity for executing the offloaded serviceCilocalComputation capacity for executing the service deployed on the devicesDjreData communication size between interacting servicesSince the signal will be affected Gaussian white noise during the transmission process, we construct the communication model with Gaussian white noise to reflect the real communication channel. Here, we assume the transmission power is fixed, which can be denoted asP. The standard path loss propagation index can be denoted as θ. In IoV, the vehicles move along the road and access the service from the nearest edge server at RSU; thus, the distance between the vehicle and edge server in a time slot t can be denoted as dt. The signal-to-noise ration SNR of the communication channel can be expressed as follows:(2)SNR=PHdt−θσ2,where σ2 is the power of Gaussian white noise and H denotes the channel gain. So the transmission rate can be given as follows:(3)R=Wlog1+SNR,where W denotes the channel bandwidth.


### 3.3.2. Computation Model

In this paper, we considered the interrelationships between interacting services. If some services are offloaded to remote cloud servers, which may produce lots of data communications between them. In the VEC system, due to the computation limitation of vehicular devices, the vehicular devices offload the portion services to RSUs equipped with edge servers, while the remaining services are locally executed at devices. The RSUs will receive the preprocessed data and execute the computing-intensive services, and then the decision will be sent back to vehicular users. Hence, the partial service offloading manner is adapted in each time slotT. The offloading manner contains three parts, which are locally computing, partial offloading on edge servers, and fully edge computing. Next, we will give the service delay model for these three parts:(A)
Locally computing. In our approach, the services without high computation capacity can be computed on the devices directly. So the computation delay can be given as follows:(4)Tilocal=Cifdev,wherefdev denotes the computation capacity of the vehicular device and the computation capacity for executing the service si can be denoted as Ci.(B)
Partial offloading. Due to the computation capacity limitation of the vehicular devices, the services are offloaded by a partial offloading strategy. In these composition services, some subservices are executed on vehicular devices, but some computation-intensive subservices should be offloaded on RSUs equipped with the edge servers; therefore, the data produced by these services should be transmitted to the edge servers. The time delay can be expressed as follows:(5)Tipar=maxTilocal,2Titran+Tiedge,whereTitran denotes the transmission delay for data transmitting and Tiedge denotes the computation delay for the edge computing part. These can be found as follows:(6)Titran=DioffR,Tiedge=Ciofffedge,whereDioff denotes the transmission data to edge, Cioff denotes the computation capacity for executing the offloaded services, and fedge denotes the computation capacity of the edge server. Thus, the delay for partial offloading can be expressed as follows:(8)Tipar=maxCilocalfdev,2DioffR+Ciofffedge,whereCilocal denotes the computation capacity for executing the services deployed on devices.(C)
Fully offloading. For the computation-intensive services, all of these should be offloaded on edge servers; thus, the delay can be expressed as follows:(9)Tifull=2DiR+Cifedge,whereDi is the data for executing these services and the Ci denotes the computation capacity for executing the service.


## 3.3.1. Communication Model

In order to make this paper clear, we give the important notations in our paper first, which can be found in Table1.Table 1
List of important notations.
NotationsDescriptionHChannel power gainPTransmission powerσ2Noise powerWTransmission bandwidthCiComputation capacity for executing servicesifdevComputation capacity of devicesfedgeComputation capacity of edge serversDioffData transmission size from devices to edge serversCioffComputation capacity for executing the offloaded serviceCilocalComputation capacity for executing the service deployed on the devicesDjreData communication size between interacting servicesSince the signal will be affected Gaussian white noise during the transmission process, we construct the communication model with Gaussian white noise to reflect the real communication channel. Here, we assume the transmission power is fixed, which can be denoted asP. The standard path loss propagation index can be denoted as θ. In IoV, the vehicles move along the road and access the service from the nearest edge server at RSU; thus, the distance between the vehicle and edge server in a time slot t can be denoted as dt. The signal-to-noise ration SNR of the communication channel can be expressed as follows:(2)SNR=PHdt−θσ2,where σ2 is the power of Gaussian white noise and H denotes the channel gain. So the transmission rate can be given as follows:(3)R=Wlog1+SNR,where W denotes the channel bandwidth.


## 3.3.2. Computation Model

In this paper, we considered the interrelationships between interacting services. If some services are offloaded to remote cloud servers, which may produce lots of data communications between them. In the VEC system, due to the computation limitation of vehicular devices, the vehicular devices offload the portion services to RSUs equipped with edge servers, while the remaining services are locally executed at devices. The RSUs will receive the preprocessed data and execute the computing-intensive services, and then the decision will be sent back to vehicular users. Hence, the partial service offloading manner is adapted in each time slotT. The offloading manner contains three parts, which are locally computing, partial offloading on edge servers, and fully edge computing. Next, we will give the service delay model for these three parts:(A)
Locally computing. In our approach, the services without high computation capacity can be computed on the devices directly. So the computation delay can be given as follows:(4)Tilocal=Cifdev,wherefdev denotes the computation capacity of the vehicular device and the computation capacity for executing the service si can be denoted as Ci.(B)
Partial offloading. Due to the computation capacity limitation of the vehicular devices, the services are offloaded by a partial offloading strategy. In these composition services, some subservices are executed on vehicular devices, but some computation-intensive subservices should be offloaded on RSUs equipped with the edge servers; therefore, the data produced by these services should be transmitted to the edge servers. The time delay can be expressed as follows:(5)Tipar=maxTilocal,2Titran+Tiedge,whereTitran denotes the transmission delay for data transmitting and Tiedge denotes the computation delay for the edge computing part. These can be found as follows:(6)Titran=DioffR,Tiedge=Ciofffedge,whereDioff denotes the transmission data to edge, Cioff denotes the computation capacity for executing the offloaded services, and fedge denotes the computation capacity of the edge server. Thus, the delay for partial offloading can be expressed as follows:(8)Tipar=maxCilocalfdev,2DioffR+Ciofffedge,whereCilocal denotes the computation capacity for executing the services deployed on devices.(C)
Fully offloading. For the computation-intensive services, all of these should be offloaded on edge servers; thus, the delay can be expressed as follows:(9)Tifull=2DiR+Cifedge,whereDi is the data for executing these services and the Ci denotes the computation capacity for executing the service.


## 3.4. Problem Formulation

Based on the description of the system problem in the above section, the delay can be expressed as follows:(10)Ti=Cifdev,maxCilocalfdev,2DioffR+Ciofffedge,2DiR+Cifedge.Besides the system computation model produced in the above section, we noticed that the data communication delay between interacting services is the nonignore problem. In this system, we assume the number of interacting service pairs isn, and the number of interacting service pairs executed on vehicular devices is a; thus, the latency of the system can be described as the following equation:(11)Ti=Cifdev,maxCilocalfdev,2DioffR+∑j=1n−aDjreR+Ciofffedge,2DiR+∑j=1nDjreR+Cifedge.where Djre denotes the data communication size between the interacting services.In our approach, we letλ∈0,1 denotes the ratio of service offloading. So Dioff=λ∗Di. Finally, the latency of our system can be given as follows:(12)Tisum=max1−λCilocalfdev,λ2DiR+Ciofffedge+∑j=1n−aDjreR.In this paper, the purpose is to optimize the latency, which combines the service requesting delay and data communication delay between interacting services. Thus, the optimization function can be given as follows:(13)MinTisum.Based on the analysis of the environments, the constraint conditions can be expressed as follows:(14)Cilocal≤Ci≤fdev,Cioff≤Ci≤fedge.


## 4. Algorithm for DRL-Based Collaborative Service Offloading

In this section, an algorithm of collaborative service offloading based on DDPG is presented to offload the interacting services. The details of this algorithm can be described in the next contents.Deep reinforcement learning is the algorithm mixed with deep learning and reinforcement learning. The reinforcement learning model is an iteration process, which can be divided into environment and agent. For each step, the agent observes the statest and takes actions according to the current policy π. When the state of the environment changes to st+1, the reward value rt is received in the next step. The process for state transition of the environment and the reward computing can be constructed as a Markov process; thus, the probability and reward of the state transition only depend on the environment state st and action at. The agent receives these quantities according to the decision policy and interacts with the environment, and then the state is back propagated for maximizing the expected reward rt.The DDPG algorithm is a model-free nonpolicy action-critic algorithm based on DNN ideas, which can learn policies in the continuous action spaces [43]. The action-critic algorithm consists of the policy function and the q-value function, where the policy function undertakes the task to generate actions like an actor and the q-value function undertakes the task to evaluate the actor’s performance and direct the actor’s following actions.As Figure3 shows, the structure of DDPG is similar to actor-critic algorithm [44]. DDPG can be divided into actor network and critic network, which continues the idea of a fixed target network such as DQN [45]. First, the actor network generates μsi after the training step, and then the action space with Gaussian noise is constructed. After performing the at in the environment, the agent observes the next state and the immediate reward. Second, the transition is stored in the experience replay butter, and N transitions are selected by the model to make up a minibatch, which can be transferred into the actor network and critic network. With the minibatch, the actor target network outputs the action to the critic target network, and then the critic network calculates the target value. Furthermore, the critic network will be updated, and the actor network gives the minibatch action to the critic network to achieve the action’s gradient. Finally, the DDPG agent updates the critic target network and the actor target network. The details can be found in [43].Figure 3
Deep deterministic policy gradient (DDPG) model.Compared with the traditional DDPG algorithm, we adapt a state normalization algorithm to preprocess the observed states and add the behavior noise to the environments, which takes the difference between the maximum and minimum of each variable as the scaling factor [46]. The state normalization algorithm can solve the problem of the magnitude difference of input variables. According to the previous work [46], we can find that after normalizing the state and adding the behavior noise, our algorithm can obtain a better performance than the traditional algorithm.


### 4.1. State Space

In our approach, the users’ requirements, network conditions, and channel and resource conditions are used as the state of the system, which will produce the state space explosion problems. In order to simplify the system, we only consider the offloading requirement in a single user scenario, which can reduce the dimension of the state space increasingly and make the algorithm easy to converge. So the state space can be expressed as follows:(15)St=qt,pt,Dremaint,Dt,Dret,where qt represents the location information of the edge server and pt denotes the location information of the vehicle. The data size of the remaining subservices is Dremain, and Dt denotes the task size requested by the vehicle; finally, Dret denotes the data communication size between the interacting services.


### 4.2. Action Space

We define the action space as the size of the offloading task, which can be expressed as follows:(16)at=kt,λk,where kt indicates the edge server that undertakes the task for executing the services and λk denotes the service offloading ratio.


### 4.3. Reward Function

In this paper, the main purpose of our algorithm is to minimize the latency, which contains the service requesting delay and communication delay. Thert is defined as the reward when the action at is performed at state st. In order to reduce the system latency while guaranteeing the sufficient computation resources of devices, the reward function can be given as follows:(17)rt=−Tisum.The action value function is used to describe the expected return of policyπ at time episode t in the reinforcement learning algorithm, which can be expressed as follows:(18)Qπst,at=Ert,st+1∼Ertst,at+γEat+1∼πQπst+1,at+1.Therefore, if the update of the target policy is continuous, the target policy can be described as a functionμ:A←S, which can be expressed as follows:(19)Qπst,at=Ert,st+1∼Ertst,at+γEat+1∼πQπst+1,at+1.Algorithm 2 demonstrates the details of our CSO-DRL algorithm, which is an iteration process. During the training process, the parameters of the critic network and the actor network are trained with the iteratively update process, and then the trained actor network parameters are adopted to offload the interacting services.Algorithm 2: Algorithm for collaborative service offloading with DDPG.
Input. Training data set I; critic learning rate λc; actor learning rate λa; discount factor γ; soft update factor τ; experience replay buffer Bm; minibatch size N; Gaussian noise ni, current state qi,pi,Dremaini,Di,fi; state parameters with normalized γDremain,γDue,γx,γyOutput. Reward ri(1)
Initialize the critic network and the actor network with weightsθQ and θμ(2)
Initialize the target network with weightsθQ′⟵θQ, θμ′⟵θμ(3)
Empty the experience replay bufferBm(4)
for episode = 1 to Mdo(5)
Reset the simulation parameters and receive the initial observation states1(6)
fort = 1, 2, …, Tdo(7)
Initializest as the current state(8)
Normalize statest′⟵st, and obtain the feature vector ϕs′(9)
Get the actionat=πθμϕs′+nt(10)
Execute the action to get the rewardret, and observe the next state st+1(11)
Normalize statest+1′⟵st+1(12)
ifBm is not full then(13)
Storest′,at,rt,st+1′ to Bm(14)
else(15)
Randomly replace a transition inBm with st′,at,rt,st+1′(16)
Randomly sample a mini-batch fromBm(17)
yj=rj+γQ′sj+1′,μ′sj+1′|θμ′,θQ′(18)
LθQ=1/N∑j=1Nyj−Qsj′,aj|θQ2(19)
Updateθμ by the sampled policy gradient(20)
Soft update the critic target network and actor target network(21)
end if(22)
end for(23)
end for(24)
Getμs′|θμ(25)
for episode = 1, 2, …, E do(26)
Reset the VEC parameters and receive the initial observations1(27)
fort = 1, 2, …, Tdo(28)
Normalize statest′⟵st(29)
at=μs′|θμ(30)
Execute the actionat and get the reward rt(31)
end for(32)
end for


## 4.1. State Space

In our approach, the users’ requirements, network conditions, and channel and resource conditions are used as the state of the system, which will produce the state space explosion problems. In order to simplify the system, we only consider the offloading requirement in a single user scenario, which can reduce the dimension of the state space increasingly and make the algorithm easy to converge. So the state space can be expressed as follows:(15)St=qt,pt,Dremaint,Dt,Dret,where qt represents the location information of the edge server and pt denotes the location information of the vehicle. The data size of the remaining subservices is Dremain, and Dt denotes the task size requested by the vehicle; finally, Dret denotes the data communication size between the interacting services.


## 4.2. Action Space

We define the action space as the size of the offloading task, which can be expressed as follows:(16)at=kt,λk,where kt indicates the edge server that undertakes the task for executing the services and λk denotes the service offloading ratio.


## 4.3. Reward Function

In this paper, the main purpose of our algorithm is to minimize the latency, which contains the service requesting delay and communication delay. Thert is defined as the reward when the action at is performed at state st. In order to reduce the system latency while guaranteeing the sufficient computation resources of devices, the reward function can be given as follows:(17)rt=−Tisum.The action value function is used to describe the expected return of policyπ at time episode t in the reinforcement learning algorithm, which can be expressed as follows:(18)Qπst,at=Ert,st+1∼Ertst,at+γEat+1∼πQπst+1,at+1.Therefore, if the update of the target policy is continuous, the target policy can be described as a functionμ:A←S, which can be expressed as follows:(19)Qπst,at=Ert,st+1∼Ertst,at+γEat+1∼πQπst+1,at+1.Algorithm 2 demonstrates the details of our CSO-DRL algorithm, which is an iteration process. During the training process, the parameters of the critic network and the actor network are trained with the iteratively update process, and then the trained actor network parameters are adopted to offload the interacting services.Algorithm 2: Algorithm for collaborative service offloading with DDPG.
Input. Training data set I; critic learning rate λc; actor learning rate λa; discount factor γ; soft update factor τ; experience replay buffer Bm; minibatch size N; Gaussian noise ni, current state qi,pi,Dremaini,Di,fi; state parameters with normalized γDremain,γDue,γx,γyOutput. Reward ri(1)
Initialize the critic network and the actor network with weightsθQ and θμ(2)
Initialize the target network with weightsθQ′⟵θQ, θμ′⟵θμ(3)
Empty the experience replay bufferBm(4)
for episode = 1 to Mdo(5)
Reset the simulation parameters and receive the initial observation states1(6)
fort = 1, 2, …, Tdo(7)
Initializest as the current state(8)
Normalize statest′⟵st, and obtain the feature vector ϕs′(9)
Get the actionat=πθμϕs′+nt(10)
Execute the action to get the rewardret, and observe the next state st+1(11)
Normalize statest+1′⟵st+1(12)
ifBm is not full then(13)
Storest′,at,rt,st+1′ to Bm(14)
else(15)
Randomly replace a transition inBm with st′,at,rt,st+1′(16)
Randomly sample a mini-batch fromBm(17)
yj=rj+γQ′sj+1′,μ′sj+1′|θμ′,θQ′(18)
LθQ=1/N∑j=1Nyj−Qsj′,aj|θQ2(19)
Updateθμ by the sampled policy gradient(20)
Soft update the critic target network and actor target network(21)
end if(22)
end for(23)
end for(24)
Getμs′|θμ(25)
for episode = 1, 2, …, E do(26)
Reset the VEC parameters and receive the initial observations1(27)
fort = 1, 2, …, Tdo(28)
Normalize statest′⟵st(29)
at=μs′|θμ(30)
Execute the actionat and get the reward rt(31)
end for(32)
end for


## 5. Experimental Evaluation

In this section, we conduct experiments to evaluate the performance of our algorithm. First, all the simulation parameters are indicated, and then the efficiency of our algorithm is conducted to compare with other baseline algorithms in the same simulation environment.


### 5.1. Experiment Setup

To ensure the practicability of the algorithm, we use a real-world data set of taxi trajectories in Shanghai (https://cse.hkust.edu.hk/scrg/) to evaluate the experiment in this paper. This data set records the GPS trajectory data on February 20, 2007, in Shanghai, which contains the longitude and latitude of 4316 taxies. Here, we conduct the experiment in the real world to obtain the road segments and determine the location of service offloading. Furthermore, the simulation experiments are conducted to compare the latency with other baseline algorithms, which are described as follows:(i)
Local-only: executing all services locally(ii)
Offload-only: offloading all services to RSUs deployed with edge servers(iii)
AC: actor-critic-based service offloading algorithm [44](iv)
DQN: DQN-based service offloading algorithm [45]We set the channel power gainH=−50dB when the distance is 1m, and the transmission bandwidth is W=100MHz. The noise power accepted without signal blocking is σ2=−100dBm. The computation capacity of the vehicular devices and edge servers are set to fdev=−0.8GHz and fedge=3GHz, respectively. We set the distance between the vehicular devices and the edge servers dt randomly. The size of the task is randomly assigned, and fi=ϵDi is the required computation resource, where ϵ is set at 0.5Gcyles/MB. The detailed simulation parameters are listed in Table 2.Table 2
Simulation parameters.
ParametersNumerical valueUnitChannel power gain,H−50dBTransmission bandwidth,W100MHzNoise power,σ2−100dBmDevices computation capacity,fdev0.8GHzEdge computation capacity,fedge3GHzTask size with random values,Di<60MbitData communication size between interacting services,Djre<MiMbitSince the data set records the location information of the vehicles every60s, so the time slot is set t=60s, and the total time is 24 hours. For fairness, we conduct all the algorithms in the same simulation environment. During the time slots, all the services are offloaded continually by our CSO-DRL algorithm, and then the delay is conducted to evaluate the efficiency of our algorithm.


### 5.2. Experimental Results

The hyperparameters in our algorithm affect the overall performance of the algorithm seriously. The experiments are conducted to determine the optimal values of these hyperparameters in our algorithm. First, we conduct experiments to select the optimal hyperparameters. Figure4 shows the convergence of the CSO-DRL algorithm under different learning rates. In the DDPG model, there are two training networks, which are the actor network and the critic network. We set the learning rates of the critic network and actor network separately. According to the training, we find that the algorithm cannot converge when the learning rate is λa=0.1 and λc=0.2. When the learning rate is λa=0.001 and λc=0.002, the algorithm can get a better convergence value. Therefore, the optimal learning rate of actor network λa=0.01, and the learning rate of critic network λc=0.02.Figure 4
Convergence of CSO-DRL under different learning rates.Next, the convergence of the algorithm under different discount factorsγ are conducted, which can indicate the weight of the value for the next action. The value of γ can affect the efficiency of the algorithm. As Figure 5 shows, the trained service offloading strategy performs best when the discount factor γ=0.5. Therefore, we set the value γ=0.5 in the following experiments.Figure 5
Convergence of CSO-DRL under different discount factors.Furthermore, the convergence of the algorithm under different exploration parametersσ is also conducted. Noticed that, the larger value for σ allows the agent to choose a prudent strategy, and make the evaluation of the q-value become more accurate, and then the training process will become more stable. As Figure 6, when our algorithm converges at σ=0.1, the delay fluctuates at 35s. The initial delay with the σ=0.1 is lower than the situation with σ=0.01. We noticed that when σ=1, the algorithm decreases at 280 episodes, but the convergence is unstable. Hence, we set σ=0.1 to obtain better performance in the following experiments.Figure 6
Convergence of CSO-DRL under different exploration parameters.With the parameters determined, we compare the delay of our algorithm with baseline algorithms. Figure7 shows the performance of different algorithms. From this figure, it can be observed that the delay of the CSO-DRL algorithm is lower than the other baseline algorithms significantly under the same service size. As the number of iterations increases, the AC algorithm cannot converge, while the DQN algorithm and CSO-DRL algorithm can achieve convergence. Because the actor network and critic network of the AC algorithm are updated at the same time, the action selection of the actor network depends on the value function of the critic network, but the critic network is difficult to converge. In contrast, since both DQN and DDPG have a dual network structure. While DQN is only suitable for discrete action spaces and the state space for service offloading is continuous, DQN cannot search the optimal offloading strategy accurately. On the other hand, the CSO-DRL algorithm explores a continuous action space and takes precise actions, which may result in an optimal policy and make the delay reduce significantly. We can also find that the CSO-DRL algorithm and the DQN algorithm can converge at 300 episodes, and the CSO-DRL algorithm obtains a better delay than DQN and converges at 35, which is more stable than the DQN algorithm.Figure 7
Convergence performance of different algorithms.Besides the comparison with the AC algorithm and the DQN algorithm, we also conduct the experiments by comparing the CSO-DRL algorithm with the local-only algorithm and the offload-only algorithm. In Figure7, it is obvious that the delay of local computing is too large due to the limitation of the computation capacity of the vehicular devices. Since the offload-only algorithm depends on the running state and the computation resources of the edge server, and the vehicles keep the state with resource competition for a long time, the CSO-DRL algorithm can obtain a better delay than offload-only algorithm.We also conduct the experiment under different service sizes (also called task sizes). Since the AC algorithm cannot converge, we compare the delay with other algorithms. As Figure8 shows, with the task sizes increased, the delay of the CSO-DRL algorithm is always lower than the other baseline algorithm for the same task size. We notice that the delay of CSO-DRL increases much slower than other baseline algorithms, which is significant when the task sizes increase doubled.Figure 8
Delay comparison under different task sizes.In order to explore the effect of the number of edge servers for the delay, we compare the delay of our proposed algorithm with other algorithms under different numbers of edge servers. As Figure9 shows, the delay of all algorithms except DQN is almost constant as the number of edge servers increases. With the increase in the number of edge servers, the DQN algorithm fluctuates at about 40s. Besides, the proposed CSO-DRL algorithm achieves the lowest delay. The reason is DDPG model can find the optimal value in the continuous action and obtain the optimal control policy.Figure 9
Delay comparison under different numbers of edge servers.Besides the delay comparison with other algorithms under different numbers of edge servers, we compare these algorithms under different computing capabilities of edge servers and computing capabilities of devices, which can be found in Figures10 and 11, respectively. Due to the nonconvergence of the AC algorithm, we only compared our proposed CSO-DRL algorithm with the DQN algorithm.Figure 10
(a) Convergence of different algorithms under different computation capabilities of the edge server. (b) Comparison between different algorithms.
(a)(b)Figure 11
(a) Convergence of different algorithms under different computation capabilities of devices. (b) Comparison between different algorithms.
(a)(b)Figure10 shows the same group of experiments in terms of convergence performance and delay under different computing capabilities of edge servers. We vary the computation capabilities of edge servers from 2GHz to 4GHz. Figure 10(a) shows the convergence performance of the DQN algorithm and our CSO-DRL algorithm. We find that when the computation capability of the edge server is 3GHz, the delay of two algorithms is higher than that when the computation capability of the edge server is 4GHz. Thus, the smaller the computation capability of the edge server, the slower the service execution speed of the system at the same time, which results in a larger delay of the algorithm. We also notice that the CSO-DRL algorithm obtains a lower delay than DQN and converges at 35s, which is stable than DQN algorithm. Figure 10(b) shows the delay comparison between the DQN algorithm and the CSO-DRL algorithm under different CPU frequency of edge servers. It is obvious that the CSO-DRL algorithm achieves a lower delay than DQN. As the CPU frequencies of edge servers increase from 2GHz to 4GHz, the delay decreases following and remains at 27s when the CPU frequency of edge server is 4GHz.Figure11 shows the same group of experiments in terms of convergence performance and delay under different computation capabilities of edge servers. We vary the computation capabilities of edge servers from 0.6GHz to 1GHz. Figure 11(a) shows the convergence performance of the DQN algorithm and our CSO-DRL algorithm. We find that when the computation capability of devices is 0.8GHz, the delay of two algorithms is higher than that when the computation capability of devices is 1GHz. Thus, the smaller the computation capability of devices, the slower the service execution speed of the system at the same time, which results in a larger delay of the algorithm. We also notice that the CSO-DRL algorithm obtains a better delay than DQN and converges at 35s, which is stable than the DQN algorithm. Figure 11(b) shows the delay comparison between the DQN algorithm and the CSO-DRL algorithm under different CPU frequencies of devices. The delay of CSO-DRL algorithm increases slowly from 34.5s to 35s as the CPU frequency of devices increases from 0.7GHz to 0.8GHz and remains at about 34.3s when the CPU frequency is 1GHz. It is obvious that the CSO-DRL algorithm achieves a lower delay than DQN under the same CPU frequency of devices.


## 5.1. Experiment Setup

To ensure the practicability of the algorithm, we use a real-world data set of taxi trajectories in Shanghai (https://cse.hkust.edu.hk/scrg/) to evaluate the experiment in this paper. This data set records the GPS trajectory data on February 20, 2007, in Shanghai, which contains the longitude and latitude of 4316 taxies. Here, we conduct the experiment in the real world to obtain the road segments and determine the location of service offloading. Furthermore, the simulation experiments are conducted to compare the latency with other baseline algorithms, which are described as follows:(i)
Local-only: executing all services locally(ii)
Offload-only: offloading all services to RSUs deployed with edge servers(iii)
AC: actor-critic-based service offloading algorithm [44](iv)
DQN: DQN-based service offloading algorithm [45]We set the channel power gainH=−50dB when the distance is 1m, and the transmission bandwidth is W=100MHz. The noise power accepted without signal blocking is σ2=−100dBm. The computation capacity of the vehicular devices and edge servers are set to fdev=−0.8GHz and fedge=3GHz, respectively. We set the distance between the vehicular devices and the edge servers dt randomly. The size of the task is randomly assigned, and fi=ϵDi is the required computation resource, where ϵ is set at 0.5Gcyles/MB. The detailed simulation parameters are listed in Table 2.Table 2
Simulation parameters.
ParametersNumerical valueUnitChannel power gain,H−50dBTransmission bandwidth,W100MHzNoise power,σ2−100dBmDevices computation capacity,fdev0.8GHzEdge computation capacity,fedge3GHzTask size with random values,Di<60MbitData communication size between interacting services,Djre<MiMbitSince the data set records the location information of the vehicles every60s, so the time slot is set t=60s, and the total time is 24 hours. For fairness, we conduct all the algorithms in the same simulation environment. During the time slots, all the services are offloaded continually by our CSO-DRL algorithm, and then the delay is conducted to evaluate the efficiency of our algorithm.


## 5.2. Experimental Results

The hyperparameters in our algorithm affect the overall performance of the algorithm seriously. The experiments are conducted to determine the optimal values of these hyperparameters in our algorithm. First, we conduct experiments to select the optimal hyperparameters. Figure4 shows the convergence of the CSO-DRL algorithm under different learning rates. In the DDPG model, there are two training networks, which are the actor network and the critic network. We set the learning rates of the critic network and actor network separately. According to the training, we find that the algorithm cannot converge when the learning rate is λa=0.1 and λc=0.2. When the learning rate is λa=0.001 and λc=0.002, the algorithm can get a better convergence value. Therefore, the optimal learning rate of actor network λa=0.01, and the learning rate of critic network λc=0.02.Figure 4
Convergence of CSO-DRL under different learning rates.Next, the convergence of the algorithm under different discount factorsγ are conducted, which can indicate the weight of the value for the next action. The value of γ can affect the efficiency of the algorithm. As Figure 5 shows, the trained service offloading strategy performs best when the discount factor γ=0.5. Therefore, we set the value γ=0.5 in the following experiments.Figure 5
Convergence of CSO-DRL under different discount factors.Furthermore, the convergence of the algorithm under different exploration parametersσ is also conducted. Noticed that, the larger value for σ allows the agent to choose a prudent strategy, and make the evaluation of the q-value become more accurate, and then the training process will become more stable. As Figure 6, when our algorithm converges at σ=0.1, the delay fluctuates at 35s. The initial delay with the σ=0.1 is lower than the situation with σ=0.01. We noticed that when σ=1, the algorithm decreases at 280 episodes, but the convergence is unstable. Hence, we set σ=0.1 to obtain better performance in the following experiments.Figure 6
Convergence of CSO-DRL under different exploration parameters.With the parameters determined, we compare the delay of our algorithm with baseline algorithms. Figure7 shows the performance of different algorithms. From this figure, it can be observed that the delay of the CSO-DRL algorithm is lower than the other baseline algorithms significantly under the same service size. As the number of iterations increases, the AC algorithm cannot converge, while the DQN algorithm and CSO-DRL algorithm can achieve convergence. Because the actor network and critic network of the AC algorithm are updated at the same time, the action selection of the actor network depends on the value function of the critic network, but the critic network is difficult to converge. In contrast, since both DQN and DDPG have a dual network structure. While DQN is only suitable for discrete action spaces and the state space for service offloading is continuous, DQN cannot search the optimal offloading strategy accurately. On the other hand, the CSO-DRL algorithm explores a continuous action space and takes precise actions, which may result in an optimal policy and make the delay reduce significantly. We can also find that the CSO-DRL algorithm and the DQN algorithm can converge at 300 episodes, and the CSO-DRL algorithm obtains a better delay than DQN and converges at 35, which is more stable than the DQN algorithm.Figure 7
Convergence performance of different algorithms.Besides the comparison with the AC algorithm and the DQN algorithm, we also conduct the experiments by comparing the CSO-DRL algorithm with the local-only algorithm and the offload-only algorithm. In Figure7, it is obvious that the delay of local computing is too large due to the limitation of the computation capacity of the vehicular devices. Since the offload-only algorithm depends on the running state and the computation resources of the edge server, and the vehicles keep the state with resource competition for a long time, the CSO-DRL algorithm can obtain a better delay than offload-only algorithm.We also conduct the experiment under different service sizes (also called task sizes). Since the AC algorithm cannot converge, we compare the delay with other algorithms. As Figure8 shows, with the task sizes increased, the delay of the CSO-DRL algorithm is always lower than the other baseline algorithm for the same task size. We notice that the delay of CSO-DRL increases much slower than other baseline algorithms, which is significant when the task sizes increase doubled.Figure 8
Delay comparison under different task sizes.In order to explore the effect of the number of edge servers for the delay, we compare the delay of our proposed algorithm with other algorithms under different numbers of edge servers. As Figure9 shows, the delay of all algorithms except DQN is almost constant as the number of edge servers increases. With the increase in the number of edge servers, the DQN algorithm fluctuates at about 40s. Besides, the proposed CSO-DRL algorithm achieves the lowest delay. The reason is DDPG model can find the optimal value in the continuous action and obtain the optimal control policy.Figure 9
Delay comparison under different numbers of edge servers.Besides the delay comparison with other algorithms under different numbers of edge servers, we compare these algorithms under different computing capabilities of edge servers and computing capabilities of devices, which can be found in Figures10 and 11, respectively. Due to the nonconvergence of the AC algorithm, we only compared our proposed CSO-DRL algorithm with the DQN algorithm.Figure 10
(a) Convergence of different algorithms under different computation capabilities of the edge server. (b) Comparison between different algorithms.
(a)(b)Figure 11
(a) Convergence of different algorithms under different computation capabilities of devices. (b) Comparison between different algorithms.
(a)(b)Figure10 shows the same group of experiments in terms of convergence performance and delay under different computing capabilities of edge servers. We vary the computation capabilities of edge servers from 2GHz to 4GHz. Figure 10(a) shows the convergence performance of the DQN algorithm and our CSO-DRL algorithm. We find that when the computation capability of the edge server is 3GHz, the delay of two algorithms is higher than that when the computation capability of the edge server is 4GHz. Thus, the smaller the computation capability of the edge server, the slower the service execution speed of the system at the same time, which results in a larger delay of the algorithm. We also notice that the CSO-DRL algorithm obtains a lower delay than DQN and converges at 35s, which is stable than DQN algorithm. Figure 10(b) shows the delay comparison between the DQN algorithm and the CSO-DRL algorithm under different CPU frequency of edge servers. It is obvious that the CSO-DRL algorithm achieves a lower delay than DQN. As the CPU frequencies of edge servers increase from 2GHz to 4GHz, the delay decreases following and remains at 27s when the CPU frequency of edge server is 4GHz.Figure11 shows the same group of experiments in terms of convergence performance and delay under different computation capabilities of edge servers. We vary the computation capabilities of edge servers from 0.6GHz to 1GHz. Figure 11(a) shows the convergence performance of the DQN algorithm and our CSO-DRL algorithm. We find that when the computation capability of devices is 0.8GHz, the delay of two algorithms is higher than that when the computation capability of devices is 1GHz. Thus, the smaller the computation capability of devices, the slower the service execution speed of the system at the same time, which results in a larger delay of the algorithm. We also notice that the CSO-DRL algorithm obtains a better delay than DQN and converges at 35s, which is stable than the DQN algorithm. Figure 11(b) shows the delay comparison between the DQN algorithm and the CSO-DRL algorithm under different CPU frequencies of devices. The delay of CSO-DRL algorithm increases slowly from 34.5s to 35s as the CPU frequency of devices increases from 0.7GHz to 0.8GHz and remains at about 34.3s when the CPU frequency is 1GHz. It is obvious that the CSO-DRL algorithm achieves a lower delay than DQN under the same CPU frequency of devices.


## 6. Conclusion

In this paper, we propose a collaborative service offloading approach with deep reinforcement learning in vehicular edge computing named CSO-DRL. Our approach first divides the road segments by k-means-based algorithm through analyzing the trajectory data of vehicles, and then the offloading location is determined by observing the vehicle running status. Secondly, the interacting services are discovered by a parallel frequent pattern-based algorithm efficiency. Furthermore, the collaborative service offloading algorithm is presented by the DDPG model for offloading the interacting services, which can reduce the service requesting delay and communication delay together. Finally, the efficiency of the algorithms is evaluated by real-world data-based simulation experimental evaluations. The results show our algorithm can get a better delay in obtaining the optimal service offloading strategy than other baseline algorithms.Although our approach considers the mobility of vehicles in service offloading and reveals the impact of interrelationship between services on service offloading. There are some avenues for our future studies. In reality, the vehicles are in a complex scenario with resource competition. Thus, how to design offloading strategies for multiusers VEC by considering the computation resource competition is a nonignored problem. We also noticed that due to the limitation of device power, we will present the service offloading strategy to investigate the multiobjective optimization of reducing the energy consumption of vehicular devices and latency in VEC.



---

*Source: 1163177-2022-09-05.xml*