# A Resource Allocation Scheme for Real-Time Energy-Aware Offloading in Vehicular Networks with MEC

Authors: Haibo Zhang; Xiangyu Liu; Xia Bian; Yan Cheng; Shengting Xiang
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

With the emergence of new vehicular applications, computation offloading based on mobile edge computing (MEC) has become a promising paradigm in resource-constrained vehicular networks. However, an unreasonable offloading strategy in offloading can cause serious energy consumption and latency. A real-time energy-aware offloading scheme for vehicle networks, based on MEC, is proposed to optimize communication and computation resource to decrease energy consumption and latency. Because the problem of computation offloading and resource allocation is the mixed-integer nonlinear problem (MINLP), this article uses a bi-level optimization method to transform the original MINLP into two subproblems. Furthermore, considering the mobility of vehicle users (V-UEs) and the availability of cloud resources, an offloading scheme based on deep reinforcement learning (DRL) is adopted to help users make the optimal offloading decisions. The simulation results show that the proposed bi-level optimization algorithm reduces the total overhead by nearly 40% to the compared algorithm.


---

## Body

## 1. Introduction

With the continuous progress of the sixth-generation (6G) communication and the Internet of vehicles [1–6], novel mobile applications in vehicular networks have promoted demand for the low-delay high-quality services, for example, interactive gaming, augmented reality/virtual reality (AR/VR), face recognition, and natural language processing [7]. The demand of computing is prominent, so that it frequently exceeds the capacity that local mobile devices can provide [8]. In general, vehicle units have limited computation resources (e.g., central process unit (CPU) frequency and memory) and battery lifetime, which will bring an unprecedented challenge for effectively executing these mobile applications [9–11]. To cope with the explosive application demands of vehicular terminals, tasks of vehicles are migrated from the vehicular units to the cloud servers [12], referred to as computation offloading. However, the cloud servers are spatially far from mobile vehicles, which may cause high transmission latency and severe degradation of the offloading efficiency.Mobile edge computing (MEC) technology [13, 14] provided the cloud computation resource close to the mobile vehicular terminals. Compared with MCC, MEC can provide lower latency, high bandwidth, and computational flexibility in computation offloading. In [15], the authors studied the offloading problem among multiple devices for mobile edge cloud computing and proposed a game-theoretic approach for achieving efficient offloading mechanisms. To minimize the energy consumption of the MEC offloading system, the authors in [10] proposed an energy-efficient computation offloading scheme, which jointly optimized the offloading decisions and the radio resource allocation strategies. In [16], the authors proposed the energy-aware offloading scheme to investigate the trade-off between the energy consumption and the latency, optimizing the computation offloading and resource allocation under the limited energy and sensitive latency. But the weighting factor to investigate the trade-off between energy consumption and latency is a constant, which can define the value of the weighting factor subjectively. The work ignored the service conditions of the battery and the user-specific demands. In addition, the computation offloading, especially in ultra-dense networks (UDNs), may cause more interference and result in unexpected transmission latency [17]. Therefore, it is unreasonable to offload all computation tasks to the MEC servers. As a result, it is critical to make an efficient offloading decision and study the trade-off between the energy consumption of vehicle units and the latency of the corresponding tasks. Based on the above discussion, in this article, we propose a real-time energy-aware offloading scheme to study the trade-off between the energy consumption and the task latency of the vehicle units, which will optimize the allocation of communication and computing resources. The motivations behind our work are attributed to the following observations:(1)
Build a network scenario that deploys multiple MEC servers and multiple request vehicles, where service nodes (MEC servers and vehicles) are equipped with limited wireless and computational resource.(2)
Because of the limited computation resource, the MEC-enabled base stations (MEC-BSs) cannot provide endless computation offloading services for all tasks from V-UEs. Therefore, the computation offloading decision should be rationally determined.(3)
The energy consumption and latency affect the quality of service of the vehicle units. The energy consumption and latency depend mainly on allocating transmission power and wireless channels when offloading the tasks to the MEC server.(4)
Depending on the using conditions of a battery of V-UE and user-specific demands, the user preference (i.e., the weighting factor) should be redefined to allow vehicle units to choose different optimization objectives according to the actual energy consumption.Therefore, based on the above discussions, we propose a real-time energy-aware offloading scheme to study the trade-off between the energy consumption and latency (transmission latency and execution latency) of the vehicle units, which optimizes the allocation of communication and computing resource. The main contributions of this article are as follows:(i)
We present an integrated framework for computation offloading and resource allocation in multicell scenarios. For V-UEs, we present a real-time energy-aware offloading scheme that combines with computation and communication resource allocation to minimize the weighted sum of energy consumption and latency.(ii)
We redefine the weighting factor with the residual energy, which is no longer the weighting factor of the V-UEs' subjective energy consumption and latency. The advantage of the updated definition is that the actual energy consumption is taken into account, and it can help maintain the lifetime of vehicle users.(iii)
For the mixed-integer non-linear problem (MINLP) of computation offloading and resource allocation, a bi-level optimization approach is adopted to transform the original MINLP problem into two subproblems, that is, a lower-level problem for seeking the allocation of power and channels, and an upper-level problem for task offloading. Furthermore, taking into account the mobility of vehicle users (V-UEs) and the availability of cloud resources, an offloading scheme based on deep reinforcement learning (DRL) is proposed to enable the users to make the optimal offloading decisions.The rest of the article is organized as follows: Section2 describes the system model and problem formulation. Section 3 presents the system model and problem formulations. Section 4 presents the optimal computation offloading and resource allocation scheme. The simulation results are discussed in Section 5, and Section 6 concludes the article.


## 2. Related Works

### 2.1. MEC-Enabled Vehicular Networks

The MEC-enabled vehicular network has been extensively studied recently, since it can support the communication of vehicles and edge servers for real-time information sharing in the Internet of vehicles. In [18], the authors presented a cooperative autonomous driving-oriented MEC-enabled 5G-V2X prototype system design with the knowledge of big data gathered from the environment. The authors in [19] geared the intelligent battery switch service management for electric vehicles by a MEC-driven architecture. In [20], the application of multisource data fusion to support vehicle on-road analysis was proposed. It can provide some services for vehicles of peeking around the corner, extending sensing range and reinforcing local observations. In [21], the traffic prediction algorithm based on MEC was proposed for addressing short-term traffic prediction. A positioning calibration mechanism by calculating the one-way delay existing in the transmission of vehicles based on MEC and global positioning system (GPS) was proposed in [22]. The above works demonstrated that the combination of MEC and vehicular network can be meaningful. For realizing these applications, the resource allocation and offloading scheme become topics worth studying.


### 2.2. Offloading Scheme in Vehicular Networks

In order to reduce the latency and the energy consumption in the network as well as ensure the reliability of V-UEs, many works have studied resource allocation and offloading scheme in the vehicular network. The authors in [23] designed a distributed best response algorithm based on the computation offloading game model to maximize the utility of each vehicle. But this work only regarded the MEC server as the object to offload tasks. In [24], the authors proposed a joint allocation of wireless resource and MEC computing resource algorithm including V2X links clustering and MEC computation resource scheduling. A hierarchical architecture was constructed based on MEC in [25] to minimize the total network delay, which jointly considers vehicle-to-RSU computation offloading, RSU peer offloading, and content caching. In [26], the authors constructed a three-layer architecture in vehicular network and formulated an energy-efficient computational offloading problem to minimize the overall power consumption while satisfying the delay constraint. The authors in [27] constructed an SDN-assisted MEC architecture for the vehicular network to reduce the system overhead by optimizing offloading decision, transmission power control, subchannels assignment, and computing resource allocation scheme. However, the above works ignored the dynamic balance of the latency and the energy consumption.Different from the above computation offloading and resource allocation strategies for vehicular MEC networks, we designed a real-time energy-aware offloading scheme that combines with computation and communication resource allocation considering the task deadline constraint and the balance of the energy consumption and latency. On this basis, we proposed an offloading and resource allocation scheme based on deep reinforcement learning (DRL) to minimize the total overhead.


## 2.1. MEC-Enabled Vehicular Networks

The MEC-enabled vehicular network has been extensively studied recently, since it can support the communication of vehicles and edge servers for real-time information sharing in the Internet of vehicles. In [18], the authors presented a cooperative autonomous driving-oriented MEC-enabled 5G-V2X prototype system design with the knowledge of big data gathered from the environment. The authors in [19] geared the intelligent battery switch service management for electric vehicles by a MEC-driven architecture. In [20], the application of multisource data fusion to support vehicle on-road analysis was proposed. It can provide some services for vehicles of peeking around the corner, extending sensing range and reinforcing local observations. In [21], the traffic prediction algorithm based on MEC was proposed for addressing short-term traffic prediction. A positioning calibration mechanism by calculating the one-way delay existing in the transmission of vehicles based on MEC and global positioning system (GPS) was proposed in [22]. The above works demonstrated that the combination of MEC and vehicular network can be meaningful. For realizing these applications, the resource allocation and offloading scheme become topics worth studying.


## 2.2. Offloading Scheme in Vehicular Networks

In order to reduce the latency and the energy consumption in the network as well as ensure the reliability of V-UEs, many works have studied resource allocation and offloading scheme in the vehicular network. The authors in [23] designed a distributed best response algorithm based on the computation offloading game model to maximize the utility of each vehicle. But this work only regarded the MEC server as the object to offload tasks. In [24], the authors proposed a joint allocation of wireless resource and MEC computing resource algorithm including V2X links clustering and MEC computation resource scheduling. A hierarchical architecture was constructed based on MEC in [25] to minimize the total network delay, which jointly considers vehicle-to-RSU computation offloading, RSU peer offloading, and content caching. In [26], the authors constructed a three-layer architecture in vehicular network and formulated an energy-efficient computational offloading problem to minimize the overall power consumption while satisfying the delay constraint. The authors in [27] constructed an SDN-assisted MEC architecture for the vehicular network to reduce the system overhead by optimizing offloading decision, transmission power control, subchannels assignment, and computing resource allocation scheme. However, the above works ignored the dynamic balance of the latency and the energy consumption.Different from the above computation offloading and resource allocation strategies for vehicular MEC networks, we designed a real-time energy-aware offloading scheme that combines with computation and communication resource allocation considering the task deadline constraint and the balance of the energy consumption and latency. On this basis, we proposed an offloading and resource allocation scheme based on deep reinforcement learning (DRL) to minimize the total overhead.


## 3. System Model and Problem Formulations

### 3.1. System Model

M Road Side Units (RSUs) are located along the straight road, each of which is connected to a MEC server. Each RSU covers one region, and vehicles can only access an RSU in the region where they are located. The vehicle can communicate with neighboring vehicles or RSUs to send or receive information. As shown in Figure 1, L request vehicles with a Poisson distribution are deployed, expressed as Vii∈1,2,…L. Each RSU serves Ujj∈1,2,…,M V-UEs. To multiplex spectrum, we assume that multiple RSUs share a single spectral resource, so there exists interference between RSUs. The bandwidth W is divided into N channels. V-UEs connect to the RSUs in the orthogonal frequency-division multiple access (OFDMA) mode, where each V-UE channel in the same RSU is orthogonal to others. In this network, V-UE i in RSU j has a computation-intensive task Ai,j=di,j,ci,j,Di,jmax to be completed, in which di,j is the size of computation input data, ci,j denotes the total number of CPU cycle required to accomplish the computation task, and Di,jmax denotes the maximal latency that the V-UE can tolerate. For each V-UE, its task can be executed either locally on itself or remotely in the MEC server via computation offloading. Let si,j be the offloading decision of V-UE i in RSU j. If the V-UE i in RSU j offloads its task to the MEC server, si,j=1, and si,j=0 otherwise. In order to increase the readability of the article, the definitions of some variables are listed in Table 1.Figure 1
System model.Table 1
Abbreviated parameters.
VariableConceptMThe number of RSUsLThe number of request vehiclesNThe number of channelsViThe request vehiclesUjThe RSU servesdi,jThe size of computation input dataci,jThe total number of CPU cycle to accomplish the computation taskDi,jmaxThe tolerable latency for the tasksi,jThe offloading decision of V-UEi in RSU jfi,locThe computation ability of V-UEfCThe computation ability of MEC serverti,jLThe time of local computationti,jCThe time of transmissionei,jLThe energy consumption of local computationei,jCThe energy transmission of local computationkThe coefficient that depends on the chip architectureRi,j,nThe transmission ratepi,j,nThe transmission powerhi,j,nThe channel gainIi,j,nThe interference of V-UEi in RSU jai,j,nThe factor of channel selectionwi,jThe weighting factorri,jEThe residual energy rateOi,jThe weighted sum of energy consumption and latency


#### 3.1.1. Local Computing

Definefi,loc as the computation power (i.e., CPU cycles per second) of V-UEs. When the task Ai,j is executed locally, the computation execution time ti,jL is expressed as(1)ti,jL=ci,jfi,loc.The corresponding energy consumption of the V-UE for local execution is given by(2)ei,jL=kfi,loc2ci,j,where k=10−26 is a coefficient that depends on the chip architecture [28, 29].Note thatfi,loc affects the computation time and energy consumptions simultaneously. The CPU cycle frequency can be scheduled via dynamic voltage and frequency scaling (DVS) technology [30].


#### 3.1.2. MEC Edge Computing

When the input data are transmitted to the MEC server through RSU, the transmission expenditure between the MEC server and the RSU is neglected [31, 32]. If the V-UE accesses the RSU over channel n, the achievable uplink transmission rate can be represented as(3)Ri,j,n=wlog21+pi,j,nhi,j,nσ2+Ii,j,n,where w is the channel bandwidth, w=W/N. pi,j,n and hi,j,n stand for the uplink transmission power and channel gain between the V-UE i and the RSU j via channel n, respectively. σ2 denotes the noise power. Ii,j,n is the interference of V-UE i in RSU j suffering from other V-UEs in adjacent RSUs on the same channel n, and it can be expressed as(4)Ii,j,n=∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,nj,where l denotes the lth except the jth RSU, hk,l,nj is the channel gain from V-UE k in RSU j over channel n, and Ul is the number of V-UEs in RSU l. Hence, the total uplink transmission rate for V-UE i in RSU j can be obtained as(5)Ri,j=∑n=1Nai,j,nRi,j,n,where ai,j,n∈0,1. ai,j,n=1 indicates that the channel n is assigned to V-UE i in RSU j to offload its task, and ai,j,n=0 otherwise. Let fC express the CPU cycle frequency of the MEC server, which is fixed during computation task execution [30, 33]. Then, the total edge computing time of the task includes the transmission time and computation time on the MEC server, which can be written as(6)ti,jC=di,jRi,j+ci,jfC.The corresponding energy consumption of V-UE is expressed as(7)ei,jC=∑n=1Nai,j,npi,j,ndi,jRi,j.Here, we assume that the time and energy consumption of computation result from the MEC server to the V-UE are neglected in this case because the size of output data is much smaller than the size of input data and the download data rate is very high in general, which is similar to the study [32].During the execution of a task, both latency and energy consumption will affect the V-UEs (i.e., battery energy limitation of V-UEs). Hence, we introduce the weighting factorwi,jwi,j∈0,1 to study the trade-off between latency and energy consumption, which can define to meet the user-specific demands [31]. Energy and latency can be saved by adjusting the weighting factor. However, we introduce the residual energy rate ri,jE of the battery into our model's weighting factor. It is defined as(8)wi,j'=wi,jri,jE,where ri,jE=Etotal−1−si,j∗ei,jL+si,j∗ei,jC/Etotal. Molecule Etotal−1−si,j∗ei,jL+si,j∗ei,jC is the residual energy of V-UE i in RSU j. Etotal is the battery capacity in Joules [34]. ri,jE is a variable that affects the real-time service conditions of the battery.According to (1) and (2), the overhead of the task locally computed on V-UE i in RSU j, namely, the weighted sum of energy consumption and latency Oi,jL, can be obtained as(9)Oi,jL=wi,j'∗ti,jL+1−wi,j'∗ei,jL.Letwi,jt=wi,j', and wi,je=1−wi,j'. Equation (9) can be simplified to(10)Oi,jL=wi,jt∗ti,jL+wi,je∗ei,jL.Similarly, the overhead of the task computed in the MEC sever can be expressed as(11)Oi,jC=wi,jt∗ti,jC+wi,je∗ei,jC.Therefore, the overhead of the V-UEi in RSU j can be obtained by(12)Oi,j=si,j∗Oi,jC+1−si,j∗Oi,jL.


### 3.2. Problem Formulation

As mentioned above, we model the problem in a multicell scenario. In the multicell network, interference management and computation offloading are considered [16]. We formulate the offloading and resource allocation for the MEC system as an optimization problem. The objective is to minimize the overhead of V-UEs, which is formulated as follows:(13)P1:mins,p,a,f∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jwi,jtci,jfi,local+wi,jekfi,local2ci,j,s.t.C1:si,jdi,jRi,j+ci,jfC+1−si,jci,jfi,local≤Di,jmax,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j+1−si,jwi,jekfi,local2ci,j≤Ei,jmax,∀i,j,C3:flocalmin≤fi,llocal≤flocalmax,∀i,j,C4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C6∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.ConstraintC1 is the maximum tolerance latency. C2 is the energy consumption that should not exceed the residual energy of the V-UE. C3 restricts the local CPU cycle frequency into a finite set of values. C4 guarantees the maximum transmission power on V-UE i in RSU j. C5 addresses that the interference on RSU j caused by V-UEs with offloading tasks in other RSUs on each channel must meet the predefined threshold I. C6 indicates that each V-UE can only be allocated at most one channel. C7 states the binary variables used to present channel allocation. ai,j,n=1 indicates that V-UE i uses subchannel n for task offloading, and ai,j,n=0 otherwise. C8 denotes offloading decisions as to the binary variables. In other words, C8 specifies that each V-UE completes its task either by local execution or by edge execution.


## 3.1. System Model

M Road Side Units (RSUs) are located along the straight road, each of which is connected to a MEC server. Each RSU covers one region, and vehicles can only access an RSU in the region where they are located. The vehicle can communicate with neighboring vehicles or RSUs to send or receive information. As shown in Figure 1, L request vehicles with a Poisson distribution are deployed, expressed as Vii∈1,2,…L. Each RSU serves Ujj∈1,2,…,M V-UEs. To multiplex spectrum, we assume that multiple RSUs share a single spectral resource, so there exists interference between RSUs. The bandwidth W is divided into N channels. V-UEs connect to the RSUs in the orthogonal frequency-division multiple access (OFDMA) mode, where each V-UE channel in the same RSU is orthogonal to others. In this network, V-UE i in RSU j has a computation-intensive task Ai,j=di,j,ci,j,Di,jmax to be completed, in which di,j is the size of computation input data, ci,j denotes the total number of CPU cycle required to accomplish the computation task, and Di,jmax denotes the maximal latency that the V-UE can tolerate. For each V-UE, its task can be executed either locally on itself or remotely in the MEC server via computation offloading. Let si,j be the offloading decision of V-UE i in RSU j. If the V-UE i in RSU j offloads its task to the MEC server, si,j=1, and si,j=0 otherwise. In order to increase the readability of the article, the definitions of some variables are listed in Table 1.Figure 1
System model.Table 1
Abbreviated parameters.
VariableConceptMThe number of RSUsLThe number of request vehiclesNThe number of channelsViThe request vehiclesUjThe RSU servesdi,jThe size of computation input dataci,jThe total number of CPU cycle to accomplish the computation taskDi,jmaxThe tolerable latency for the tasksi,jThe offloading decision of V-UEi in RSU jfi,locThe computation ability of V-UEfCThe computation ability of MEC serverti,jLThe time of local computationti,jCThe time of transmissionei,jLThe energy consumption of local computationei,jCThe energy transmission of local computationkThe coefficient that depends on the chip architectureRi,j,nThe transmission ratepi,j,nThe transmission powerhi,j,nThe channel gainIi,j,nThe interference of V-UEi in RSU jai,j,nThe factor of channel selectionwi,jThe weighting factorri,jEThe residual energy rateOi,jThe weighted sum of energy consumption and latency


### 3.1.1. Local Computing

Definefi,loc as the computation power (i.e., CPU cycles per second) of V-UEs. When the task Ai,j is executed locally, the computation execution time ti,jL is expressed as(1)ti,jL=ci,jfi,loc.The corresponding energy consumption of the V-UE for local execution is given by(2)ei,jL=kfi,loc2ci,j,where k=10−26 is a coefficient that depends on the chip architecture [28, 29].Note thatfi,loc affects the computation time and energy consumptions simultaneously. The CPU cycle frequency can be scheduled via dynamic voltage and frequency scaling (DVS) technology [30].


### 3.1.2. MEC Edge Computing

When the input data are transmitted to the MEC server through RSU, the transmission expenditure between the MEC server and the RSU is neglected [31, 32]. If the V-UE accesses the RSU over channel n, the achievable uplink transmission rate can be represented as(3)Ri,j,n=wlog21+pi,j,nhi,j,nσ2+Ii,j,n,where w is the channel bandwidth, w=W/N. pi,j,n and hi,j,n stand for the uplink transmission power and channel gain between the V-UE i and the RSU j via channel n, respectively. σ2 denotes the noise power. Ii,j,n is the interference of V-UE i in RSU j suffering from other V-UEs in adjacent RSUs on the same channel n, and it can be expressed as(4)Ii,j,n=∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,nj,where l denotes the lth except the jth RSU, hk,l,nj is the channel gain from V-UE k in RSU j over channel n, and Ul is the number of V-UEs in RSU l. Hence, the total uplink transmission rate for V-UE i in RSU j can be obtained as(5)Ri,j=∑n=1Nai,j,nRi,j,n,where ai,j,n∈0,1. ai,j,n=1 indicates that the channel n is assigned to V-UE i in RSU j to offload its task, and ai,j,n=0 otherwise. Let fC express the CPU cycle frequency of the MEC server, which is fixed during computation task execution [30, 33]. Then, the total edge computing time of the task includes the transmission time and computation time on the MEC server, which can be written as(6)ti,jC=di,jRi,j+ci,jfC.The corresponding energy consumption of V-UE is expressed as(7)ei,jC=∑n=1Nai,j,npi,j,ndi,jRi,j.Here, we assume that the time and energy consumption of computation result from the MEC server to the V-UE are neglected in this case because the size of output data is much smaller than the size of input data and the download data rate is very high in general, which is similar to the study [32].During the execution of a task, both latency and energy consumption will affect the V-UEs (i.e., battery energy limitation of V-UEs). Hence, we introduce the weighting factorwi,jwi,j∈0,1 to study the trade-off between latency and energy consumption, which can define to meet the user-specific demands [31]. Energy and latency can be saved by adjusting the weighting factor. However, we introduce the residual energy rate ri,jE of the battery into our model's weighting factor. It is defined as(8)wi,j'=wi,jri,jE,where ri,jE=Etotal−1−si,j∗ei,jL+si,j∗ei,jC/Etotal. Molecule Etotal−1−si,j∗ei,jL+si,j∗ei,jC is the residual energy of V-UE i in RSU j. Etotal is the battery capacity in Joules [34]. ri,jE is a variable that affects the real-time service conditions of the battery.According to (1) and (2), the overhead of the task locally computed on V-UE i in RSU j, namely, the weighted sum of energy consumption and latency Oi,jL, can be obtained as(9)Oi,jL=wi,j'∗ti,jL+1−wi,j'∗ei,jL.Letwi,jt=wi,j', and wi,je=1−wi,j'. Equation (9) can be simplified to(10)Oi,jL=wi,jt∗ti,jL+wi,je∗ei,jL.Similarly, the overhead of the task computed in the MEC sever can be expressed as(11)Oi,jC=wi,jt∗ti,jC+wi,je∗ei,jC.Therefore, the overhead of the V-UEi in RSU j can be obtained by(12)Oi,j=si,j∗Oi,jC+1−si,j∗Oi,jL.


## 3.1.1. Local Computing

Definefi,loc as the computation power (i.e., CPU cycles per second) of V-UEs. When the task Ai,j is executed locally, the computation execution time ti,jL is expressed as(1)ti,jL=ci,jfi,loc.The corresponding energy consumption of the V-UE for local execution is given by(2)ei,jL=kfi,loc2ci,j,where k=10−26 is a coefficient that depends on the chip architecture [28, 29].Note thatfi,loc affects the computation time and energy consumptions simultaneously. The CPU cycle frequency can be scheduled via dynamic voltage and frequency scaling (DVS) technology [30].


## 3.1.2. MEC Edge Computing

When the input data are transmitted to the MEC server through RSU, the transmission expenditure between the MEC server and the RSU is neglected [31, 32]. If the V-UE accesses the RSU over channel n, the achievable uplink transmission rate can be represented as(3)Ri,j,n=wlog21+pi,j,nhi,j,nσ2+Ii,j,n,where w is the channel bandwidth, w=W/N. pi,j,n and hi,j,n stand for the uplink transmission power and channel gain between the V-UE i and the RSU j via channel n, respectively. σ2 denotes the noise power. Ii,j,n is the interference of V-UE i in RSU j suffering from other V-UEs in adjacent RSUs on the same channel n, and it can be expressed as(4)Ii,j,n=∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,nj,where l denotes the lth except the jth RSU, hk,l,nj is the channel gain from V-UE k in RSU j over channel n, and Ul is the number of V-UEs in RSU l. Hence, the total uplink transmission rate for V-UE i in RSU j can be obtained as(5)Ri,j=∑n=1Nai,j,nRi,j,n,where ai,j,n∈0,1. ai,j,n=1 indicates that the channel n is assigned to V-UE i in RSU j to offload its task, and ai,j,n=0 otherwise. Let fC express the CPU cycle frequency of the MEC server, which is fixed during computation task execution [30, 33]. Then, the total edge computing time of the task includes the transmission time and computation time on the MEC server, which can be written as(6)ti,jC=di,jRi,j+ci,jfC.The corresponding energy consumption of V-UE is expressed as(7)ei,jC=∑n=1Nai,j,npi,j,ndi,jRi,j.Here, we assume that the time and energy consumption of computation result from the MEC server to the V-UE are neglected in this case because the size of output data is much smaller than the size of input data and the download data rate is very high in general, which is similar to the study [32].During the execution of a task, both latency and energy consumption will affect the V-UEs (i.e., battery energy limitation of V-UEs). Hence, we introduce the weighting factorwi,jwi,j∈0,1 to study the trade-off between latency and energy consumption, which can define to meet the user-specific demands [31]. Energy and latency can be saved by adjusting the weighting factor. However, we introduce the residual energy rate ri,jE of the battery into our model's weighting factor. It is defined as(8)wi,j'=wi,jri,jE,where ri,jE=Etotal−1−si,j∗ei,jL+si,j∗ei,jC/Etotal. Molecule Etotal−1−si,j∗ei,jL+si,j∗ei,jC is the residual energy of V-UE i in RSU j. Etotal is the battery capacity in Joules [34]. ri,jE is a variable that affects the real-time service conditions of the battery.According to (1) and (2), the overhead of the task locally computed on V-UE i in RSU j, namely, the weighted sum of energy consumption and latency Oi,jL, can be obtained as(9)Oi,jL=wi,j'∗ti,jL+1−wi,j'∗ei,jL.Letwi,jt=wi,j', and wi,je=1−wi,j'. Equation (9) can be simplified to(10)Oi,jL=wi,jt∗ti,jL+wi,je∗ei,jL.Similarly, the overhead of the task computed in the MEC sever can be expressed as(11)Oi,jC=wi,jt∗ti,jC+wi,je∗ei,jC.Therefore, the overhead of the V-UEi in RSU j can be obtained by(12)Oi,j=si,j∗Oi,jC+1−si,j∗Oi,jL.


## 3.2. Problem Formulation

As mentioned above, we model the problem in a multicell scenario. In the multicell network, interference management and computation offloading are considered [16]. We formulate the offloading and resource allocation for the MEC system as an optimization problem. The objective is to minimize the overhead of V-UEs, which is formulated as follows:(13)P1:mins,p,a,f∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jwi,jtci,jfi,local+wi,jekfi,local2ci,j,s.t.C1:si,jdi,jRi,j+ci,jfC+1−si,jci,jfi,local≤Di,jmax,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j+1−si,jwi,jekfi,local2ci,j≤Ei,jmax,∀i,j,C3:flocalmin≤fi,llocal≤flocalmax,∀i,j,C4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C6∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.ConstraintC1 is the maximum tolerance latency. C2 is the energy consumption that should not exceed the residual energy of the V-UE. C3 restricts the local CPU cycle frequency into a finite set of values. C4 guarantees the maximum transmission power on V-UE i in RSU j. C5 addresses that the interference on RSU j caused by V-UEs with offloading tasks in other RSUs on each channel must meet the predefined threshold I. C6 indicates that each V-UE can only be allocated at most one channel. C7 states the binary variables used to present channel allocation. ai,j,n=1 indicates that V-UE i uses subchannel n for task offloading, and ai,j,n=0 otherwise. C8 denotes offloading decisions as to the binary variables. In other words, C8 specifies that each V-UE completes its task either by local execution or by edge execution.


## 4. Optimal Computation Offloading and Resource Allocation Scheme

### 4.1. Optimal Local Computing via CPU Cycle Frequency Scheduling

The latency and energy consumption of local computing is determined by local computing capabilityfi,loc. To minimize the local overhead, we can schedule the CPU cycle frequency of V-UE. Considering the constraints of C1, C2, and C3 in problem P1, P1.1 can be formed as follows:(14)P1.1:minfwi,jtci,jfi,loc+wi,jekfi,loc2ci,j,s.t.C1:ci,jfi,loc≤Di,jmax,C2:kfi,loc2ci,j≤Ei,jmax,C3:flmin≤fi,loc≤flmax.LetOi,jLfi,loc=wi,jtci,j/fi,j+wi,jekfi,loc2ci,j; we can see that the value of Oi,jLfi,loc depends only on fi,loc. We take the derivative of Oi,jLfi,loc with respect to fi,loc and set it to be zero. Then, we can obtain(15)41−si,jk2ci,j2fi,loc5Etotal+2si,jei,jCkci,jfi,loc3Etotal−1−si,jkci,j2fi,loc2Etotal+si,jci,jei,jCEtotal−ci,j=0.According to [25], it can be seen that fi,loc is a large order of magnitude value; therefore, fi,loc2 and fi,loc3 are negligible. Then, the above equation can be obtained as follows:(16)f∗=Etotal−si,jei,jC41−si,jci,jk5.Forfi,loc>f∗, Oi,jLfi,loc monotonously increases with the increase of fi,loc. Otherwise, it monotonously decreases with the increase of fi,loc. According to C1 and C2, we have(17)fi,loc≥ci,jDi,jmax=fl,fi,loc≤Ei,jmaxkci,j=fh.Combining (16) with the constraint C3, we define(18)fl′=maxflmin,fl,fh′=minflmax,fh.To ensure the feasible region offi,loc to be nonempty, fl′≤fh′ should hold. Hence, the optimal Oi,jLfi,loc can be calculated as(19)Oi,jL∗fi,loc=Oi,jLfl′,f∗≤fi,jl,Oi,jLf∗,fi,jl<f∗≤fh′,Oi,jLfh′,f∗>fh′.


### 4.2. Optimal Edge Computing through Resource Allocation and Computation Offloading

In a multicell scenario, we combine power allocation, channel allocation, and computation offloading to minimize the V-UEs' weighted sum of latency and energy consumption. Because there are two binary variables, a large number of other variables and interference terms exist. Therefore, the problem is nonconvex and MINLP, the bi-level optimization approach is taken to solve it. The original MINLP problem is decoupled into a lower-level problem for seeking the optimal power allocation and optimal channel allocation and an upper-level issue for task offloading [35]. Simultaneously, considering the mobility of vehicle users (V-UEs) and the availability of cloud resources, an offloading scheme based on DRL is proposed to enable the users to make the optimal offloading decisions [36].(20)P1.2:mins,p,aFs,p,a=∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j+ci,jfC+1−si,jci,jfi,local≤Di,jmax,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j+1−si,jwi,jekfi,local2ci,j≤Ei,jmax,∀i,j,C3:flocalmin≤fi,local≤flocalmax,∀i,j,C4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C6:∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.Because both the objective function and constraints are nonconvex,P1.2 is a nonconvex problem and belongs to mixed-integer non-linear problem.Proof: the proof of nonconvexity of objective function is shown in AppendixA. The proof of nonconvexity of constraints refers to the proof of objective functionHowever, formula (21) is an MINLP and it is very hard to find its optimal solution. Next, we use the bi-level optimization approach to solve it. Note that P1.2 involves two embedded problems. One is to calculate the computing overhead of the entire system only when the offloading strategy s is known, and the other is to be inversely influenced by power p and channel allocation strategy a.


### 4.3. Bi-Level Optimization Approach

In the view ofP1.2, we adopt a bi-level optimization approach to solve the original problem. The bi-level optimization problem is treated as a multistage game between the upper problem and the lower problem. Firstly, given the task offloading strategy s, the optimal power allocation p and channel allocation strategy a can be solved by function Fs,p,a. Then, according to the optimal power p∗s and channel allocation strategy a∗s, which have been calculated in lower-level problem, the optimal task offloading strategy s∗ is solved by function Fs,p∗s,a∗s in the upper-level problem.


#### 4.3.1. Lower-Level Problem

Given the task offloading strategys, the local overhead becomes the known quantity, and the lower-level problem can be written in the form of P1.2 as follows:(21)P2:mins,p,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j+ci,jfC≤Di,jmax,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,jC4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,jC5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,nC6:∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.ConstraintC1 can be written as si,jdi,j/Ri,j≤χi, where χi=Di,jmax−si,jci,j/fC. si,jdi,j/Ri,j is upper-bounded by χi for all feasible solutions. By replacing each term si,jdi,j/Ri,j in the objective function of P2 with its upper bound si,jχi, we obtain the following convex approximation of P2.1:(22)P2.1:minp,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j≤χi,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,j,C3:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C7:ai,j,n∈0,1,∀i,j,n.Obviously,P2.1 is almost a strictly convex problem, except for the discrete channel assignment value ai,j,n. Relaxing the integer variable ai,j,n to be continuous between [0, 1], the problem of P2.1 is written as the Lagrangian function:(23)Lλ1,λ2,λ3,p,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗+λ1si,jdi,jχi−Ri,j+λ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j+λ3ai,j,npi,j,n−pmax.For fixeds, the optimal power and channel allocation value can be obtained by partial derivation of the pi,j,n and ai,j,n in formula (23). Therefore, the deviation of pi,j,n in formula (23) can be calculated:(24)∂Lλ1,λ2,λ3,p,a∂pi,j,n|pi,j,n=pi,j,n∗=0.Then, we can obtain the optimal power of V-UEi on channel n.(25)pi,j,n∗=−B+B2+4AC2A,A=2si,j1−si,jai,j,n2wi,jχiEtotal,B=1−wi,jai,j,nsi,jχi+si,j2ei,jLwi,jai,j,nχiEtotal+λ2ai,j,nsi,jdi,jEi,jmax3i,j,n−1−si,jai,j,nsi,jχi2Etotal−1−si,jai,j,nsi,jwi,jci,jχiEtotalfC,C=−λ1+λ2wi,jln2.Once the optimal power allocationp∗s is calculated, the optimal channel allocation strategy can be calculated through the following expression:(26)∂Lλ1,λ2,λ3,p,a∂ai,j,n|ai,j,n=ai,j,n∗=0.Then, the optimal channel allocationai,j,n∗ can be obtained as follows:(27)ai,j,n∗=12si,jpi,j,n∗+ci,j2si,jpi,j,n∗fC−1−wi,jEtotal2si,j1−si,jpi,j,n∗wi,j−ei,jL2si,jpi,j,n∗−λ2di,jEtotal2si,j1−si,jpi,j,n∗wi,jχi2Ei,jmax−λ3Etotal2si,j21−si,jpi,j,n∗χi2,where variable matrices λ1, λ2, and λ3 are updated using their corresponding subgradients:(28)λ1m+1=λ1m+μ1si,jdi,jχi−Ri,j,(29)λ2m+1=λ2m+μ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j,(30)λ3m+1=λ3m+μ3ai,j,npi,j,n−pmax,where μ1, μ2, and μ3 are the appropriate step sizes of the subgradient algorithm.Hence, the pseudocode of the algorithm is shown in Algorithm 1. Algorithm 1 details the iterative solution process for resource allocation using the two-tier optimization algorithm.Algorithm 1: Optimal resource allocation algorithm.
(1)
Input：si,jεIdd, Oi,j(2)
Output:pi,j,n∗ai,j,n∗,(3)
Initialize weights matrixpi,j,n, ai,j,n(4)
foro=1:Oi,j(5)
m⟵0(6)
whilem≤Idd orpi,j,n∗m+1−pi,j,n∗m>ε(7)
forn=1:N(8)
fori=1:L(9)
Calculatepi,j,n∗ according to (25)(10)
Obtain the optimal channel allocationai,j,n∗ using (27)(11)
end for(12)
end for(13)
Update the variablesλ1, λ2 and λ3 using (28), (29) and (30)(14)
end while(15)
end for


#### 4.3.2. Upper-Level Problem

Given the power and channel allocation strategiesp∗s,a∗s for fixed s, at this point, the optimal objection becomes(31)P3:minsFp∗s,a∗s,s=∑i=1Ul∑j=1Msi,jwi,jtdi,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+ci,jfC+si,jwi,je∑n=1Nai,j,n∗pi,j,n∗di,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C8:si,j∈0,1,∀i,j,where the optimization problem is convex concerning si,j, and if the constraint C1 is satisfied, the optimal offloading decision can be found by Q-learning deep reinforcement learning offloading scheme.


### 4.4. Q-Learning Deep Reinforcement Learning Algorithm

Q-Learning deep reinforcement learning algorithm can ensure the V-UEs make optimal offloading decisions based on the dynamics of the system in terms of V-UE and cloudlet behaviors. The problem is represented as a Markov decision process (MDP), taking into account the number of task requests for all V-UEs, the maximum number of service resources that a V-UE can provide and the size of the number of remaining tasks in the edge cloud, and the distance between the V-UE and edge cloud as state space. The primary objective is to find the optimal actions on how many tasks the V-UE should process locally and how many tasks will be offloaded to each cloudlet such that the V-UE’s utility obtained by task execution is minimized while minimizing the energy consumption and latency [36].


#### 4.4.1. MDP-Based Offloading Problem Formulation

The problem of making an offloading decision for the V-UE is expressed as a finite MDP. The MDP can be represented as a tupleM=S,A,P,R, where S and A represent state and action spaces, Rs,a is the immediate reward obtained by doing action a at state s [36]. π is a policy that maps from a state s to an action a, that is, πs=a. The primary goal of the V-UE is to find the optimal policy π∗ to minimize the V-UE’s utility.Therefore, the state spaceS is defined as follows:(32)S=s=Qu,Qum,Qc,D,where Qu, Qum, Qc, and D denote the number of task requests for all V-UEs, the maximum number of service resources that a V-UE can provide, and the size of the number of remaining tasks in the edge cloud and distance state, respectively.The action spaceA is defined as(33)A=a=a0,…,ai,…,aN|ai∈0,1,…,amax,where a0 represents locally processed task sequence, and aii=1,…,N is the number of tasks to be offloaded to the cloudlet i. amax is maximum number of tasks that are processed locally or offloaded to the MEC within each decision period. In addition, the total number of tasks per action a must be equal to or less than the number of tasks currently remained in V-UE’s queue.Immediate reward for V-UEs minimizes overhead when making optimal offloading decisions in each system state. Therefore, we define the immediate reward functionRs,a given an action a at state s as follows:(34)Rs,a=Us,a−Cs,a,where Us,a and Cs,a are immediate utility and cost functions. For the utility, we define the utility function as (35)Us,a=ρOi,jL+Oi,jC,where ρ stands for a utility constant. The immediate cost function Cs,a is expressed as(36)Cs,a=η1Es,a+η2Ts,a,where η1 and η2 are a constant. Es,a is energy consumption, and Ts,a is delay, that is,(37)Es,a=a0ei,jLs,a+aiei,jCs,a,Ts,a=a0ti,jLs,a+aiti,jCs,a.


#### 4.4.2. DQN-Based Offloading Decision Algorithm

To find out the solution based on MDP offloading problem, we propose an online learning scheme based on the model-free deep RL algorithm, called deep Q-network (DQN) [37]. In the DQN-based learning scheme, the V-UE acts as an agent that interacts with the mobile edge cloud environment, receives sensory data, and performs a valid action. Then, the agent selects an offloading action at for state st at time slot t in a fashion that minimizes the future discounted reward over a long run. A deep neural network, called a deep Q-network, is used to approximate the optimal action-value function [37]:(38)Q∗s,a=−maxπErt+∑k=1∞γkrt+k|st=s,at=a,π.Taking actiona at the state s at the time slot t in the offloading policy π discounted through γ to minimize the sum of reward rt. γ∈0,1 is a discount factor to balance immediate reward and future reward. Q∗s,a denotes the expected total reward in state s and action a. Note that E. denotes the expectation function.TheQ-network can be regarded as a neural network approximator, which is used to estimate the action-value function. θ is a weight value of Qs,a;θ. At each decision period, the V-UE first takes the state vector S=Qu,Qum,Qc,D as the input of the Q-network and obtains the Q-values Qs,. for all possible action a as outputs. Then, the V-UE selects the action according to the ε−greedy exploration method.Furthermore, theQ-network can be trained by iteratively adjusting the weights θ to minimize the loss function sequence, where the loss function at time slot t is(39)Ltθt=−Ert+γmaxa′Qst+1,a′;θt−1−Qst,at,θt2,where θt represents the network parameters in iteration t, and the previous network parameters θt−1 are used to compute the target rt+γmaxa′Qst+1,a′.In other words, given a transitionst,at,rt,st+1, the weights θ updates the Q-network in a way that minimizes the squared error loss between the current predicted Q-value of Qst,at and the target Q-value of rt+γmaxa′Qst+1,a′.In addition, in the DQN algorithm, the empirical replay technique is used as the training method to improve theQ-network instability due to the use of a non-linear approximation function. More specifically, the V-UE’s experiences set et=st,at,rt,st+1 are stored in a replay memory Ω=et−ψ,…,et, where ψ is the replay memory capacity. While training the Q-network, a random mini-batch is taken from the replay memory instead of the most recent transition et.The pseudocode of Algorithm 2, a detailed DQN-based offloading decision algorithm is presented. The lines (2)-(4) are recursive, presenting the V-UE’s action on making an offloading decision at the beginning of each decision period. Then, theQ-values are estimated by using Q-network. The lines (5)-(7) are to train the Q-network by using the experience replay method.Algorithm 2: DQN-based offloading decision algorithm.
(1)
：Initialize:t=1, weight value θQ=0, max_convergence_time(2)
: fort=1,2,..., do(3)
: whilet<max_convergence_time+1 do(4)
: Observer current statest(5)
: Select actionat according to ε−greedy policy(6)
: Offload the tasks according to actionat and observe reward rt=Rst,at and next statest+1(7)
: Store experiencest,at,rt,st+1 into replay memoryΩ(8)
: Randomly select a set of transitionss,a,r,s′ from replay memoryΩ(9)
: Train the Q-network based on selected transitions usingrt+γmaxa′Qst+1,a′;θt−1−Qst,at;θt2 as loss function(10)
: end for(11)
t=t+1:(12)
: end while(13)
: until the algorithm converges


## 4.1. Optimal Local Computing via CPU Cycle Frequency Scheduling

The latency and energy consumption of local computing is determined by local computing capabilityfi,loc. To minimize the local overhead, we can schedule the CPU cycle frequency of V-UE. Considering the constraints of C1, C2, and C3 in problem P1, P1.1 can be formed as follows:(14)P1.1:minfwi,jtci,jfi,loc+wi,jekfi,loc2ci,j,s.t.C1:ci,jfi,loc≤Di,jmax,C2:kfi,loc2ci,j≤Ei,jmax,C3:flmin≤fi,loc≤flmax.LetOi,jLfi,loc=wi,jtci,j/fi,j+wi,jekfi,loc2ci,j; we can see that the value of Oi,jLfi,loc depends only on fi,loc. We take the derivative of Oi,jLfi,loc with respect to fi,loc and set it to be zero. Then, we can obtain(15)41−si,jk2ci,j2fi,loc5Etotal+2si,jei,jCkci,jfi,loc3Etotal−1−si,jkci,j2fi,loc2Etotal+si,jci,jei,jCEtotal−ci,j=0.According to [25], it can be seen that fi,loc is a large order of magnitude value; therefore, fi,loc2 and fi,loc3 are negligible. Then, the above equation can be obtained as follows:(16)f∗=Etotal−si,jei,jC41−si,jci,jk5.Forfi,loc>f∗, Oi,jLfi,loc monotonously increases with the increase of fi,loc. Otherwise, it monotonously decreases with the increase of fi,loc. According to C1 and C2, we have(17)fi,loc≥ci,jDi,jmax=fl,fi,loc≤Ei,jmaxkci,j=fh.Combining (16) with the constraint C3, we define(18)fl′=maxflmin,fl,fh′=minflmax,fh.To ensure the feasible region offi,loc to be nonempty, fl′≤fh′ should hold. Hence, the optimal Oi,jLfi,loc can be calculated as(19)Oi,jL∗fi,loc=Oi,jLfl′,f∗≤fi,jl,Oi,jLf∗,fi,jl<f∗≤fh′,Oi,jLfh′,f∗>fh′.


## 4.2. Optimal Edge Computing through Resource Allocation and Computation Offloading

In a multicell scenario, we combine power allocation, channel allocation, and computation offloading to minimize the V-UEs' weighted sum of latency and energy consumption. Because there are two binary variables, a large number of other variables and interference terms exist. Therefore, the problem is nonconvex and MINLP, the bi-level optimization approach is taken to solve it. The original MINLP problem is decoupled into a lower-level problem for seeking the optimal power allocation and optimal channel allocation and an upper-level issue for task offloading [35]. Simultaneously, considering the mobility of vehicle users (V-UEs) and the availability of cloud resources, an offloading scheme based on DRL is proposed to enable the users to make the optimal offloading decisions [36].(20)P1.2:mins,p,aFs,p,a=∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j+ci,jfC+1−si,jci,jfi,local≤Di,jmax,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j+1−si,jwi,jekfi,local2ci,j≤Ei,jmax,∀i,j,C3:flocalmin≤fi,local≤flocalmax,∀i,j,C4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C6:∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.Because both the objective function and constraints are nonconvex,P1.2 is a nonconvex problem and belongs to mixed-integer non-linear problem.Proof: the proof of nonconvexity of objective function is shown in AppendixA. The proof of nonconvexity of constraints refers to the proof of objective functionHowever, formula (21) is an MINLP and it is very hard to find its optimal solution. Next, we use the bi-level optimization approach to solve it. Note that P1.2 involves two embedded problems. One is to calculate the computing overhead of the entire system only when the offloading strategy s is known, and the other is to be inversely influenced by power p and channel allocation strategy a.


## 4.3. Bi-Level Optimization Approach

In the view ofP1.2, we adopt a bi-level optimization approach to solve the original problem. The bi-level optimization problem is treated as a multistage game between the upper problem and the lower problem. Firstly, given the task offloading strategy s, the optimal power allocation p and channel allocation strategy a can be solved by function Fs,p,a. Then, according to the optimal power p∗s and channel allocation strategy a∗s, which have been calculated in lower-level problem, the optimal task offloading strategy s∗ is solved by function Fs,p∗s,a∗s in the upper-level problem.


### 4.3.1. Lower-Level Problem

Given the task offloading strategys, the local overhead becomes the known quantity, and the lower-level problem can be written in the form of P1.2 as follows:(21)P2:mins,p,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j+ci,jfC≤Di,jmax,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,jC4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,jC5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,nC6:∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.ConstraintC1 can be written as si,jdi,j/Ri,j≤χi, where χi=Di,jmax−si,jci,j/fC. si,jdi,j/Ri,j is upper-bounded by χi for all feasible solutions. By replacing each term si,jdi,j/Ri,j in the objective function of P2 with its upper bound si,jχi, we obtain the following convex approximation of P2.1:(22)P2.1:minp,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j≤χi,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,j,C3:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C7:ai,j,n∈0,1,∀i,j,n.Obviously,P2.1 is almost a strictly convex problem, except for the discrete channel assignment value ai,j,n. Relaxing the integer variable ai,j,n to be continuous between [0, 1], the problem of P2.1 is written as the Lagrangian function:(23)Lλ1,λ2,λ3,p,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗+λ1si,jdi,jχi−Ri,j+λ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j+λ3ai,j,npi,j,n−pmax.For fixeds, the optimal power and channel allocation value can be obtained by partial derivation of the pi,j,n and ai,j,n in formula (23). Therefore, the deviation of pi,j,n in formula (23) can be calculated:(24)∂Lλ1,λ2,λ3,p,a∂pi,j,n|pi,j,n=pi,j,n∗=0.Then, we can obtain the optimal power of V-UEi on channel n.(25)pi,j,n∗=−B+B2+4AC2A,A=2si,j1−si,jai,j,n2wi,jχiEtotal,B=1−wi,jai,j,nsi,jχi+si,j2ei,jLwi,jai,j,nχiEtotal+λ2ai,j,nsi,jdi,jEi,jmax3i,j,n−1−si,jai,j,nsi,jχi2Etotal−1−si,jai,j,nsi,jwi,jci,jχiEtotalfC,C=−λ1+λ2wi,jln2.Once the optimal power allocationp∗s is calculated, the optimal channel allocation strategy can be calculated through the following expression:(26)∂Lλ1,λ2,λ3,p,a∂ai,j,n|ai,j,n=ai,j,n∗=0.Then, the optimal channel allocationai,j,n∗ can be obtained as follows:(27)ai,j,n∗=12si,jpi,j,n∗+ci,j2si,jpi,j,n∗fC−1−wi,jEtotal2si,j1−si,jpi,j,n∗wi,j−ei,jL2si,jpi,j,n∗−λ2di,jEtotal2si,j1−si,jpi,j,n∗wi,jχi2Ei,jmax−λ3Etotal2si,j21−si,jpi,j,n∗χi2,where variable matrices λ1, λ2, and λ3 are updated using their corresponding subgradients:(28)λ1m+1=λ1m+μ1si,jdi,jχi−Ri,j,(29)λ2m+1=λ2m+μ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j,(30)λ3m+1=λ3m+μ3ai,j,npi,j,n−pmax,where μ1, μ2, and μ3 are the appropriate step sizes of the subgradient algorithm.Hence, the pseudocode of the algorithm is shown in Algorithm 1. Algorithm 1 details the iterative solution process for resource allocation using the two-tier optimization algorithm.Algorithm 1: Optimal resource allocation algorithm.
(1)
Input：si,jεIdd, Oi,j(2)
Output:pi,j,n∗ai,j,n∗,(3)
Initialize weights matrixpi,j,n, ai,j,n(4)
foro=1:Oi,j(5)
m⟵0(6)
whilem≤Idd orpi,j,n∗m+1−pi,j,n∗m>ε(7)
forn=1:N(8)
fori=1:L(9)
Calculatepi,j,n∗ according to (25)(10)
Obtain the optimal channel allocationai,j,n∗ using (27)(11)
end for(12)
end for(13)
Update the variablesλ1, λ2 and λ3 using (28), (29) and (30)(14)
end while(15)
end for


### 4.3.2. Upper-Level Problem

Given the power and channel allocation strategiesp∗s,a∗s for fixed s, at this point, the optimal objection becomes(31)P3:minsFp∗s,a∗s,s=∑i=1Ul∑j=1Msi,jwi,jtdi,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+ci,jfC+si,jwi,je∑n=1Nai,j,n∗pi,j,n∗di,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C8:si,j∈0,1,∀i,j,where the optimization problem is convex concerning si,j, and if the constraint C1 is satisfied, the optimal offloading decision can be found by Q-learning deep reinforcement learning offloading scheme.


## 4.3.1. Lower-Level Problem

Given the task offloading strategys, the local overhead becomes the known quantity, and the lower-level problem can be written in the form of P1.2 as follows:(21)P2:mins,p,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtdi,jRi,j+ci,jfC+wi,je∑n=1Nai,j,npi,j,ndi,jRi,j+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j+ci,jfC≤Di,jmax,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,jC4:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,jC5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,nC6:∑n=1Nai,j,n≤1,∀i,j,C7:ai,j,n∈0,1,∀i,j,n,C8:si,j∈0,1,∀i,j.ConstraintC1 can be written as si,jdi,j/Ri,j≤χi, where χi=Di,jmax−si,jci,j/fC. si,jdi,j/Ri,j is upper-bounded by χi for all feasible solutions. By replacing each term si,jdi,j/Ri,j in the objective function of P2 with its upper bound si,jχi, we obtain the following convex approximation of P2.1:(22)P2.1:minp,aFp,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C1:si,jdi,jRi,j≤χi,∀i,j,C2:si,j∑n=1Nai,j,npi,j,ndi,jRi,j≤Ei,jmax,∀i,j,C3:0≤∑n=1Nai,j,npi,j,n≤pmax,∀i,j,C5:∑k=1Ul∑l=1,l≠jMak,l,npk,l,nhk,l,n≤I,∀j,n,C7:ai,j,n∈0,1,∀i,j,n.Obviously,P2.1 is almost a strictly convex problem, except for the discrete channel assignment value ai,j,n. Relaxing the integer variable ai,j,n to be continuous between [0, 1], the problem of P2.1 is written as the Lagrangian function:(23)Lλ1,λ2,λ3,p,a=∑i=1Ul∑j=1Msi,jwi,jtχi+ci,jfC+wi,je∑n=1Nai,j,npi,j,nχi+∑i=1Uj∑j=1M1−si,jOi,jL∗+λ1si,jdi,jχi−Ri,j+λ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j+λ3ai,j,npi,j,n−pmax.For fixeds, the optimal power and channel allocation value can be obtained by partial derivation of the pi,j,n and ai,j,n in formula (23). Therefore, the deviation of pi,j,n in formula (23) can be calculated:(24)∂Lλ1,λ2,λ3,p,a∂pi,j,n|pi,j,n=pi,j,n∗=0.Then, we can obtain the optimal power of V-UEi on channel n.(25)pi,j,n∗=−B+B2+4AC2A,A=2si,j1−si,jai,j,n2wi,jχiEtotal,B=1−wi,jai,j,nsi,jχi+si,j2ei,jLwi,jai,j,nχiEtotal+λ2ai,j,nsi,jdi,jEi,jmax3i,j,n−1−si,jai,j,nsi,jχi2Etotal−1−si,jai,j,nsi,jwi,jci,jχiEtotalfC,C=−λ1+λ2wi,jln2.Once the optimal power allocationp∗s is calculated, the optimal channel allocation strategy can be calculated through the following expression:(26)∂Lλ1,λ2,λ3,p,a∂ai,j,n|ai,j,n=ai,j,n∗=0.Then, the optimal channel allocationai,j,n∗ can be obtained as follows:(27)ai,j,n∗=12si,jpi,j,n∗+ci,j2si,jpi,j,n∗fC−1−wi,jEtotal2si,j1−si,jpi,j,n∗wi,j−ei,jL2si,jpi,j,n∗−λ2di,jEtotal2si,j1−si,jpi,j,n∗wi,jχi2Ei,jmax−λ3Etotal2si,j21−si,jpi,j,n∗χi2,where variable matrices λ1, λ2, and λ3 are updated using their corresponding subgradients:(28)λ1m+1=λ1m+μ1si,jdi,jχi−Ri,j,(29)λ2m+1=λ2m+μ2si,jai,j,npi,j,ndi,jEi,jmax−Ri,j,(30)λ3m+1=λ3m+μ3ai,j,npi,j,n−pmax,where μ1, μ2, and μ3 are the appropriate step sizes of the subgradient algorithm.Hence, the pseudocode of the algorithm is shown in Algorithm 1. Algorithm 1 details the iterative solution process for resource allocation using the two-tier optimization algorithm.Algorithm 1: Optimal resource allocation algorithm.
(1)
Input：si,jεIdd, Oi,j(2)
Output:pi,j,n∗ai,j,n∗,(3)
Initialize weights matrixpi,j,n, ai,j,n(4)
foro=1:Oi,j(5)
m⟵0(6)
whilem≤Idd orpi,j,n∗m+1−pi,j,n∗m>ε(7)
forn=1:N(8)
fori=1:L(9)
Calculatepi,j,n∗ according to (25)(10)
Obtain the optimal channel allocationai,j,n∗ using (27)(11)
end for(12)
end for(13)
Update the variablesλ1, λ2 and λ3 using (28), (29) and (30)(14)
end while(15)
end for


## 4.3.2. Upper-Level Problem

Given the power and channel allocation strategiesp∗s,a∗s for fixed s, at this point, the optimal objection becomes(31)P3:minsFp∗s,a∗s,s=∑i=1Ul∑j=1Msi,jwi,jtdi,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+ci,jfC+si,jwi,je∑n=1Nai,j,n∗pi,j,n∗di,jwlog1+pi,j,n∗hi,j,n/σ2+Ii,j,n+∑i=1Uj∑j=1M1−si,jOi,jL∗,s.t.C8:si,j∈0,1,∀i,j,where the optimization problem is convex concerning si,j, and if the constraint C1 is satisfied, the optimal offloading decision can be found by Q-learning deep reinforcement learning offloading scheme.


## 4.4. Q-Learning Deep Reinforcement Learning Algorithm

Q-Learning deep reinforcement learning algorithm can ensure the V-UEs make optimal offloading decisions based on the dynamics of the system in terms of V-UE and cloudlet behaviors. The problem is represented as a Markov decision process (MDP), taking into account the number of task requests for all V-UEs, the maximum number of service resources that a V-UE can provide and the size of the number of remaining tasks in the edge cloud, and the distance between the V-UE and edge cloud as state space. The primary objective is to find the optimal actions on how many tasks the V-UE should process locally and how many tasks will be offloaded to each cloudlet such that the V-UE’s utility obtained by task execution is minimized while minimizing the energy consumption and latency [36].


### 4.4.1. MDP-Based Offloading Problem Formulation

The problem of making an offloading decision for the V-UE is expressed as a finite MDP. The MDP can be represented as a tupleM=S,A,P,R, where S and A represent state and action spaces, Rs,a is the immediate reward obtained by doing action a at state s [36]. π is a policy that maps from a state s to an action a, that is, πs=a. The primary goal of the V-UE is to find the optimal policy π∗ to minimize the V-UE’s utility.Therefore, the state spaceS is defined as follows:(32)S=s=Qu,Qum,Qc,D,where Qu, Qum, Qc, and D denote the number of task requests for all V-UEs, the maximum number of service resources that a V-UE can provide, and the size of the number of remaining tasks in the edge cloud and distance state, respectively.The action spaceA is defined as(33)A=a=a0,…,ai,…,aN|ai∈0,1,…,amax,where a0 represents locally processed task sequence, and aii=1,…,N is the number of tasks to be offloaded to the cloudlet i. amax is maximum number of tasks that are processed locally or offloaded to the MEC within each decision period. In addition, the total number of tasks per action a must be equal to or less than the number of tasks currently remained in V-UE’s queue.Immediate reward for V-UEs minimizes overhead when making optimal offloading decisions in each system state. Therefore, we define the immediate reward functionRs,a given an action a at state s as follows:(34)Rs,a=Us,a−Cs,a,where Us,a and Cs,a are immediate utility and cost functions. For the utility, we define the utility function as (35)Us,a=ρOi,jL+Oi,jC,where ρ stands for a utility constant. The immediate cost function Cs,a is expressed as(36)Cs,a=η1Es,a+η2Ts,a,where η1 and η2 are a constant. Es,a is energy consumption, and Ts,a is delay, that is,(37)Es,a=a0ei,jLs,a+aiei,jCs,a,Ts,a=a0ti,jLs,a+aiti,jCs,a.


### 4.4.2. DQN-Based Offloading Decision Algorithm

To find out the solution based on MDP offloading problem, we propose an online learning scheme based on the model-free deep RL algorithm, called deep Q-network (DQN) [37]. In the DQN-based learning scheme, the V-UE acts as an agent that interacts with the mobile edge cloud environment, receives sensory data, and performs a valid action. Then, the agent selects an offloading action at for state st at time slot t in a fashion that minimizes the future discounted reward over a long run. A deep neural network, called a deep Q-network, is used to approximate the optimal action-value function [37]:(38)Q∗s,a=−maxπErt+∑k=1∞γkrt+k|st=s,at=a,π.Taking actiona at the state s at the time slot t in the offloading policy π discounted through γ to minimize the sum of reward rt. γ∈0,1 is a discount factor to balance immediate reward and future reward. Q∗s,a denotes the expected total reward in state s and action a. Note that E. denotes the expectation function.TheQ-network can be regarded as a neural network approximator, which is used to estimate the action-value function. θ is a weight value of Qs,a;θ. At each decision period, the V-UE first takes the state vector S=Qu,Qum,Qc,D as the input of the Q-network and obtains the Q-values Qs,. for all possible action a as outputs. Then, the V-UE selects the action according to the ε−greedy exploration method.Furthermore, theQ-network can be trained by iteratively adjusting the weights θ to minimize the loss function sequence, where the loss function at time slot t is(39)Ltθt=−Ert+γmaxa′Qst+1,a′;θt−1−Qst,at,θt2,where θt represents the network parameters in iteration t, and the previous network parameters θt−1 are used to compute the target rt+γmaxa′Qst+1,a′.In other words, given a transitionst,at,rt,st+1, the weights θ updates the Q-network in a way that minimizes the squared error loss between the current predicted Q-value of Qst,at and the target Q-value of rt+γmaxa′Qst+1,a′.In addition, in the DQN algorithm, the empirical replay technique is used as the training method to improve theQ-network instability due to the use of a non-linear approximation function. More specifically, the V-UE’s experiences set et=st,at,rt,st+1 are stored in a replay memory Ω=et−ψ,…,et, where ψ is the replay memory capacity. While training the Q-network, a random mini-batch is taken from the replay memory instead of the most recent transition et.The pseudocode of Algorithm 2, a detailed DQN-based offloading decision algorithm is presented. The lines (2)-(4) are recursive, presenting the V-UE’s action on making an offloading decision at the beginning of each decision period. Then, theQ-values are estimated by using Q-network. The lines (5)-(7) are to train the Q-network by using the experience replay method.Algorithm 2: DQN-based offloading decision algorithm.
(1)
：Initialize:t=1, weight value θQ=0, max_convergence_time(2)
: fort=1,2,..., do(3)
: whilet<max_convergence_time+1 do(4)
: Observer current statest(5)
: Select actionat according to ε−greedy policy(6)
: Offload the tasks according to actionat and observe reward rt=Rst,at and next statest+1(7)
: Store experiencest,at,rt,st+1 into replay memoryΩ(8)
: Randomly select a set of transitionss,a,r,s′ from replay memoryΩ(9)
: Train the Q-network based on selected transitions usingrt+γmaxa′Qst+1,a′;θt−1−Qst,at;θt2 as loss function(10)
: end for(11)
t=t+1:(12)
: end while(13)
: until the algorithm converges


## 4.4.1. MDP-Based Offloading Problem Formulation

The problem of making an offloading decision for the V-UE is expressed as a finite MDP. The MDP can be represented as a tupleM=S,A,P,R, where S and A represent state and action spaces, Rs,a is the immediate reward obtained by doing action a at state s [36]. π is a policy that maps from a state s to an action a, that is, πs=a. The primary goal of the V-UE is to find the optimal policy π∗ to minimize the V-UE’s utility.Therefore, the state spaceS is defined as follows:(32)S=s=Qu,Qum,Qc,D,where Qu, Qum, Qc, and D denote the number of task requests for all V-UEs, the maximum number of service resources that a V-UE can provide, and the size of the number of remaining tasks in the edge cloud and distance state, respectively.The action spaceA is defined as(33)A=a=a0,…,ai,…,aN|ai∈0,1,…,amax,where a0 represents locally processed task sequence, and aii=1,…,N is the number of tasks to be offloaded to the cloudlet i. amax is maximum number of tasks that are processed locally or offloaded to the MEC within each decision period. In addition, the total number of tasks per action a must be equal to or less than the number of tasks currently remained in V-UE’s queue.Immediate reward for V-UEs minimizes overhead when making optimal offloading decisions in each system state. Therefore, we define the immediate reward functionRs,a given an action a at state s as follows:(34)Rs,a=Us,a−Cs,a,where Us,a and Cs,a are immediate utility and cost functions. For the utility, we define the utility function as (35)Us,a=ρOi,jL+Oi,jC,where ρ stands for a utility constant. The immediate cost function Cs,a is expressed as(36)Cs,a=η1Es,a+η2Ts,a,where η1 and η2 are a constant. Es,a is energy consumption, and Ts,a is delay, that is,(37)Es,a=a0ei,jLs,a+aiei,jCs,a,Ts,a=a0ti,jLs,a+aiti,jCs,a.


## 4.4.2. DQN-Based Offloading Decision Algorithm

To find out the solution based on MDP offloading problem, we propose an online learning scheme based on the model-free deep RL algorithm, called deep Q-network (DQN) [37]. In the DQN-based learning scheme, the V-UE acts as an agent that interacts with the mobile edge cloud environment, receives sensory data, and performs a valid action. Then, the agent selects an offloading action at for state st at time slot t in a fashion that minimizes the future discounted reward over a long run. A deep neural network, called a deep Q-network, is used to approximate the optimal action-value function [37]:(38)Q∗s,a=−maxπErt+∑k=1∞γkrt+k|st=s,at=a,π.Taking actiona at the state s at the time slot t in the offloading policy π discounted through γ to minimize the sum of reward rt. γ∈0,1 is a discount factor to balance immediate reward and future reward. Q∗s,a denotes the expected total reward in state s and action a. Note that E. denotes the expectation function.TheQ-network can be regarded as a neural network approximator, which is used to estimate the action-value function. θ is a weight value of Qs,a;θ. At each decision period, the V-UE first takes the state vector S=Qu,Qum,Qc,D as the input of the Q-network and obtains the Q-values Qs,. for all possible action a as outputs. Then, the V-UE selects the action according to the ε−greedy exploration method.Furthermore, theQ-network can be trained by iteratively adjusting the weights θ to minimize the loss function sequence, where the loss function at time slot t is(39)Ltθt=−Ert+γmaxa′Qst+1,a′;θt−1−Qst,at,θt2,where θt represents the network parameters in iteration t, and the previous network parameters θt−1 are used to compute the target rt+γmaxa′Qst+1,a′.In other words, given a transitionst,at,rt,st+1, the weights θ updates the Q-network in a way that minimizes the squared error loss between the current predicted Q-value of Qst,at and the target Q-value of rt+γmaxa′Qst+1,a′.In addition, in the DQN algorithm, the empirical replay technique is used as the training method to improve theQ-network instability due to the use of a non-linear approximation function. More specifically, the V-UE’s experiences set et=st,at,rt,st+1 are stored in a replay memory Ω=et−ψ,…,et, where ψ is the replay memory capacity. While training the Q-network, a random mini-batch is taken from the replay memory instead of the most recent transition et.The pseudocode of Algorithm 2, a detailed DQN-based offloading decision algorithm is presented. The lines (2)-(4) are recursive, presenting the V-UE’s action on making an offloading decision at the beginning of each decision period. Then, theQ-values are estimated by using Q-network. The lines (5)-(7) are to train the Q-network by using the experience replay method.Algorithm 2: DQN-based offloading decision algorithm.
(1)
：Initialize:t=1, weight value θQ=0, max_convergence_time(2)
: fort=1,2,..., do(3)
: whilet<max_convergence_time+1 do(4)
: Observer current statest(5)
: Select actionat according to ε−greedy policy(6)
: Offload the tasks according to actionat and observe reward rt=Rst,at and next statest+1(7)
: Store experiencest,at,rt,st+1 into replay memoryΩ(8)
: Randomly select a set of transitionss,a,r,s′ from replay memoryΩ(9)
: Train the Q-network based on selected transitions usingrt+γmaxa′Qst+1,a′;θt−1−Qst,at;θt2 as loss function(10)
: end for(11)
t=t+1:(12)
: end while(13)
: until the algorithm converges


## 5. Simulation Results

In this section, the MATLAB simulation platform verifies that the proposed resource allocation mechanism is efficient in the MEC-based vehicle network. We follow the highway parameter settings described in the 3GPP TR 36.885 and build the system model as specified in the MEC white paper. Among them, the RSU is deployed on the highway side with a communication radius of 250 meters. In addition, the MEC server is deployed on the RSU to provide services to the vehicles. The specific simulation parameters are shown in Table2.Table 2
Simulation parameters.
ParametersValuesMaximum transmit powerpmax23dBmBandwidthW20MHzThe CPU frequency of MEC serversfC4GHz/cycleThe total number of CPU cyclesci,j[0.1,1]GHzThe minimum CPU frequency of V-UEsflmin0.2GHz/cycleThe maximum CPU frequency of V-UEsflmax1GHz/cycleNoise powerσ2-113dBmThe total energy consumptionEtotal100JThe maximum latencyDi,jmax0.5sThe maximum number of iterationsIdd600Step sizeμ12×10−18Step sizeμ210−5Step sizeμ310−10Power allocation precisionε10−5V-UE’s queue sizeQu10Cloudlets’ queue sizeQic4Number of distance statesH4Decision periodT60sInput data sizedi,j300,1600KBIn a multicell scenario, each RSU shares the same spectrum to improve spectral efficiency. However, V-UEs are connected to the RSU by OFDMA technology, and they suffer from the interference of neighboring RSUs. For example, assuming that each RSU has ten channels, each RSU has 25 V-UEs, and there are 5 RSUs.Figure2 shows the bi-level optimization approach compared with the F–W algorithm [16], the exhaustive resource allocation algorithm (ERAA) [38], and the nonresource allocation (no power allocation and channel allocation) in preference analysis. Although the iterative times of the ERAA are high, the optimal allocation value is found in all the cases, so it converges to almost optimal value in each iteration. As seen from Figure 2, the latency without resource allocation is the highest, followed by the F–W algorithm, and the bi-level optimization approach has the lowest latency. This fully shows that resource allocation can reduce the latency of the system. Furthermore, compared with the F–W algorithm, the proposed bi-level optimization approach can find better power allocation and channel allocation values so that the latency is lower.Figure 2
The total latency with different V-UEs.Figure3 shows that the total energy consumption increases with the increase in the number of users. When the number of users is less than 10, there is a slight difference in the total energy consumption among different algorithms due to the small number of users. When the number of users is more than 10, the total energy consumption using F–W algorithm increases gradually as the number of users increases. The total energy consumption without resource allocation keeps growing at high speed all the time. It can be seen that resource allocation is significant for reducing energy consumption. ERAA algorithm can converge to the optimal value through multiple iterations. The performance of the bi-level optimization approach we adopted is very close to that of the ERAA algorithm in terms of reducing energy consumption. Still, the complexity of ERAA is much higher than that of the bi-level optimization approach. Because the energy consumption is related to the transmission power, and the advantage of the bi-level optimization approach compared with the F–W algorithm is that it can find a better power allocation value, so the performance of the bi-level optimization approach is better than that of the F–W algorithm in terms of reducing the total energy consumption.Figure 3
The total energy consumption with different V-UEs.Figure4 shows the relationship between the number of V-UEs and the total overhead. The total overhead is the weighted sum of the latency and energy consumption. It can be seen from the figure that when the number of users reaches 20, the ERAA algorithm becomes stable and no longer increases. In other words, the ERAA algorithm eventually converges to the optimal value. But it needs to be iterated many times, which makes the complexity very high compared with the proposed bi-level optimization approach. It can be seen from Figure 4 that the total overhead is the largest when the resources do not allocate. Compared with the F–W algorithm, the bi-level approach we used better affects the channel and power allocation. In summary, considering the problem of minimum complexity and overhead, the bi-level optimization algorithm can provide better performance.Figure 4
The total overhead with different V-UEs.Figure5 shows the effect of different weighting factors on V-UEs’ sum of energy consumption and latency. The definition of the weighting factor considers the real-time residual energy awareness of V-UEs. Compared with the subjective weighting, we redefine the weighting factor w1=w∗rE, which can achieve a better compromise, and save more energy than w=0.8 and w=0.5 from Figure 6. Because energy consumption is proportional to the weighting factor and inversely proportional to latency, the weighting factor w1=w∗rE can decrease latency comparing with w=0.2 from Figure 7. It shows that the proposed algorithm can obtain lower cost than other weighting factors.Figure 5
The total overhead under different weighting factors.Figure 6
The total latency under various weighting factors.Figure 7
The total energy consumption under different weighting factors.Figure8 shows the learning curves, which present the total reward obtained by the offloading policy learned in each training episode. Note that an episode includes 5000 iterations (i.e., the number of decision periods), in each of which the V-UE selects an offloading action for each state according to the current policy learned by the proposed algorithm. As shown in Figure 8, the total reward obtained in each episode decreases steadily when the learning time, that is, the number of episodes, increases from 1 to 200. Then, all learning curves become stable and no longer decrease when the episode number is higher than 200. This result indicates that the proposed DQN-based learning algorithm converges after 200 learning episodes. It can also be seen from Figure 8 that when the discount factor gamma=0.4, the convergence speed is the fastest, and when gamma=0.8, the overhead is the smallest. Because minimal overhead is the primary consideration, gamma=0.8 is chosen.Figure 8
Learning curves.


## 6. Conclusions

In this work, we proposed a resource allocation scheme for offloading in real-time energy-aware vehicular networks with MEC to minimize the total overhead, which balanced latency and energy consumption. We jointly considered the offloading resource allocation and redefined the weighting factor based on the real-time residual energy aware of the V-UEs. In the multicell networks, a bi-level optimization approach was proposed to approximate the MINLP problem to a convex problem. By taking partial derivatives, we found a better resource allocation scheme, which simplified the problem and reduced the computational complexity. The scheme in this article reduced the total overhead by nearly 40% to the compared algorithm. Also, the proposed offloading scheme based on deep reinforcement learning (DRL) enabled the users to make the optimal offloading decisions by considering uncertainties of resource availabilities of users and cloudlets.



---

*Source: 8138079-2022-02-10.xml*