# Security and Cost-Aware Computation Offloading via Deep Reinforcement Learning in Mobile Edge Computing

Authors: Binbin Huang; Yangyang Li; Zhongjin Li; Linxuan Pan; Shangguang Wang; Yunqiu Xu; Haiyang Hu
Journal: Wireless Communications and Mobile Computing
 (2019)

---

## Abstract

With the explosive growth of mobile applications, mobile devices need to be equipped with abundant resources to process massive and complex mobile applications. However, mobile devices are usually resource-constrained due to their physical size. Fortunately, mobile edge computing, which enables mobile devices to offload computation tasks to edge servers with abundant computing resources, can significantly meet the ever-increasing computation demands from mobile applications. Nevertheless, offloading tasks to the edge servers are liable to suffer from external security threats (e.g., snooping and alteration). Aiming at this problem, we propose a security and cost-aware computation offloading (SCACO) strategy for mobile users in mobile edge computing environment, the goal of which is to minimize the overall cost (including mobile device’s energy consumption, processing delay, and task loss probability) under the risk probability constraints. Specifically, we first formulate the computation offloading problem as a Markov decision process (MDP). Then, based on the popular deep reinforcement learning approach, deep Q-network (DQN), the optimal offloading policy for the proposed problem is derived. Finally, extensive experimental results demonstrate that SCACO can achieve the security and cost efficiency for the mobile user in the mobile edge computing environment.


---

## Body

## 1. Introduction

With the increasing popularity of mobile devices (e.g., smart phones and tablets), the number of various mobile applications (i.e., virtual reality (VR) and face recognition) is explosively growing [1]. To process such a large number of complex mobile applications efficiently, mobile devices need to be equipped with considerable resources (i.e., high computing capacity and battery power) [2, 3]. Unfortunately, due to the limited physical size of mobile devices, they are usually resource-constrained. The conflict between the resource demand for executing complex tasks and the limited resource capacity of mobile devices impose a big challenge for mobile application execution, which drives the transformation of computing paradigm [4].To reduce this conflict, mobile edge computing has emerged as a promising computing paradigm with the objective of bringing the computing and storage capacities close to mobile devices [5, 6]. Within one-hop communication range of mobile devices, there are a number of edge servers who have enormous computation and storage resources. Therefore, mobile devices can offload computation tasks to edge servers directly through wireless channels [7], thereby significantly meeting the ever-increasing computation demands from mobile applications, reducing their processing delay, and saving the mobile devices’ energy consumption.Despite its advantages, computation task offloading via mobile edge computing inevitably encounters two challenges as follows: one is the offloading environmental dynamics, such as the time-varying channel quality and the task arrival, which can impact the offloading decisions. And the other is the security. According to several surveys, security is one of the critical issues in mobile edge computing [8–19]. Due to the open nature of the mobile edge computing environment, these tasks offloaded to the edge servers are susceptible to hostile attacks from outside. For example, the offloaded computation tasks from the mobile device to the edge servers can be intentionally overheard by a malicious eavesdropper. Hence, it needs to employ various types of security services to effectively defend against the hostile attacks and protect these tasks. However, using security services inevitably incurs lots of extra security time and security workload, which will increase the mobile device’s energy consumption and task’s processing time, and thereby influencing the offloading decisions. Therefore, it is a big challenge to design a proper task offloading policy to optimize the long-term weighted cost of mobile device’s energy consumption, task processing delay, and number of dropping tasks while satisfying risk rate constraint.To meet the aforementioned challenges, we propose a security and cost-aware computation offloading (SCACO) strategy in the mobile edge computing environment, the goal of which is to minimize the long-term cost under the risk rate constraint. More specifically, we formulate the computation offloading problem as a Markov decision process (MDP). Our formulated offloading problem is a high-dimensional decision-making problem. The DQN algorithm has achieved excellent performance in solving this kind of problem. To achieve this, a deep Q-network- (DQN-) based offloading scheme is proposed. Figure1 illustrates a DQN framework for computation task offloading in mobile edge computing. As shown in Figure 1, the environment state which consists of the number of arriving tasks, the mobile device and edge servers’ execution queue states, and the channel quality states can be observed. Based on the current state, an optimal action, e.g., how many tasks should be assigned to execute locally, how many tasks should be offload to edge servers, and which security level should be employed, is chosen by the agent. After taking an action at current state, the reward can be calculated, and the current state, the action taken, and the reward are stored into replay memory. The main contributions of this paper can be summarized as follows:(i)
We select appropriate security servers to guarantee the offloaded tasks’ security. The security overhead model [20] is exploited to quantify the security time. Based on it, the security workload can be measured. In our architecture, the total workload of the offloaded task consists of task execution workload and security workload.
(ii)
We formulate the security-aware task offloading problem as an infinite Markov decision process with the risk rate constraint, the main goal of which is to minimize the long-term computation offloading costs while satisfying the risk rate constraint in the dynamic environment.
(iii)
We propose a SCACO strategy based on the DQN algorithm to solve the proposed formulation. To demonstrate the efficiency of the SCACO strategy and the impact of the security requirement, we conduct extensive experiments, with respect to (w.r.t.) various performance parameters (such as the task arrival rate, the task workload, the risk coefficient, and the risk rate). The experimental results demonstrate that the SCACO strategy can minimize the long-term cost while satisfying the risk rate constraint.Figure 1
DQN framework for computation task offloading in mobile edge computing.We organize this paper as follows: Section2 summarizes the related work. Section 3 describes the system models. Section 4 formulates security-aware computation offloading problem formulation. Section 5 describes the SCACO strategy based on the DQN algorithm. Section 6 describes the experimental setup and analyzes experimental results. Section 7 concludes this paper and identifies future directions.


## 2. Related Work

To meet the quality of service for different types of mobile applications, the computation tasks can be offloaded to the edge servers with sufficient computation capacity. Accordingly, an increasing amount of research has focused on the problem of computation offloading in mobile edge computing. Specifically, in [21], an efficient one-dimensional search algorithm is designed to minimize the computation task scheduling delay under the power consumption constraint. In [2], a computation task offloading optimization framework is designed to jointly optimize the computation task execution latency and the mobile device’s energy. In [2], an online dynamic task offloading scheme is proposed to achieve a trade-off between the task execution delay and the mobile device’s energy consumption in mobile edge computing with energy harvesting devices. In [22], a suboptimal algorithm is proposed to minimize the maximal weighted cost of the task execution latency and the mobile device’s energy consumption while guaranteeing user fairness. In [23], a workload allocation scheme is proposed to jointly optimize the energy consumption and the execution delay in mobile edge computing with grid-powered system. However, all the above works mainly focus on optimizing the one-shot offloading cost and fail to characterize long-term computation offloading performance. Accordingly, these offloading schemes may not suit for some applications which the long-term stability is more important than the profits of handing one task.To optimize the long-term computation offloading performance, a lot of related works have been done. In particular, in [24], an efficient reinforcement learning-based algorithm is proposed to minimize the long-term computation task offloading cost (including both service delay and operational cost) in energy harvesting mobile edge computing systems. In [25], a deep Q-network-based strategic computation offloading algorithm is proposed to minimize the long-term cost which is the weighted sum of the execution delay, the handover delay, and the computation task dropping cost. In [26], the Lyapunov optimization-based dynamic computation offloading policy is proposed to minimize the long-term execution latency and task failure for a green mobile edge computing system with wireless energy harvesting-enabled mobile devices. In [27], the Lyapunov optimization method is utilized to minimize the long-term average weighted sum energy consumption of the mobile devices and the edge server in a multiuser mobile edge computing environment. In [28], the game theory and reinforcement learning is utilized to efficiently manage the distributed resource in mobile edge computing. However, none of the above work considers the impact of security issue on computation offloading. In fact, security cannot be ignored because it is a key issue in mobile edge computing. Therefore, the above schemes are not suitable for security-aware dynamic computation offloading in mobile edge computing.With the escalation of the security threatens of data in the cloud, mobile cloud environments, and mobile edge computing [8–19], some measures have been implemented to protect security-critical applications. Specifically, in [29], a task-scheduling framework with three features is presented for security sensitive workflow framework. In [30], an SCAS scheduling scheme is proposed to optimize the workflow execution cost under the makespan and security constraints in clouds. In [31], an SABA scheduling scheme is designed to minimize the makespan under the security and budget constraints. In [32], a security-aware workflow scheduling framework is designed to minimize the makespan and execution cost of workflow while meeting the security requirement. However, to the best of our knowledge, all of the above methods are mainly designed for the workflow scheduling in the cloud computing or mobile cloud computing environment. They are not suitable for computation offloading in mobile edge computing. In [17], a deep-learning-based approach is proposed to detect malicious applications on the mobile device, which greatly enhances the security of mobile edge computing. However, this paper mainly focuses on the detection of security threats. In [33], a joint optimization problem which includes the secrecy provisioning, computation offloading, and radio resource allocation is formulated, and an efficient algorithm is proposed to solve this joint optimization problem. However, this paper failed to optimize the long-term computation task offloading cost.All the above studies have focused on the security problem of workflow scheduling. Little attention has been paid to the effect of the security problem of task offloading on the long-term offloading cost. Motivated by that, in this paper, we mainly focus on security and cost-aware dynamic computation offloading in mobile edge computing. We try to minimize the long-term computation offloading cost while satisfying risk rate constraint.


## 3. System Models

In this section, we first provide an overview of the security-aware mobile edge computing model. Then, the security overhead model and network and energy model are presented. At last, we formulate the problem of security-aware computation task offloading. To improve the readability, the main notations and their semantics used in this paper are given in Table1.Table 1
Notations.
Symbols
Definition
n
Number of edge servers
λ
task
Tasks’ arrival rate
T
slot
The duration of the time slot
R
i
The link rate between MD and edge servereNBi
sl
c
,
i
The set of security levels of taskt offloaded to the ith edge server
Q
arr
τ
The number of arrival tasks at theτth time slot
Q
u
e
τ
The user executing queue’s state at theτth time slot
Q
c
e
τ
The edge server executing queue’s state at theτth time slot
C
τ
The immediate cost at theτth time slot
E
τ
The immediate energy consumption at theτth time slot
T
τ
The immediate delay at theτth time slot
D
τ
The number of dropping tasks at theτth time slot
P
max
The risk probability constraint of each offloaded task


### 3.1. Mobile Edge Computing Model

As depicted in Figure2, we consider in this paper a mobile edge computing system consisting of a single mobile user and n edge servers. The mobile user can generate a series of independent computation tasks [34, 35] which need to be scheduled to execute locally or to execute remotely. Due to the mobile device’s limited computing resource and battery capacity, all computation tasks cannot be executed locally within a timely manner. Therefore, a part of these can be offloaded to n edge servers with relatively rich resources. These offloaded tasks are first stored in a dedicated executing queue and then are executed sequentially. The system time is logically divided to equal length time slots, and the duration of each time slot is Tslot (in seconds). For convenience, we denote the index sets of the time slots as τ∈T=0,1,…. The value of Tslot is usually decided by the channel coherence time. It means that the channel remains static within a time slot but varies among different time slots [26]. At the beginning of each time slot, the user makes an optimal offloading decision.Figure 2
Computation task offloading framework with multiple edge servers.The mobile device can be denoted by a tripleMD=fu,Nu,Pu, where fu denotes the mobile device’s CPU cycle frequency, Nu denotes the number of the mobile device’s processor cores, and Pu denotes the mobile device’s power. Especially, Pu can be further represented by a tuple Pu=Pue,Putx, where Pue and Putx denote the mobile device’s computation power (in Watt) and transmitting power, respectively. The mobile device has an execution queue with its size Que. If the queue is already full, the new arrival tasks will be dropped.The computation task model widely used in the existing literature [34, 35] is adopted in this paper. According to it, a computation task can be abstracted into a three-field notation t=W,Dtx,Drx, in which W denotes the task computation workload (in CPU cycles per bit), Dtx denotes the task input data size (in MB), and Drx denotes the task output data size. In addition, we assume that the computation tasks’ arrival process for the mobile device MD follows a Poisson distribution with a parameter λtask.The set ofn edge servers can be denoted by eNB=eNB1,…,eNBi,…,eNBn, where eNBi denotes the ith edge server. Each edge server eNBi has different configurations, such as the number of processor cores and the processor frequency. We use a two tuple eNBi=fc,i,Nc,i to represent the edge server eNBi, where fc,i denotes its processor frequency and Nc,i denotes the number of processor cores. Each edge server has an execution queue with the size Qci. When the edge server receives the offloaded tasks from the mobile device, it firstly stores the tasks in its execution queue and then executes them sequentially. Let μci denote the task processing rate of the ith edge server. Therefore, μci can be calculated by the following equation:(1)μci=fc,iW.


### 3.2. Security Overhead Model

The computation tasks offloaded to the edge servers are confronted with security threats. Fortunately, confidentiality service and integrity service can guard against these common security threats [29–33, 36, 37]. Confidentiality service can protect data from stealing by enciphering methods. Meanwhile, integrity service can ensure that data are not tampered. By flexibly selecting these two security services, an integrated security protection is formed to protect against a diversity of security threats. Based on these above services, the offloading process of a task with security protection is shown in Figure 3. We can observe from Figure 3 that confidential service (denoted as E) is first employed to encrypt the offloaded task. The security levels and processing rates of cryptographic algorithms for confidential service [20, 30, 36, 37] can be found in Table 2. Then, to protect the offloaded computation task from alteration attacks, integrity service (denoted as H) is successively employed to implement a hash algorithm to it. The security levels and processing rates of hash functions for integrity service [20, 30, 36, 37] can be seen in Table 3. After the encrypted task is delivered to the ith edge server, it is firstly decrypted (denoted as DE) and its integrity is verified (denoted as IV), and then it is executed, and finally the computation result is sent to the mobile device.Figure 3
The task execution process with security services.Table 2
The encryption algorithms for confidential service.
Encryption algorithms
sl
c
f: security level
V
sl
cf: processing rate (Mb/s)
IDEA
1.0
11.76
DES
0.85
13.83
Blowfish
0.56
20.87
AES
0.53
22.03
RC4
0.32
37.17Table 3
The hash functions for integrity service.
Hash functions
sl
cf: security level
V
sl
cf: processing rate (Mb/s)
TIGER
1.0
75.76
RipeMD160
0.75
101.01
SHA-1
0.69
109.89
RipeMD128
0.63
119.05
MD5
0.44
172.41Each offloaded computation task may require the confidentiality service and integrity service with various security levels. For the sake of simplicity, letcf and ig represent the confidentiality service and integrity service, respectively. Let slc,i=slc,icfτ,slc,iigτ to be the set of security levels of task t offloaded to the ith edge server, where slc,icfτ represents the security level of confidentiality service and slc,iigτ represents the security level of integrity service. Security service incurs sometime overhead. According to [20], the total encryption overheads of task t offloaded to the ith edge server can be calculated by the following equation:(2)Tc,iE=∑type∈cf,igk⋅Dtxfu⋅Nu⋅Vslc,itype,where k=2.2 and Vslc,icf and Vslc,iig denote the processing rates of the security levels slcf and slig, respectively. The total decryption overheads of the offloaded task t in the ith edge server can be computed by the following equation:(3)Tc,iDE=fu⋅Nu⋅TuEfc,i⋅Nc,i.


### 3.3. Workload Shaping with Security Guarantee

The security services incur not only time overhead but also security workload. The security workload of taskt offloaded to the ith edge server is incurred by encrypting it in mobile device and decrypting it in edge server. Based on the security time overhead of task t, the security workload which is incurred by encrypting task t in the mobile device can be computed by the following equation:(4)SWc,iEt=Tc,iE⋅fu⋅Nu.The security workload which is incurred by the decrypting taskt in the ith edge server can be computed by the following equation:(5)SWc,iDEDtx=Tc,iDE⋅fc,i⋅Nc,i.Based on the security services introduced above, we further quantitatively calculate the risk probability of taskt with different security levels. Without loss of generality, we assume that the distribution of risk probability follows a Poisson probability distribution for any given time interval. The risk probability of an offloaded computational task t is related to the set slc,i=slc,icfτ,slc,iigτ of security levels. Let Pslc,ik denote the risk probability of an offloaded task which employs either of the two security services. Pslc,ik can be denoted by the following equation [38, 39]:(6)Pslc,ik=1−exp−λk1−slk,k∈cf,ig.The risk probabilityPt of an offloaded task t which employs these two security services with different levels can be computed by the following equation:(7)Pt=1−∏k∈cf,ig1−Pslc,ik.Given the risk probability constraint of each offloaded computational task isPmax, the risk probability Pt must meet the following constraint:(8)Pack≤Pmax.In order to minimize the security workload while satisfying security requirement, it is a critical problem to select the levels of security services for each task. We formulate this problem as follows:(9)minimizeSWt=SWuEt+SWc,iDEtsubject to8.As shown in Tables2 and 3, the levels of each security service are discrete. For a task t, there are 5×5 types of security service composition. Hence, we can traverse these compositions to find the optimal service levels.


### 3.4. Communication Model

The computation tasks offloaded from the mobile device to edge servers are performed by wireless channels. Due to user’s mobility, the wireless channel gain state is dynamically changing at each time slot, which induces dynamic change of wireless channel transmission rate. LetRiτ denote the transmission rate of wireless channel (i.e., data transmission capacity) from the mobile device to the ith edge server eNBi in the τth time slot, which is given as follows:(10)Riτ=Bilog21+PuGiσ2,∀i∈N,where Bi is the wireless channel bandwidth of the ith edge server, Pu is the transmission power of the mobile device, σ2 is the Gaussian white noise power, and Gi denotes the wireless channel gain of eNBi.


### 3.5. Problem Statement

The computation offloading process can be formulated as an infinite Markov decision process. At the beginning of each time slot, an offloading decision is made based on the current system state, which consists of the number of arrival tasks, the remaining number of tasks in the user’s executing queue, the transmission rates between user andn edge servers, and the remaining number of tasks in n edge servers’ executing queues. The offloading decision mainly determines the number of tasks assigned to execute locally and the number of tasks offloaded to different edge servers. To protect the offloaded computation tasks from malicious attacks, security services need to be employed. Security services incur time overheads and security workload. Hence, the objective of this paper is to minimize the long-term cost while meeting the risk probability constraint.


## 3.1. Mobile Edge Computing Model

As depicted in Figure2, we consider in this paper a mobile edge computing system consisting of a single mobile user and n edge servers. The mobile user can generate a series of independent computation tasks [34, 35] which need to be scheduled to execute locally or to execute remotely. Due to the mobile device’s limited computing resource and battery capacity, all computation tasks cannot be executed locally within a timely manner. Therefore, a part of these can be offloaded to n edge servers with relatively rich resources. These offloaded tasks are first stored in a dedicated executing queue and then are executed sequentially. The system time is logically divided to equal length time slots, and the duration of each time slot is Tslot (in seconds). For convenience, we denote the index sets of the time slots as τ∈T=0,1,…. The value of Tslot is usually decided by the channel coherence time. It means that the channel remains static within a time slot but varies among different time slots [26]. At the beginning of each time slot, the user makes an optimal offloading decision.Figure 2
Computation task offloading framework with multiple edge servers.The mobile device can be denoted by a tripleMD=fu,Nu,Pu, where fu denotes the mobile device’s CPU cycle frequency, Nu denotes the number of the mobile device’s processor cores, and Pu denotes the mobile device’s power. Especially, Pu can be further represented by a tuple Pu=Pue,Putx, where Pue and Putx denote the mobile device’s computation power (in Watt) and transmitting power, respectively. The mobile device has an execution queue with its size Que. If the queue is already full, the new arrival tasks will be dropped.The computation task model widely used in the existing literature [34, 35] is adopted in this paper. According to it, a computation task can be abstracted into a three-field notation t=W,Dtx,Drx, in which W denotes the task computation workload (in CPU cycles per bit), Dtx denotes the task input data size (in MB), and Drx denotes the task output data size. In addition, we assume that the computation tasks’ arrival process for the mobile device MD follows a Poisson distribution with a parameter λtask.The set ofn edge servers can be denoted by eNB=eNB1,…,eNBi,…,eNBn, where eNBi denotes the ith edge server. Each edge server eNBi has different configurations, such as the number of processor cores and the processor frequency. We use a two tuple eNBi=fc,i,Nc,i to represent the edge server eNBi, where fc,i denotes its processor frequency and Nc,i denotes the number of processor cores. Each edge server has an execution queue with the size Qci. When the edge server receives the offloaded tasks from the mobile device, it firstly stores the tasks in its execution queue and then executes them sequentially. Let μci denote the task processing rate of the ith edge server. Therefore, μci can be calculated by the following equation:(1)μci=fc,iW.


## 3.2. Security Overhead Model

The computation tasks offloaded to the edge servers are confronted with security threats. Fortunately, confidentiality service and integrity service can guard against these common security threats [29–33, 36, 37]. Confidentiality service can protect data from stealing by enciphering methods. Meanwhile, integrity service can ensure that data are not tampered. By flexibly selecting these two security services, an integrated security protection is formed to protect against a diversity of security threats. Based on these above services, the offloading process of a task with security protection is shown in Figure 3. We can observe from Figure 3 that confidential service (denoted as E) is first employed to encrypt the offloaded task. The security levels and processing rates of cryptographic algorithms for confidential service [20, 30, 36, 37] can be found in Table 2. Then, to protect the offloaded computation task from alteration attacks, integrity service (denoted as H) is successively employed to implement a hash algorithm to it. The security levels and processing rates of hash functions for integrity service [20, 30, 36, 37] can be seen in Table 3. After the encrypted task is delivered to the ith edge server, it is firstly decrypted (denoted as DE) and its integrity is verified (denoted as IV), and then it is executed, and finally the computation result is sent to the mobile device.Figure 3
The task execution process with security services.Table 2
The encryption algorithms for confidential service.
Encryption algorithms
sl
c
f: security level
V
sl
cf: processing rate (Mb/s)
IDEA
1.0
11.76
DES
0.85
13.83
Blowfish
0.56
20.87
AES
0.53
22.03
RC4
0.32
37.17Table 3
The hash functions for integrity service.
Hash functions
sl
cf: security level
V
sl
cf: processing rate (Mb/s)
TIGER
1.0
75.76
RipeMD160
0.75
101.01
SHA-1
0.69
109.89
RipeMD128
0.63
119.05
MD5
0.44
172.41Each offloaded computation task may require the confidentiality service and integrity service with various security levels. For the sake of simplicity, letcf and ig represent the confidentiality service and integrity service, respectively. Let slc,i=slc,icfτ,slc,iigτ to be the set of security levels of task t offloaded to the ith edge server, where slc,icfτ represents the security level of confidentiality service and slc,iigτ represents the security level of integrity service. Security service incurs sometime overhead. According to [20], the total encryption overheads of task t offloaded to the ith edge server can be calculated by the following equation:(2)Tc,iE=∑type∈cf,igk⋅Dtxfu⋅Nu⋅Vslc,itype,where k=2.2 and Vslc,icf and Vslc,iig denote the processing rates of the security levels slcf and slig, respectively. The total decryption overheads of the offloaded task t in the ith edge server can be computed by the following equation:(3)Tc,iDE=fu⋅Nu⋅TuEfc,i⋅Nc,i.


## 3.3. Workload Shaping with Security Guarantee

The security services incur not only time overhead but also security workload. The security workload of taskt offloaded to the ith edge server is incurred by encrypting it in mobile device and decrypting it in edge server. Based on the security time overhead of task t, the security workload which is incurred by encrypting task t in the mobile device can be computed by the following equation:(4)SWc,iEt=Tc,iE⋅fu⋅Nu.The security workload which is incurred by the decrypting taskt in the ith edge server can be computed by the following equation:(5)SWc,iDEDtx=Tc,iDE⋅fc,i⋅Nc,i.Based on the security services introduced above, we further quantitatively calculate the risk probability of taskt with different security levels. Without loss of generality, we assume that the distribution of risk probability follows a Poisson probability distribution for any given time interval. The risk probability of an offloaded computational task t is related to the set slc,i=slc,icfτ,slc,iigτ of security levels. Let Pslc,ik denote the risk probability of an offloaded task which employs either of the two security services. Pslc,ik can be denoted by the following equation [38, 39]:(6)Pslc,ik=1−exp−λk1−slk,k∈cf,ig.The risk probabilityPt of an offloaded task t which employs these two security services with different levels can be computed by the following equation:(7)Pt=1−∏k∈cf,ig1−Pslc,ik.Given the risk probability constraint of each offloaded computational task isPmax, the risk probability Pt must meet the following constraint:(8)Pack≤Pmax.In order to minimize the security workload while satisfying security requirement, it is a critical problem to select the levels of security services for each task. We formulate this problem as follows:(9)minimizeSWt=SWuEt+SWc,iDEtsubject to8.As shown in Tables2 and 3, the levels of each security service are discrete. For a task t, there are 5×5 types of security service composition. Hence, we can traverse these compositions to find the optimal service levels.


## 3.4. Communication Model

The computation tasks offloaded from the mobile device to edge servers are performed by wireless channels. Due to user’s mobility, the wireless channel gain state is dynamically changing at each time slot, which induces dynamic change of wireless channel transmission rate. LetRiτ denote the transmission rate of wireless channel (i.e., data transmission capacity) from the mobile device to the ith edge server eNBi in the τth time slot, which is given as follows:(10)Riτ=Bilog21+PuGiσ2,∀i∈N,where Bi is the wireless channel bandwidth of the ith edge server, Pu is the transmission power of the mobile device, σ2 is the Gaussian white noise power, and Gi denotes the wireless channel gain of eNBi.


## 3.5. Problem Statement

The computation offloading process can be formulated as an infinite Markov decision process. At the beginning of each time slot, an offloading decision is made based on the current system state, which consists of the number of arrival tasks, the remaining number of tasks in the user’s executing queue, the transmission rates between user andn edge servers, and the remaining number of tasks in n edge servers’ executing queues. The offloading decision mainly determines the number of tasks assigned to execute locally and the number of tasks offloaded to different edge servers. To protect the offloaded computation tasks from malicious attacks, security services need to be employed. Security services incur time overheads and security workload. Hence, the objective of this paper is to minimize the long-term cost while meeting the risk probability constraint.


## 4. Security-Aware Computation Offloading Problem Formulation

In this section, we first define the state and action spaces. Then, the system state transition and reward function are derived. Finally, we define the objective and constraints.


### 4.1. State Space

At theτth time slot, the system state for the security-aware computation offloading problem can be denoted by the following equation:(11)sτ=Qarrτ,Queτ,Rτ,Qceτ∈S,where Qarrτ∈0,1,… denotes the number of arrival tasks at the τth time slot. Queτ∈0,1,…,Queτ denotes the user’s executing queue’s state at the τth time slot. Rτ=R1τ,…,Riτ,…,RnτRiτ∈10,100∈ℝ is a vector of the transmission rate state between user and n edge servers at the τth time slot. Qceτ=Qc,1eτ,…,Qc,ieτ,…,Qc,neτQc,ieτ∈0,1,…,Qc,ieτ∈Q is the vector of the edge servers’ execution queue state at the τth time slot. Riτ and Qc,ieτ denote the ith edge server’s transmission rate and the number of the remaining tasks, respectively. Note that Queτ and Qc,ieτ denote the size of the mobile device and the ith edger server’s execution queue, respectively. The system state sτ can be observed at the beginning of the τth time slot. Queτ and Qceτ dynamically evolve according to the offloading policy, while Riτ can be dynamically calculated.


### 4.2. Action Space

The actionaτ at the τth time slot can be defined by the following equation:(12)aτ=aueτ,aceτ,slccfτ,slcigτ∈A,where aueτ is the number of tasks for executing locally. aceτ=ac,1eτ,…,ac,ieτ,…,ac,neτ is a vector of the number of tasks to be offloaded to n edge servers. ac,ieτ is the number of tasks to be offloaded to the ith edge servers. Therefore, the total number of tasks to be processed at the τth time slot is maτ=aueτ+∑i=1nac,ieτ. slccfτ=slc,1cfτ,…,slc,icfτ,…,slc,ncfτ is a vector of the confidentiality service’s level which is employed by tasks offloaded to n edge servers. slc,icfτ∈0,1.0,0.85,0.56,0.53,0.32 denotes the security level of confidentiality service employed by the tasks offloaded to the ith edge server. slcigτ=slc,1igτ,…,slc,iigτ,…,slc,nigτ is a vector of the integrity service’s level which employed by tasks offloaded to n edge servers. slc,iigτ∈0,1.0,0.75,0.69,0.63,0.44 denotes the security level of integrity service employed by the tasks offloaded to the ith edge server. Note that an action aτ must satisfy the constraint condition that the number of assigned tasks is equal to the current number of arrival tasks.


### 4.3. State Transition and Reward Function

Given the current statesτ=Quwτ,Queτ,Rτ,Qceτ∈S, after taking an action aτ∈A, the state is transferred to the next state s_τ=Quw_τ,Que_τ,R_τ,Qce_τ∈S, and the transition probability can be denoted as ps_τsτ,aτ. The immediate cost obtained by taking the action aτ∈A can be denoted as Cτ. The immediate cost Cτ is the weighted sum of immediate energy consumption Eτ, immediate processing delay Tτ, and immediate task’s dropping probability Dτ. Based on the above, the average long-term cost function can be denoted by equation (14):(13)Cτ=−φ1Eτ+φ2Tτ+φ3Dτ,(14)C¯=limτ⟶∞supEπ∑τ=0n−1Cτ,where φ1, φ2, and φ3 are the weights of immediate energy consumption, immediate processing delay, immediate task’s dropping probability, respectively. We further derive Eτ, Tτ, and Dτ as follows:(1)
Immediate Processing Delay. When action aτ is taken at state sτ, the immediate processing delay is induced by the waiting time at user and edge servers’ execution queue, the encryption and transmission time by user, the decryption time by edge servers, and the execution time by user and edge servers.According to Little’s law, the average waiting timetuw at user’s execution queue can be calculated by equation (15). Therefore, after taking an action aτ∈A, the total local waiting time ttuw and total local execution time ttue can be calculated by equations (16) and (17), respectively:(15)tuw=Queτfu⋅Nu,(16)ttuw=aueτ−dueτ∗tuw,(17)ttue=aueτ−dueτ⋅Wfu⋅Nu,wheredueτ denotes the number of dropping tasks at user’s execution queue. It can be calculated by equation (30).Whenac,ieτ tasks are scheduled to the τth edge server’s execution queue, tasks first need to employ security services to encrypted security-critical data. Let ttcs denote the total encryption time of aceτ tasks. Therefore, ttcs can be calculated by equation (18). And then, these encrypted tasks are transmitted to the edge servers by the wireless channel. Let ttctr denotes the total transmission time. Therefore, ttctr can be calculated by equation (19). Next, these encrypted tasks are received and stored in its execution queue by edge servers. Due to insufficient queue space, dc,ieτ tasks are dropped. The number dc,ieτ of dropping tasks can be calculated by equation (31). The remaining ac,ieτ−dc,ieτ tasks wait to be executed in execution queue. The total waiting time ttcw can be calculated by equation (20). At last, they are further decrypted and executed. Let ttcds and ttce denote the total encryption time and the total execution time, respectively. Therefore, they can be calculated by equations (22) and (23):(18)ttcs=∑i=1nac,ieτ⋅Tc,iE,(19)ttctr=∑i=1nac,ieτ⋅DtxRiτ,(20)ttcw=∑i=1nac,ieτ−dc,ieτ⋅Queτfc,i⋅Nc,i/W,(21)ttc,ids=ac,ieτ−dc,ieτ⋅Tc,iDE⋅fu⋅Nufc,i⋅Nc,i,(22)ttcds=∑i=1nttc,ids,(23)ttce=∑i=1nac,ieτ−dc,ieτ⋅Wfc,i⋅Nc,i.The total immediate processing timeTτ at the kth time slot is defined as an average value of the sum of the encryption time, the transmission time, the waiting time, the execution time, and the decryption time. Based on the aforementioned values, Tτ can be calculated by the following equation:(24)Tτ=ttuw+ttue+ttcs+ttctr+ttcw+ttcds+ttcemaτ.(2)
Immediate Energy Consumption. When action aτ is taken at state sτ, the immediate mobile device’s energy is consumed by executing tasks locally, encrypting offloaded tasks and transmitting the offloaded tasks to edge server. We define the immediate energy consumption as the average energy consumed by performing action aτ.When the user decides to executeaueτ tasks locally, the local execution energy consumption Eueτ can be calculated by the following equation:(25)Eueτ=Pue⋅ttue.Whenmaτ number of tasks are to be offloaded to n edge servers, the encryption energy consumption Ecsτ and the transmission energy consumption Ectrτ can be calculated by equations (26) and (27), respectively:(26)Ecsτ=Pue⋅ttus,(27)Ectrτ=Putx⋅ttctr.Therefore, the immediate energy consumptionEτ can be denoted by the following equation:(28)Eτ=Eueτ+Eusτ+Ectrτmaτ.(3)
Immediate Task Dropping Number. The arriving tasks will be dropped if the mobile device and edge servers’ execution queue are full or have insufficient space at the τth time slot. Let Dτ denote the number of dropped tasks. It can be calculated by the following equation:(29)Dτ=dueτ+∑i=1ndc,ieτ,(30)dueτ=max0,aueτ−Quavaτ,(31)dc,ieτ=max0,ac,ieτ−Qc,iavaτ,whereQuavaτ=Queτ−Queτ+euτ denotes the residual available space of the mobile device’s execution queue. euτ=minTslot−ttcs−ttctr⋅fu⋅Nu/W,Queτ denotes the number of tasks actually executed for the mobile device’s execution queue at the τth time slot. Qc,iavaτ=Qc,ieτ−Qc,ieτ+ec,iτ denotes the residual available space of the ith edge server. ec,iτ=minTslot−ttc,ids⋅fc,i⋅Nc,i/W,Qc,ieτ denotes the number of tasks actually executed at the ith edge server at the τth time slot.


### 4.4. Problem Formulation

The objective of this paper is to find an offloading policyπ, which minimizes the average long-term cost over the infinite time horizon while meeting the risk probability constraint. Thus, the problem can be formally formulated as(32)minimizeC¯,(33)subject toPτ≤Pmax,(34)ttcs+tctr≤Tslot,where equation (32) is the objective of this paper, equation (33) is the risk probability constraint, and equation (34) indicates that the task transmission and security service should be conducted in each time slot.Lacking of a prior knowledge of the task arrival and channel state, it is very difficult to solve this optimization problem by traditional methods. Fortunately, without the prior knowledge of network state transition, deep Q-network (DQN) can solve this kind of stochastic optimization problem. In the next section, a SCACO strategy based on deep Q-network is introduced to solve our security-aware computation offloading problem.


## 4.1. State Space

At theτth time slot, the system state for the security-aware computation offloading problem can be denoted by the following equation:(11)sτ=Qarrτ,Queτ,Rτ,Qceτ∈S,where Qarrτ∈0,1,… denotes the number of arrival tasks at the τth time slot. Queτ∈0,1,…,Queτ denotes the user’s executing queue’s state at the τth time slot. Rτ=R1τ,…,Riτ,…,RnτRiτ∈10,100∈ℝ is a vector of the transmission rate state between user and n edge servers at the τth time slot. Qceτ=Qc,1eτ,…,Qc,ieτ,…,Qc,neτQc,ieτ∈0,1,…,Qc,ieτ∈Q is the vector of the edge servers’ execution queue state at the τth time slot. Riτ and Qc,ieτ denote the ith edge server’s transmission rate and the number of the remaining tasks, respectively. Note that Queτ and Qc,ieτ denote the size of the mobile device and the ith edger server’s execution queue, respectively. The system state sτ can be observed at the beginning of the τth time slot. Queτ and Qceτ dynamically evolve according to the offloading policy, while Riτ can be dynamically calculated.


## 4.2. Action Space

The actionaτ at the τth time slot can be defined by the following equation:(12)aτ=aueτ,aceτ,slccfτ,slcigτ∈A,where aueτ is the number of tasks for executing locally. aceτ=ac,1eτ,…,ac,ieτ,…,ac,neτ is a vector of the number of tasks to be offloaded to n edge servers. ac,ieτ is the number of tasks to be offloaded to the ith edge servers. Therefore, the total number of tasks to be processed at the τth time slot is maτ=aueτ+∑i=1nac,ieτ. slccfτ=slc,1cfτ,…,slc,icfτ,…,slc,ncfτ is a vector of the confidentiality service’s level which is employed by tasks offloaded to n edge servers. slc,icfτ∈0,1.0,0.85,0.56,0.53,0.32 denotes the security level of confidentiality service employed by the tasks offloaded to the ith edge server. slcigτ=slc,1igτ,…,slc,iigτ,…,slc,nigτ is a vector of the integrity service’s level which employed by tasks offloaded to n edge servers. slc,iigτ∈0,1.0,0.75,0.69,0.63,0.44 denotes the security level of integrity service employed by the tasks offloaded to the ith edge server. Note that an action aτ must satisfy the constraint condition that the number of assigned tasks is equal to the current number of arrival tasks.


## 4.3. State Transition and Reward Function

Given the current statesτ=Quwτ,Queτ,Rτ,Qceτ∈S, after taking an action aτ∈A, the state is transferred to the next state s_τ=Quw_τ,Que_τ,R_τ,Qce_τ∈S, and the transition probability can be denoted as ps_τsτ,aτ. The immediate cost obtained by taking the action aτ∈A can be denoted as Cτ. The immediate cost Cτ is the weighted sum of immediate energy consumption Eτ, immediate processing delay Tτ, and immediate task’s dropping probability Dτ. Based on the above, the average long-term cost function can be denoted by equation (14):(13)Cτ=−φ1Eτ+φ2Tτ+φ3Dτ,(14)C¯=limτ⟶∞supEπ∑τ=0n−1Cτ,where φ1, φ2, and φ3 are the weights of immediate energy consumption, immediate processing delay, immediate task’s dropping probability, respectively. We further derive Eτ, Tτ, and Dτ as follows:(1)
Immediate Processing Delay. When action aτ is taken at state sτ, the immediate processing delay is induced by the waiting time at user and edge servers’ execution queue, the encryption and transmission time by user, the decryption time by edge servers, and the execution time by user and edge servers.According to Little’s law, the average waiting timetuw at user’s execution queue can be calculated by equation (15). Therefore, after taking an action aτ∈A, the total local waiting time ttuw and total local execution time ttue can be calculated by equations (16) and (17), respectively:(15)tuw=Queτfu⋅Nu,(16)ttuw=aueτ−dueτ∗tuw,(17)ttue=aueτ−dueτ⋅Wfu⋅Nu,wheredueτ denotes the number of dropping tasks at user’s execution queue. It can be calculated by equation (30).Whenac,ieτ tasks are scheduled to the τth edge server’s execution queue, tasks first need to employ security services to encrypted security-critical data. Let ttcs denote the total encryption time of aceτ tasks. Therefore, ttcs can be calculated by equation (18). And then, these encrypted tasks are transmitted to the edge servers by the wireless channel. Let ttctr denotes the total transmission time. Therefore, ttctr can be calculated by equation (19). Next, these encrypted tasks are received and stored in its execution queue by edge servers. Due to insufficient queue space, dc,ieτ tasks are dropped. The number dc,ieτ of dropping tasks can be calculated by equation (31). The remaining ac,ieτ−dc,ieτ tasks wait to be executed in execution queue. The total waiting time ttcw can be calculated by equation (20). At last, they are further decrypted and executed. Let ttcds and ttce denote the total encryption time and the total execution time, respectively. Therefore, they can be calculated by equations (22) and (23):(18)ttcs=∑i=1nac,ieτ⋅Tc,iE,(19)ttctr=∑i=1nac,ieτ⋅DtxRiτ,(20)ttcw=∑i=1nac,ieτ−dc,ieτ⋅Queτfc,i⋅Nc,i/W,(21)ttc,ids=ac,ieτ−dc,ieτ⋅Tc,iDE⋅fu⋅Nufc,i⋅Nc,i,(22)ttcds=∑i=1nttc,ids,(23)ttce=∑i=1nac,ieτ−dc,ieτ⋅Wfc,i⋅Nc,i.The total immediate processing timeTτ at the kth time slot is defined as an average value of the sum of the encryption time, the transmission time, the waiting time, the execution time, and the decryption time. Based on the aforementioned values, Tτ can be calculated by the following equation:(24)Tτ=ttuw+ttue+ttcs+ttctr+ttcw+ttcds+ttcemaτ.(2)
Immediate Energy Consumption. When action aτ is taken at state sτ, the immediate mobile device’s energy is consumed by executing tasks locally, encrypting offloaded tasks and transmitting the offloaded tasks to edge server. We define the immediate energy consumption as the average energy consumed by performing action aτ.When the user decides to executeaueτ tasks locally, the local execution energy consumption Eueτ can be calculated by the following equation:(25)Eueτ=Pue⋅ttue.Whenmaτ number of tasks are to be offloaded to n edge servers, the encryption energy consumption Ecsτ and the transmission energy consumption Ectrτ can be calculated by equations (26) and (27), respectively:(26)Ecsτ=Pue⋅ttus,(27)Ectrτ=Putx⋅ttctr.Therefore, the immediate energy consumptionEτ can be denoted by the following equation:(28)Eτ=Eueτ+Eusτ+Ectrτmaτ.(3)
Immediate Task Dropping Number. The arriving tasks will be dropped if the mobile device and edge servers’ execution queue are full or have insufficient space at the τth time slot. Let Dτ denote the number of dropped tasks. It can be calculated by the following equation:(29)Dτ=dueτ+∑i=1ndc,ieτ,(30)dueτ=max0,aueτ−Quavaτ,(31)dc,ieτ=max0,ac,ieτ−Qc,iavaτ,whereQuavaτ=Queτ−Queτ+euτ denotes the residual available space of the mobile device’s execution queue. euτ=minTslot−ttcs−ttctr⋅fu⋅Nu/W,Queτ denotes the number of tasks actually executed for the mobile device’s execution queue at the τth time slot. Qc,iavaτ=Qc,ieτ−Qc,ieτ+ec,iτ denotes the residual available space of the ith edge server. ec,iτ=minTslot−ttc,ids⋅fc,i⋅Nc,i/W,Qc,ieτ denotes the number of tasks actually executed at the ith edge server at the τth time slot.


## 4.4. Problem Formulation

The objective of this paper is to find an offloading policyπ, which minimizes the average long-term cost over the infinite time horizon while meeting the risk probability constraint. Thus, the problem can be formally formulated as(32)minimizeC¯,(33)subject toPτ≤Pmax,(34)ttcs+tctr≤Tslot,where equation (32) is the objective of this paper, equation (33) is the risk probability constraint, and equation (34) indicates that the task transmission and security service should be conducted in each time slot.Lacking of a prior knowledge of the task arrival and channel state, it is very difficult to solve this optimization problem by traditional methods. Fortunately, without the prior knowledge of network state transition, deep Q-network (DQN) can solve this kind of stochastic optimization problem. In the next section, a SCACO strategy based on deep Q-network is introduced to solve our security-aware computation offloading problem.


## 5. Algorithm Implementation

The formulated computation offloading optimization problem in Section4 is essentially an infinite-horizon Markov decision process with the discounted cost criterion. To solve the optimal computation offloading policy, we propose a SCACO scheme based on deep Q-network (DQN) [40]. In this section, we first introduce the traditional solutions for the Markov decision process and then introduce the SCACO strategy based on deep Q-network.


### 5.1. Optimal MDP Solution

To solve the optimal policyπ∗, an optimal mapping from any state sτ to the optimal action aτ need to be achieved. Since the optimal state-value function Vπ∗s can be achieved by solving Bellman’s optimality equation, Bellman’s optimality equation of the formulated computation offloading problem is defined by the following equation:(35)Vπ∗s=maxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.According to Bellman’s optimality equation, the optimal policy of statesτ can be obtained by the following equation:(36)π∗s=argmaxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.The traditional solutions to solve Bellman’s optimality equation are based on the value or the policy iteration [41]. These two solutions usually need complete knowledge of the system-state transition probabilities. However, these knowledges are difficult to obtain in advance in the dynamic system. Moreover, the network state-space with even a reasonable number of edge servers is extremely huge. Facing these problems, the two traditional solutions are inefficient. Thus, the DQN-based learning scheme which is a model-free reinforcement learning method is introduced to approach the optimal policy.


### 5.2. DQN-Based Offloading Decision Algorithm

The optimal action-value functionQ∗s,a which is on the right-hand side of equation (35) can be defined as follows:(37)Q∗s,a=maxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.To address these challenges mentioned in Section5.1, a model-free deep reinforcement learning algorithm called deep Q-network (DQN) is proposed. A deep neural network can be used to approach the optimal action-value function Q∗s,a without any information of dynamic network statistics. The DQN-based offloading decision algorithm can be illustrated in Figure 4.Figure 4
DQN-based offloading decision algorithm.At each time slotτ, the system state sτ=Qarrτ,Queτ,Rτ,Qceτ is first fed into the neural network. Based on the system state, the Q_values Qsτ,⋅ for all possible actions aτ∈A are obtained. Then, the action aτ corresponding to the state sτ can be selected according to the ε-greedy method, and the immediate cost can be calculated by using equation (13). Next, the resulting system state s_τ+1 at the next time slot τ+1 can be observed. And last, the transition experience trτ=sτ,aτ,Csτ,aτ,s_τ+1 can be obtained and stored into replay memory of a finite size. The replay memory can be denoted by Ωτ=mτ−U+1,mτ−U+2,…,mτ at τth time slot, where U is the size of the replay memory. To address the instability of the Q-network due to the use of nonlinear approximation function, the experience replay technique is adopted as the training method. At the end of each episode k, a minibatch Ω˜k⊂Ωk of transitions from the replay memory are randomly chosen to train the Q-network in the direction of minimizing a sequence of the loss functions, where the loss function at time step k is defined as(38)Lθk=ECsτ,aτ+γmaxaτ∈AQsτ+1,aτ+1;θk+1−Qsτ,aτ;θk2.In other words, given a transitiontrτ=sτ,aτ,Csτ,aτ,s_τ+1, the weight θ of the Q-network are updated in a way that the squared error loss is minimized between the current predicted Q value Qsτ,aτ;θk and the target Q value Csτ,aτ+γmaxaτ∈AQs_τ+1,aτ+;θk+1.The DQN-based offloading decision algorithm is described in Algorithm1 in detail.Algorithm 1: DQN-based computation task offloading algorithm.
Begin
(1)
Initialize the replay memoryΩ with a size of U, the minibatch Ω˜⊂Ω with a size of U˜.
(2)
for k=1,2,… do
(3)
At beginning of theτth time slot, observe the system state sτ∈S
(4)
Select an actionaτ∈A randomly with probability ε or yk=argmaxaτ∈AQsτ,aτ;θτ with probability 1 − ε
(5)
Offload the tasks according to actionaτ and observe the cost Csτ,aτ and the new system state s_τ∈S at the next time slot τ+1.
(6)
Store the transition experiencesτ,aτ,Csτ,aτ,s_τ+1 into replay memory Ω
(7)
Randomly sample a minibatch of transition experienceΩ˜k from replay memory Ωk
(8)
Train theQ-network based on the loss function of the selected transition experiences
(9)
Calculate the loss value between the current predictedQ value Qsτ,aτ;θk and the target Q value Csτ,aτ+γmaxaτ∈AQs_τ+1,aτ+1
(10)
end for
(11)
Record the set of optimal weightsθ
End


## 5.1. Optimal MDP Solution

To solve the optimal policyπ∗, an optimal mapping from any state sτ to the optimal action aτ need to be achieved. Since the optimal state-value function Vπ∗s can be achieved by solving Bellman’s optimality equation, Bellman’s optimality equation of the formulated computation offloading problem is defined by the following equation:(35)Vπ∗s=maxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.According to Bellman’s optimality equation, the optimal policy of statesτ can be obtained by the following equation:(36)π∗s=argmaxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.The traditional solutions to solve Bellman’s optimality equation are based on the value or the policy iteration [41]. These two solutions usually need complete knowledge of the system-state transition probabilities. However, these knowledges are difficult to obtain in advance in the dynamic system. Moreover, the network state-space with even a reasonable number of edge servers is extremely huge. Facing these problems, the two traditional solutions are inefficient. Thus, the DQN-based learning scheme which is a model-free reinforcement learning method is introduced to approach the optimal policy.


## 5.2. DQN-Based Offloading Decision Algorithm

The optimal action-value functionQ∗s,a which is on the right-hand side of equation (35) can be defined as follows:(37)Q∗s,a=maxa∈ACs,a+γ∑s_∈Sps_s,aVπ∗s_.To address these challenges mentioned in Section5.1, a model-free deep reinforcement learning algorithm called deep Q-network (DQN) is proposed. A deep neural network can be used to approach the optimal action-value function Q∗s,a without any information of dynamic network statistics. The DQN-based offloading decision algorithm can be illustrated in Figure 4.Figure 4
DQN-based offloading decision algorithm.At each time slotτ, the system state sτ=Qarrτ,Queτ,Rτ,Qceτ is first fed into the neural network. Based on the system state, the Q_values Qsτ,⋅ for all possible actions aτ∈A are obtained. Then, the action aτ corresponding to the state sτ can be selected according to the ε-greedy method, and the immediate cost can be calculated by using equation (13). Next, the resulting system state s_τ+1 at the next time slot τ+1 can be observed. And last, the transition experience trτ=sτ,aτ,Csτ,aτ,s_τ+1 can be obtained and stored into replay memory of a finite size. The replay memory can be denoted by Ωτ=mτ−U+1,mτ−U+2,…,mτ at τth time slot, where U is the size of the replay memory. To address the instability of the Q-network due to the use of nonlinear approximation function, the experience replay technique is adopted as the training method. At the end of each episode k, a minibatch Ω˜k⊂Ωk of transitions from the replay memory are randomly chosen to train the Q-network in the direction of minimizing a sequence of the loss functions, where the loss function at time step k is defined as(38)Lθk=ECsτ,aτ+γmaxaτ∈AQsτ+1,aτ+1;θk+1−Qsτ,aτ;θk2.In other words, given a transitiontrτ=sτ,aτ,Csτ,aτ,s_τ+1, the weight θ of the Q-network are updated in a way that the squared error loss is minimized between the current predicted Q value Qsτ,aτ;θk and the target Q value Csτ,aτ+γmaxaτ∈AQs_τ+1,aτ+;θk+1.The DQN-based offloading decision algorithm is described in Algorithm1 in detail.Algorithm 1: DQN-based computation task offloading algorithm.
Begin
(1)
Initialize the replay memoryΩ with a size of U, the minibatch Ω˜⊂Ω with a size of U˜.
(2)
for k=1,2,… do
(3)
At beginning of theτth time slot, observe the system state sτ∈S
(4)
Select an actionaτ∈A randomly with probability ε or yk=argmaxaτ∈AQsτ,aτ;θτ with probability 1 − ε
(5)
Offload the tasks according to actionaτ and observe the cost Csτ,aτ and the new system state s_τ∈S at the next time slot τ+1.
(6)
Store the transition experiencesτ,aτ,Csτ,aτ,s_τ+1 into replay memory Ω
(7)
Randomly sample a minibatch of transition experienceΩ˜k from replay memory Ωk
(8)
Train theQ-network based on the loss function of the selected transition experiences
(9)
Calculate the loss value between the current predictedQ value Qsτ,aτ;θk and the target Q value Csτ,aτ+γmaxaτ∈AQs_τ+1,aτ+1
(10)
end for
(11)
Record the set of optimal weightsθ
End


## 6. Experiments

In this section, we perform simulation experiments to demonstrate the effectiveness of our proposed SCACO strategy. First, we present our experiment parameters setup. Then, we analyze the performance of our proposed strategy with the varying of different parameters.


### 6.1. Experiment Parameters

In this section, to evaluate the performance of the proposed SCACO strategy, we implement and simulate our strategy on Python 3.6 using a Dell R530 server configured with one CPU (2.2 GHz 8 cores). We set the experimental parameters referring to the literatures [2, 25, 42, 43]. Major simulation parameters are described in detail as follows.In this paper, we mainly consider the scenario where a mobile device safely and efficiently offloads computation tasks ton edge servers. Initially, the mobile device’s CPU frequency fu is set to fu=2.5GHz, and the number Nu of its processor cores is set to Nu=4. The mobile device’s computation power Pe (in Watt), transmitting power Ptx, and receiving power Prx are set to Pe=5W, Ptx=0.25W, and Prx=0.1W, respectively. The mobile device execution queue’s size is set to Que=6. We assume that there are 2 edge servers. The CPU frequencies of these two edge servers are set fc1=fc2=2.5GHz. The processor cores of them are Nc1=Nc2=6. The execution queues’ size of them is set to Qc1=Qc2=10. Moreover, the risk coefficients of confidentiality service for these two edge servers are set λcf1=2.4 and λcf2=2.7, respectively. The risk coefficients of integrity service for these two edge servers are λig1=1.5 and λig2=1.8, respectively.For the communication model, the transmission power of the edge server isPeNB=40W, the maximum bandwidth is Bi=100MHz, the wireless channel gain is Gi=0.3,4.5, and the Gaussian white noise power is σ2=−174dbm/Hz.For the computation task model, we assume that the tasks’ arrival process follows a Poisson distribution with a parameterλtask=10, which is considered as the estimated value of the average number of arrived tasks during a period Tslot=1S. The task input is set to Dtx=0.1MB. The task’s workload is W=2.5G Hz⋅S. In addition, the maximum risk rate for task execution is set to Pmax=0.4.The weights of energy consumption, processing delay, and task’s dropping probability areφ1=8, φ2=0.2, and φ3=3, respectively.In the DQN-based learning algorithm, the discount factor isγ=0.9, the replay memory is assumed to have a capacity of U=5000, and the size of minibatch is set to batchsize=32. We implement the DQN learning algorithm based on the TensorFlow APIs and choose a gradient descent optimization algorithm called RMSProp as the optimization method to train the Q-network. During the training, we select the action based on ε-greedy with probability ε=0.1.


### 6.2. Performance Analysis

To demonstrate the effectiveness of our proposed SCACO scheme, the following peer schemes are conducted on four performance metrics, such as total cost, total energy, total delay, and total number of dropping tasks:(i)
Local. At each time slot, the arriving tasks are processed locally so that the risk probability of task execution is 0.(ii)
Max_Level. This scheme employs various security services and sets the security level of all security services to the highest level 1.(iii)
SCACO. This abbreviation stands for the security and cost-aware computation offloading scheme, the objective of which is to minimize the long-term cost under the risk probability constraints.


#### 6.2.1. Performance Impact of Risk Rate

Figure5 illustrates the performance of three schemes as the risk rate varying from 0.1 to 1.0. Figure 5(a) presents the influence of different risk rates on the long-term cost. As observed from Figure 5(a), the long-term cost obtained by SCACO decreases gradually with increasing risk probability, while the curves for the long-term cost of Local and Max_Level are flat. This is because the security levels of security services gradually decrease with increasing risk probability. With a lower security level, both the task processing time and the mobile device’s energy consumption will be less. Consequently, the long-term cost will be reduced as well. The long-term costs of Local and Max_Level are independent of the risk rate so that their long-term costs are constant and the curves are flat. Moreover, the long-term cost obtained by the SCACO scheme is lower than those of Local and Max_Level schemes. The reason is that a lower security level can meet a higher risk rate constraint. The lower the security service level, the less the task processing time and mobile device’s energy consumption, and thereby the less the long-term cost.Figure 5
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different risk rates.
(a)
(b)
(c)
(d)Figures5(b)–5(d) show the energy consumption, delay, and number of dropping tasks for three schemes when the risk rate increases. As shown in Figures 5(b)–5(d), the total energy consumption, the delay, and the number of dropping tasks obtained by SCACO decrease gradually with increasing risk rate, while these values of Local and Max_Level are constant. In addition, we can observe from Figures 5(c) and 5(d) that the energy consumption and the number of dropping tasks obtained by SCACO are lower than those by Local and Max_Level. The reason is the same to the long-term cost above. We can observe from Figure 5(b) that the delay of Max_Level is the maximum and that of Local is minimium. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


#### 6.2.2. Performance Impact of Computing Capacity of Edge Server

The computing capacity of the edge server is mainly relative to the number of CPU cores. To investigate the impact of computing capacity of the edge server, we vary the CPU cores from 4 to 8 with the increment of 1. Figure6(a) shows that the long-term costs obtained by SCACO and Max_Level decrease gradually with increasing CPU cores. This is due to the fact that the more the computing resource, the shorter the tasks’ processing delay, which leads to lower long-term cost. However, when the number of CPU cores is equal to or larger than 8, the long-term costs of SCACO and Max_Level become stable and no longer increase. The main reason is that the arriving tasks can be processed timely under this computing capacity. Specially, the long-term cost of SCACO is lower than that of Max_Level. That is because that the security level of SCACO is lower than that of Max_Level. Moreover, the curve of Local is flat. That is because Local executes all tasks locally and it is independent of edge server’s computing capacity.Figure 6
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different CPU cores.
(a)
(b)
(c)
(d)Figures6(b)–6(d) show that the energy consumptions, delay, and number of dropping tasks for SCACO and Max_Level decrease when the number of CPU cores increases, while these values obtained by the Local scheme are constant. The reason is the same to the long-term cost above. Moreover, as shown in Figures 6(c) and 6(d), the energy consumption and number of dropping tasks for SCACO are lower than those of Local. It is due to the fact that SCACO employs a lower security level than Max_Level while meeting the risk rate. We can further observe from Figure 6(b) that the delay of Max_Level is the maximum and that of Local is minimum. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than that of the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


#### 6.2.3. Performance Impact of Task Workload

To examine the influence of different task workloads on the long-term cost, we vary the value of task workload from 1.5 to 3.5 with the increment of 0.5. Figure7 illustrates the long-term cost of three schemes. As shown in Figure 7(a), the long-term costs of three schemes increase with increasing task workload. The reason is that the larger the task workload, the larger the processing time and energy consumption, and thereby the larger the long-term cost. Moreover, SCACO shows a lower cost than Local and Max_Level schemes. That is because the optimization objective of SCACO is to minimize the long-term cost while satisfying the risk rate constraints.Figure 7
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different workloads.
(a)
(b)
(c)
(d)Figures7(b)–7(d) show that the energy consumptions, delay, and number of dropping tasks for three schemes increase gradually with increasing task workload. Especially, as shown in Figures 7(c) and 7(d), the energy consumption and number of dropping tasks obtained by SCACO are lower than those of Local and Max_Level. The reason is the same to the long-term cost above. However, we can further observe from Figure 7(b) that the delay obtained by SCAC is between Local and Max_Level. The reason is the same as the previous section.


#### 6.2.4. Performance Impact of Task Data Size

Figure8 illustrates the impact of task data size on the performance. We discuss about the performance of three schemes when the task data size is varied from 0.1 to 0.5 with the increment of 0.1. As observed from Figure 8(a), the long-term costs of SCACO and Max_Level schemes increase gradually with increasing task data size. It is because that the larger the task data size, the longer the tasks’ processing delay and the higher the mobile device’s energy consumption, which leads to higher long-term cost. Moreover, the long-term cost of SCACO is lower than that of Max_Level. The reason is that comparing with Max_Level scheme, a lower security level is selected by SCACO while satisfying the risk constraint. Finally, the curve of the Local scheme is flat. It is due to that the Local scheme executes all tasks locally, and it is independent of task data size.Figure 8
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different data sizes.
(a)
(b)
(c)
(d)Figures8(b)–8(d) show that the energy consumption, delay, and number of dropping tasks for SCACO and Max_Level schemes increase gradually with increasing task data size, while these curves of Local scheme are flat. In addition, the energy consumption and number of dropping tasks for SCACO are lower than these of Local and Max_Level, while the delay of SCACO is between Local and Max_Level.


#### 6.2.5. Performance Impact of Task Arrival Rate

To investigate the impact of the task arrival rate, the experiments are conducted with the task arrival rate varying from 10 to 18. Figure9(a) shows the long-term cost which is obtained by three schemes. We observe from Figure 9(a) that the long-term costs of three schemes increase with increasing task arrival rate. That is because over the increase of the task arrival rate, a higher number of tasks need to be processed at each time slot, thereby incurring a higher long-term cost. Moreover, the SCACO shows a lower cost than Local and Max_Level schemes. That is because the main objective of the proposed schemes is to minimize the obtained average long-term cost while satisfying the risk rate constraints.Figure 9
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different task arrival rates.
(a)
(b)
(c)
(d)Figure9(b) shows that the delay of SCACO and Max_Level schemes gradually increases with increasing task arrival rate, while the delay of Local is constant. That is because the higher the task arrival rate, the more the tasks stored at the user’s and edge servers’ execution queue, and thereby the longer the waiting time. In addition, the delay of SCACO is between Local and Max_Level.As shown in Figure9(c), we observe that the mobile device’s energy consumption of SCACO and Max_Level gradually increases with the increase of the task arrival rate, while the energy consumption of Local is constant. With the increase of the task arrival rate, there are much more tasks which are offloaded to execute remotely. The more the tasks offloaded to the edge servers, the more the encryption energy consumption and transmission energy consumption consumed. Therefore, the mobile device consumes a higher amount of energy. Moreover, SCACO has a lower cost than Local and Max_Level schemes.Figure9(d) shows that the number of dropping tasks gradually increases with increasing task arrival rate. The main reason is that the higher the task arrival rate, the higher the number of arrival tasks at a time slot. However, with the limited-size executing queue of the mobile device and edge servers, newly arrived tasks will be dropped due to the lack of queue space. The more the tasks generated, the higher the number of the dropping task.


#### 6.2.6. Performance Impact of the Number of Edge Server

Figure10 illustrates the performance of three schemes when the number of edge servers varying from 2 to 4. As shown in Figure 10, the long-term costs obtained by SCACO and Max_Level schemes decrease with increasing number of the edge server, while the long-term cost of the Local scheme is constant. The reason is that there are more edge servers which are available for task offloading. The more the edge servers, the shorter the tasks’ processing delay, thereby incurring lower long-term cost. However, the performance of the Local scheme does not change with the variation of the number of edge servers since the Local scheme executes locally all tasks. Moreover, the long-term cost of SCACO is lower than that of Max_Level. That is because that SCACO employs a lower security level than Max_Level while meeting the risk rate.Figure 10
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different number of edge servers.
(a)
(b)
(c)
(d)Figures10(b)–10(d) exhibit the delay and number of dropping tasks obtained by SCACO and Max_Level schemes decrease with increasing number of edge servers, while the curves of the Local scheme are flat. The reason is that when more edge servers are available, the user offloads its tasks to more edge servers, and the processing delay of tasks and the number of tasks dropping are decreased. Figure 10(c) shows that the energy consumptions of SCACO and Max_Level increase gradually with increasing number of edge servers. This is due to that the more the edge servers, the lesser the number of tasks dropping, and the more the number of tasks which are executed locally or remotely, thereby the more the energy consumption.


#### 6.2.7. Performance Impact of SCACO

Figures11–16 show the learning curves and loss curves of the SCACO scheme over variations of risk rate, computing capacity, task workload, task data size, task arrival rate, and number of edge servers, respectively. For simplicity, we use Risk0.1, Risk0.3, Risk0.5, Risk0.8, and Risk1.0 to represent the long-term cost of SCACO for risk rates 0.1, 0.3, 0.5, 0.8, and 1.0, respectively. Nc4, Nc5, Nc6, Nc7, and Nc8 are used to represent the long-term costs of SCACO for the number of CPU cores 4, 5, 6, 7, and 8, respectively. W1.5, W2.0, W2.5, W3.0, and W3.5 are used to represent the long-term costs of SCACO for the workloads 1.5, 2.0, 2.5, 3.0, and 3.5, respectively. D_tx0.1, D_tx0.2, D_tx0.3, D_tx0.4, and D_tx0.5 are used to represent the long-term costs of SCACO for the task data sizes 0.1, 0.2, 0.3, 0.4, and 0.5, respectively. Lambda_tasks10, lambda_tasks12, lambda_tasks14, lambda_tasks16, and lambda_tasks18 are used to represent the long-term costs of SCACO for the task arriving rates 10, 12, 14, 16, and 18, respectively. Server_num2, server_num3, and server_num4 are used to represent the long-term costs of SCACO for the number of edge servers 2, 3, and 4, respectively. As shown in Figures 11(a)–16(a), the long-term cost obtained in an episode decreases gradually with increasing learning time (i.e., the number of episodes) from 1 to 1500. Moreover, Figures 11(b)–16(b) show that the loss value of cost decreases gradually with increasing number of episodes. When the episode number is higher than 1000, all learning curves become stable and no longer decrease. This result indicates that the proposed DQN-based learning algorithm converges after 1000 episodes. It means that the proposed DQN-based learning algorithm can learn an optimal strategy after 1000 episodes to minimize the long-term cost.Figure 11
Learning curves w.r.t. different risk rates.
(a)
(b)Figure 12
Learning curves w.r.t. different number of CPU cores.
(a)
(b)Figure 13
Learning curves w.r.t. different workloads.
(a)
(b)Figure 14
Learning curves w.r.t. different data sizes.
(a)
(b)Figure 15
Learning curves w.r.t. different task arrival rates.
(a)
(b)Figure 16
Learning curves w.r.t. different number of edge servers.
(a)
(b)


## 6.1. Experiment Parameters

In this section, to evaluate the performance of the proposed SCACO strategy, we implement and simulate our strategy on Python 3.6 using a Dell R530 server configured with one CPU (2.2 GHz 8 cores). We set the experimental parameters referring to the literatures [2, 25, 42, 43]. Major simulation parameters are described in detail as follows.In this paper, we mainly consider the scenario where a mobile device safely and efficiently offloads computation tasks ton edge servers. Initially, the mobile device’s CPU frequency fu is set to fu=2.5GHz, and the number Nu of its processor cores is set to Nu=4. The mobile device’s computation power Pe (in Watt), transmitting power Ptx, and receiving power Prx are set to Pe=5W, Ptx=0.25W, and Prx=0.1W, respectively. The mobile device execution queue’s size is set to Que=6. We assume that there are 2 edge servers. The CPU frequencies of these two edge servers are set fc1=fc2=2.5GHz. The processor cores of them are Nc1=Nc2=6. The execution queues’ size of them is set to Qc1=Qc2=10. Moreover, the risk coefficients of confidentiality service for these two edge servers are set λcf1=2.4 and λcf2=2.7, respectively. The risk coefficients of integrity service for these two edge servers are λig1=1.5 and λig2=1.8, respectively.For the communication model, the transmission power of the edge server isPeNB=40W, the maximum bandwidth is Bi=100MHz, the wireless channel gain is Gi=0.3,4.5, and the Gaussian white noise power is σ2=−174dbm/Hz.For the computation task model, we assume that the tasks’ arrival process follows a Poisson distribution with a parameterλtask=10, which is considered as the estimated value of the average number of arrived tasks during a period Tslot=1S. The task input is set to Dtx=0.1MB. The task’s workload is W=2.5G Hz⋅S. In addition, the maximum risk rate for task execution is set to Pmax=0.4.The weights of energy consumption, processing delay, and task’s dropping probability areφ1=8, φ2=0.2, and φ3=3, respectively.In the DQN-based learning algorithm, the discount factor isγ=0.9, the replay memory is assumed to have a capacity of U=5000, and the size of minibatch is set to batchsize=32. We implement the DQN learning algorithm based on the TensorFlow APIs and choose a gradient descent optimization algorithm called RMSProp as the optimization method to train the Q-network. During the training, we select the action based on ε-greedy with probability ε=0.1.


## 6.2. Performance Analysis

To demonstrate the effectiveness of our proposed SCACO scheme, the following peer schemes are conducted on four performance metrics, such as total cost, total energy, total delay, and total number of dropping tasks:(i)
Local. At each time slot, the arriving tasks are processed locally so that the risk probability of task execution is 0.(ii)
Max_Level. This scheme employs various security services and sets the security level of all security services to the highest level 1.(iii)
SCACO. This abbreviation stands for the security and cost-aware computation offloading scheme, the objective of which is to minimize the long-term cost under the risk probability constraints.


### 6.2.1. Performance Impact of Risk Rate

Figure5 illustrates the performance of three schemes as the risk rate varying from 0.1 to 1.0. Figure 5(a) presents the influence of different risk rates on the long-term cost. As observed from Figure 5(a), the long-term cost obtained by SCACO decreases gradually with increasing risk probability, while the curves for the long-term cost of Local and Max_Level are flat. This is because the security levels of security services gradually decrease with increasing risk probability. With a lower security level, both the task processing time and the mobile device’s energy consumption will be less. Consequently, the long-term cost will be reduced as well. The long-term costs of Local and Max_Level are independent of the risk rate so that their long-term costs are constant and the curves are flat. Moreover, the long-term cost obtained by the SCACO scheme is lower than those of Local and Max_Level schemes. The reason is that a lower security level can meet a higher risk rate constraint. The lower the security service level, the less the task processing time and mobile device’s energy consumption, and thereby the less the long-term cost.Figure 5
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different risk rates.
(a)
(b)
(c)
(d)Figures5(b)–5(d) show the energy consumption, delay, and number of dropping tasks for three schemes when the risk rate increases. As shown in Figures 5(b)–5(d), the total energy consumption, the delay, and the number of dropping tasks obtained by SCACO decrease gradually with increasing risk rate, while these values of Local and Max_Level are constant. In addition, we can observe from Figures 5(c) and 5(d) that the energy consumption and the number of dropping tasks obtained by SCACO are lower than those by Local and Max_Level. The reason is the same to the long-term cost above. We can observe from Figure 5(b) that the delay of Max_Level is the maximum and that of Local is minimium. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


### 6.2.2. Performance Impact of Computing Capacity of Edge Server

The computing capacity of the edge server is mainly relative to the number of CPU cores. To investigate the impact of computing capacity of the edge server, we vary the CPU cores from 4 to 8 with the increment of 1. Figure6(a) shows that the long-term costs obtained by SCACO and Max_Level decrease gradually with increasing CPU cores. This is due to the fact that the more the computing resource, the shorter the tasks’ processing delay, which leads to lower long-term cost. However, when the number of CPU cores is equal to or larger than 8, the long-term costs of SCACO and Max_Level become stable and no longer increase. The main reason is that the arriving tasks can be processed timely under this computing capacity. Specially, the long-term cost of SCACO is lower than that of Max_Level. That is because that the security level of SCACO is lower than that of Max_Level. Moreover, the curve of Local is flat. That is because Local executes all tasks locally and it is independent of edge server’s computing capacity.Figure 6
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different CPU cores.
(a)
(b)
(c)
(d)Figures6(b)–6(d) show that the energy consumptions, delay, and number of dropping tasks for SCACO and Max_Level decrease when the number of CPU cores increases, while these values obtained by the Local scheme are constant. The reason is the same to the long-term cost above. Moreover, as shown in Figures 6(c) and 6(d), the energy consumption and number of dropping tasks for SCACO are lower than those of Local. It is due to the fact that SCACO employs a lower security level than Max_Level while meeting the risk rate. We can further observe from Figure 6(b) that the delay of Max_Level is the maximum and that of Local is minimum. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than that of the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


### 6.2.3. Performance Impact of Task Workload

To examine the influence of different task workloads on the long-term cost, we vary the value of task workload from 1.5 to 3.5 with the increment of 0.5. Figure7 illustrates the long-term cost of three schemes. As shown in Figure 7(a), the long-term costs of three schemes increase with increasing task workload. The reason is that the larger the task workload, the larger the processing time and energy consumption, and thereby the larger the long-term cost. Moreover, SCACO shows a lower cost than Local and Max_Level schemes. That is because the optimization objective of SCACO is to minimize the long-term cost while satisfying the risk rate constraints.Figure 7
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different workloads.
(a)
(b)
(c)
(d)Figures7(b)–7(d) show that the energy consumptions, delay, and number of dropping tasks for three schemes increase gradually with increasing task workload. Especially, as shown in Figures 7(c) and 7(d), the energy consumption and number of dropping tasks obtained by SCACO are lower than those of Local and Max_Level. The reason is the same to the long-term cost above. However, we can further observe from Figure 7(b) that the delay obtained by SCAC is between Local and Max_Level. The reason is the same as the previous section.


### 6.2.4. Performance Impact of Task Data Size

Figure8 illustrates the impact of task data size on the performance. We discuss about the performance of three schemes when the task data size is varied from 0.1 to 0.5 with the increment of 0.1. As observed from Figure 8(a), the long-term costs of SCACO and Max_Level schemes increase gradually with increasing task data size. It is because that the larger the task data size, the longer the tasks’ processing delay and the higher the mobile device’s energy consumption, which leads to higher long-term cost. Moreover, the long-term cost of SCACO is lower than that of Max_Level. The reason is that comparing with Max_Level scheme, a lower security level is selected by SCACO while satisfying the risk constraint. Finally, the curve of the Local scheme is flat. It is due to that the Local scheme executes all tasks locally, and it is independent of task data size.Figure 8
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different data sizes.
(a)
(b)
(c)
(d)Figures8(b)–8(d) show that the energy consumption, delay, and number of dropping tasks for SCACO and Max_Level schemes increase gradually with increasing task data size, while these curves of Local scheme are flat. In addition, the energy consumption and number of dropping tasks for SCACO are lower than these of Local and Max_Level, while the delay of SCACO is between Local and Max_Level.


### 6.2.5. Performance Impact of Task Arrival Rate

To investigate the impact of the task arrival rate, the experiments are conducted with the task arrival rate varying from 10 to 18. Figure9(a) shows the long-term cost which is obtained by three schemes. We observe from Figure 9(a) that the long-term costs of three schemes increase with increasing task arrival rate. That is because over the increase of the task arrival rate, a higher number of tasks need to be processed at each time slot, thereby incurring a higher long-term cost. Moreover, the SCACO shows a lower cost than Local and Max_Level schemes. That is because the main objective of the proposed schemes is to minimize the obtained average long-term cost while satisfying the risk rate constraints.Figure 9
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different task arrival rates.
(a)
(b)
(c)
(d)Figure9(b) shows that the delay of SCACO and Max_Level schemes gradually increases with increasing task arrival rate, while the delay of Local is constant. That is because the higher the task arrival rate, the more the tasks stored at the user’s and edge servers’ execution queue, and thereby the longer the waiting time. In addition, the delay of SCACO is between Local and Max_Level.As shown in Figure9(c), we observe that the mobile device’s energy consumption of SCACO and Max_Level gradually increases with the increase of the task arrival rate, while the energy consumption of Local is constant. With the increase of the task arrival rate, there are much more tasks which are offloaded to execute remotely. The more the tasks offloaded to the edge servers, the more the encryption energy consumption and transmission energy consumption consumed. Therefore, the mobile device consumes a higher amount of energy. Moreover, SCACO has a lower cost than Local and Max_Level schemes.Figure9(d) shows that the number of dropping tasks gradually increases with increasing task arrival rate. The main reason is that the higher the task arrival rate, the higher the number of arrival tasks at a time slot. However, with the limited-size executing queue of the mobile device and edge servers, newly arrived tasks will be dropped due to the lack of queue space. The more the tasks generated, the higher the number of the dropping task.


### 6.2.6. Performance Impact of the Number of Edge Server

Figure10 illustrates the performance of three schemes when the number of edge servers varying from 2 to 4. As shown in Figure 10, the long-term costs obtained by SCACO and Max_Level schemes decrease with increasing number of the edge server, while the long-term cost of the Local scheme is constant. The reason is that there are more edge servers which are available for task offloading. The more the edge servers, the shorter the tasks’ processing delay, thereby incurring lower long-term cost. However, the performance of the Local scheme does not change with the variation of the number of edge servers since the Local scheme executes locally all tasks. Moreover, the long-term cost of SCACO is lower than that of Max_Level. That is because that SCACO employs a lower security level than Max_Level while meeting the risk rate.Figure 10
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different number of edge servers.
(a)
(b)
(c)
(d)Figures10(b)–10(d) exhibit the delay and number of dropping tasks obtained by SCACO and Max_Level schemes decrease with increasing number of edge servers, while the curves of the Local scheme are flat. The reason is that when more edge servers are available, the user offloads its tasks to more edge servers, and the processing delay of tasks and the number of tasks dropping are decreased. Figure 10(c) shows that the energy consumptions of SCACO and Max_Level increase gradually with increasing number of edge servers. This is due to that the more the edge servers, the lesser the number of tasks dropping, and the more the number of tasks which are executed locally or remotely, thereby the more the energy consumption.


### 6.2.7. Performance Impact of SCACO

Figures11–16 show the learning curves and loss curves of the SCACO scheme over variations of risk rate, computing capacity, task workload, task data size, task arrival rate, and number of edge servers, respectively. For simplicity, we use Risk0.1, Risk0.3, Risk0.5, Risk0.8, and Risk1.0 to represent the long-term cost of SCACO for risk rates 0.1, 0.3, 0.5, 0.8, and 1.0, respectively. Nc4, Nc5, Nc6, Nc7, and Nc8 are used to represent the long-term costs of SCACO for the number of CPU cores 4, 5, 6, 7, and 8, respectively. W1.5, W2.0, W2.5, W3.0, and W3.5 are used to represent the long-term costs of SCACO for the workloads 1.5, 2.0, 2.5, 3.0, and 3.5, respectively. D_tx0.1, D_tx0.2, D_tx0.3, D_tx0.4, and D_tx0.5 are used to represent the long-term costs of SCACO for the task data sizes 0.1, 0.2, 0.3, 0.4, and 0.5, respectively. Lambda_tasks10, lambda_tasks12, lambda_tasks14, lambda_tasks16, and lambda_tasks18 are used to represent the long-term costs of SCACO for the task arriving rates 10, 12, 14, 16, and 18, respectively. Server_num2, server_num3, and server_num4 are used to represent the long-term costs of SCACO for the number of edge servers 2, 3, and 4, respectively. As shown in Figures 11(a)–16(a), the long-term cost obtained in an episode decreases gradually with increasing learning time (i.e., the number of episodes) from 1 to 1500. Moreover, Figures 11(b)–16(b) show that the loss value of cost decreases gradually with increasing number of episodes. When the episode number is higher than 1000, all learning curves become stable and no longer decrease. This result indicates that the proposed DQN-based learning algorithm converges after 1000 episodes. It means that the proposed DQN-based learning algorithm can learn an optimal strategy after 1000 episodes to minimize the long-term cost.Figure 11
Learning curves w.r.t. different risk rates.
(a)
(b)Figure 12
Learning curves w.r.t. different number of CPU cores.
(a)
(b)Figure 13
Learning curves w.r.t. different workloads.
(a)
(b)Figure 14
Learning curves w.r.t. different data sizes.
(a)
(b)Figure 15
Learning curves w.r.t. different task arrival rates.
(a)
(b)Figure 16
Learning curves w.r.t. different number of edge servers.
(a)
(b)


## 6.2.1. Performance Impact of Risk Rate

Figure5 illustrates the performance of three schemes as the risk rate varying from 0.1 to 1.0. Figure 5(a) presents the influence of different risk rates on the long-term cost. As observed from Figure 5(a), the long-term cost obtained by SCACO decreases gradually with increasing risk probability, while the curves for the long-term cost of Local and Max_Level are flat. This is because the security levels of security services gradually decrease with increasing risk probability. With a lower security level, both the task processing time and the mobile device’s energy consumption will be less. Consequently, the long-term cost will be reduced as well. The long-term costs of Local and Max_Level are independent of the risk rate so that their long-term costs are constant and the curves are flat. Moreover, the long-term cost obtained by the SCACO scheme is lower than those of Local and Max_Level schemes. The reason is that a lower security level can meet a higher risk rate constraint. The lower the security service level, the less the task processing time and mobile device’s energy consumption, and thereby the less the long-term cost.Figure 5
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different risk rates.
(a)
(b)
(c)
(d)Figures5(b)–5(d) show the energy consumption, delay, and number of dropping tasks for three schemes when the risk rate increases. As shown in Figures 5(b)–5(d), the total energy consumption, the delay, and the number of dropping tasks obtained by SCACO decrease gradually with increasing risk rate, while these values of Local and Max_Level are constant. In addition, we can observe from Figures 5(c) and 5(d) that the energy consumption and the number of dropping tasks obtained by SCACO are lower than those by Local and Max_Level. The reason is the same to the long-term cost above. We can observe from Figure 5(b) that the delay of Max_Level is the maximum and that of Local is minimium. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


## 6.2.2. Performance Impact of Computing Capacity of Edge Server

The computing capacity of the edge server is mainly relative to the number of CPU cores. To investigate the impact of computing capacity of the edge server, we vary the CPU cores from 4 to 8 with the increment of 1. Figure6(a) shows that the long-term costs obtained by SCACO and Max_Level decrease gradually with increasing CPU cores. This is due to the fact that the more the computing resource, the shorter the tasks’ processing delay, which leads to lower long-term cost. However, when the number of CPU cores is equal to or larger than 8, the long-term costs of SCACO and Max_Level become stable and no longer increase. The main reason is that the arriving tasks can be processed timely under this computing capacity. Specially, the long-term cost of SCACO is lower than that of Max_Level. That is because that the security level of SCACO is lower than that of Max_Level. Moreover, the curve of Local is flat. That is because Local executes all tasks locally and it is independent of edge server’s computing capacity.Figure 6
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different CPU cores.
(a)
(b)
(c)
(d)Figures6(b)–6(d) show that the energy consumptions, delay, and number of dropping tasks for SCACO and Max_Level decrease when the number of CPU cores increases, while these values obtained by the Local scheme are constant. The reason is the same to the long-term cost above. Moreover, as shown in Figures 6(c) and 6(d), the energy consumption and number of dropping tasks for SCACO are lower than those of Local. It is due to the fact that SCACO employs a lower security level than Max_Level while meeting the risk rate. We can further observe from Figure 6(b) that the delay of Max_Level is the maximum and that of Local is minimum. The delay of SCACO is between Local and Max_Level. The reason is that the optimization of energy consumption is more important than that of the delay. To reduce the energy consumption, more tasks are offloaded to the edge servers, thereby incurring more longer waiting time.


## 6.2.3. Performance Impact of Task Workload

To examine the influence of different task workloads on the long-term cost, we vary the value of task workload from 1.5 to 3.5 with the increment of 0.5. Figure7 illustrates the long-term cost of three schemes. As shown in Figure 7(a), the long-term costs of three schemes increase with increasing task workload. The reason is that the larger the task workload, the larger the processing time and energy consumption, and thereby the larger the long-term cost. Moreover, SCACO shows a lower cost than Local and Max_Level schemes. That is because the optimization objective of SCACO is to minimize the long-term cost while satisfying the risk rate constraints.Figure 7
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different workloads.
(a)
(b)
(c)
(d)Figures7(b)–7(d) show that the energy consumptions, delay, and number of dropping tasks for three schemes increase gradually with increasing task workload. Especially, as shown in Figures 7(c) and 7(d), the energy consumption and number of dropping tasks obtained by SCACO are lower than those of Local and Max_Level. The reason is the same to the long-term cost above. However, we can further observe from Figure 7(b) that the delay obtained by SCAC is between Local and Max_Level. The reason is the same as the previous section.


## 6.2.4. Performance Impact of Task Data Size

Figure8 illustrates the impact of task data size on the performance. We discuss about the performance of three schemes when the task data size is varied from 0.1 to 0.5 with the increment of 0.1. As observed from Figure 8(a), the long-term costs of SCACO and Max_Level schemes increase gradually with increasing task data size. It is because that the larger the task data size, the longer the tasks’ processing delay and the higher the mobile device’s energy consumption, which leads to higher long-term cost. Moreover, the long-term cost of SCACO is lower than that of Max_Level. The reason is that comparing with Max_Level scheme, a lower security level is selected by SCACO while satisfying the risk constraint. Finally, the curve of the Local scheme is flat. It is due to that the Local scheme executes all tasks locally, and it is independent of task data size.Figure 8
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different data sizes.
(a)
(b)
(c)
(d)Figures8(b)–8(d) show that the energy consumption, delay, and number of dropping tasks for SCACO and Max_Level schemes increase gradually with increasing task data size, while these curves of Local scheme are flat. In addition, the energy consumption and number of dropping tasks for SCACO are lower than these of Local and Max_Level, while the delay of SCACO is between Local and Max_Level.


## 6.2.5. Performance Impact of Task Arrival Rate

To investigate the impact of the task arrival rate, the experiments are conducted with the task arrival rate varying from 10 to 18. Figure9(a) shows the long-term cost which is obtained by three schemes. We observe from Figure 9(a) that the long-term costs of three schemes increase with increasing task arrival rate. That is because over the increase of the task arrival rate, a higher number of tasks need to be processed at each time slot, thereby incurring a higher long-term cost. Moreover, the SCACO shows a lower cost than Local and Max_Level schemes. That is because the main objective of the proposed schemes is to minimize the obtained average long-term cost while satisfying the risk rate constraints.Figure 9
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different task arrival rates.
(a)
(b)
(c)
(d)Figure9(b) shows that the delay of SCACO and Max_Level schemes gradually increases with increasing task arrival rate, while the delay of Local is constant. That is because the higher the task arrival rate, the more the tasks stored at the user’s and edge servers’ execution queue, and thereby the longer the waiting time. In addition, the delay of SCACO is between Local and Max_Level.As shown in Figure9(c), we observe that the mobile device’s energy consumption of SCACO and Max_Level gradually increases with the increase of the task arrival rate, while the energy consumption of Local is constant. With the increase of the task arrival rate, there are much more tasks which are offloaded to execute remotely. The more the tasks offloaded to the edge servers, the more the encryption energy consumption and transmission energy consumption consumed. Therefore, the mobile device consumes a higher amount of energy. Moreover, SCACO has a lower cost than Local and Max_Level schemes.Figure9(d) shows that the number of dropping tasks gradually increases with increasing task arrival rate. The main reason is that the higher the task arrival rate, the higher the number of arrival tasks at a time slot. However, with the limited-size executing queue of the mobile device and edge servers, newly arrived tasks will be dropped due to the lack of queue space. The more the tasks generated, the higher the number of the dropping task.


## 6.2.6. Performance Impact of the Number of Edge Server

Figure10 illustrates the performance of three schemes when the number of edge servers varying from 2 to 4. As shown in Figure 10, the long-term costs obtained by SCACO and Max_Level schemes decrease with increasing number of the edge server, while the long-term cost of the Local scheme is constant. The reason is that there are more edge servers which are available for task offloading. The more the edge servers, the shorter the tasks’ processing delay, thereby incurring lower long-term cost. However, the performance of the Local scheme does not change with the variation of the number of edge servers since the Local scheme executes locally all tasks. Moreover, the long-term cost of SCACO is lower than that of Max_Level. That is because that SCACO employs a lower security level than Max_Level while meeting the risk rate.Figure 10
Reward (a), energy consumption (b), delay (c), and the number of dropping tasks (d) w.r.t. different number of edge servers.
(a)
(b)
(c)
(d)Figures10(b)–10(d) exhibit the delay and number of dropping tasks obtained by SCACO and Max_Level schemes decrease with increasing number of edge servers, while the curves of the Local scheme are flat. The reason is that when more edge servers are available, the user offloads its tasks to more edge servers, and the processing delay of tasks and the number of tasks dropping are decreased. Figure 10(c) shows that the energy consumptions of SCACO and Max_Level increase gradually with increasing number of edge servers. This is due to that the more the edge servers, the lesser the number of tasks dropping, and the more the number of tasks which are executed locally or remotely, thereby the more the energy consumption.


## 6.2.7. Performance Impact of SCACO

Figures11–16 show the learning curves and loss curves of the SCACO scheme over variations of risk rate, computing capacity, task workload, task data size, task arrival rate, and number of edge servers, respectively. For simplicity, we use Risk0.1, Risk0.3, Risk0.5, Risk0.8, and Risk1.0 to represent the long-term cost of SCACO for risk rates 0.1, 0.3, 0.5, 0.8, and 1.0, respectively. Nc4, Nc5, Nc6, Nc7, and Nc8 are used to represent the long-term costs of SCACO for the number of CPU cores 4, 5, 6, 7, and 8, respectively. W1.5, W2.0, W2.5, W3.0, and W3.5 are used to represent the long-term costs of SCACO for the workloads 1.5, 2.0, 2.5, 3.0, and 3.5, respectively. D_tx0.1, D_tx0.2, D_tx0.3, D_tx0.4, and D_tx0.5 are used to represent the long-term costs of SCACO for the task data sizes 0.1, 0.2, 0.3, 0.4, and 0.5, respectively. Lambda_tasks10, lambda_tasks12, lambda_tasks14, lambda_tasks16, and lambda_tasks18 are used to represent the long-term costs of SCACO for the task arriving rates 10, 12, 14, 16, and 18, respectively. Server_num2, server_num3, and server_num4 are used to represent the long-term costs of SCACO for the number of edge servers 2, 3, and 4, respectively. As shown in Figures 11(a)–16(a), the long-term cost obtained in an episode decreases gradually with increasing learning time (i.e., the number of episodes) from 1 to 1500. Moreover, Figures 11(b)–16(b) show that the loss value of cost decreases gradually with increasing number of episodes. When the episode number is higher than 1000, all learning curves become stable and no longer decrease. This result indicates that the proposed DQN-based learning algorithm converges after 1000 episodes. It means that the proposed DQN-based learning algorithm can learn an optimal strategy after 1000 episodes to minimize the long-term cost.Figure 11
Learning curves w.r.t. different risk rates.
(a)
(b)Figure 12
Learning curves w.r.t. different number of CPU cores.
(a)
(b)Figure 13
Learning curves w.r.t. different workloads.
(a)
(b)Figure 14
Learning curves w.r.t. different data sizes.
(a)
(b)Figure 15
Learning curves w.r.t. different task arrival rates.
(a)
(b)Figure 16
Learning curves w.r.t. different number of edge servers.
(a)
(b)


## 7. Conclusion and Future Work

In this paper, we investigate the security and cost-aware offloading problem and formulate it as a Markov decision process. To find the optimal offloading policy, we propose a security and cost-aware computing offloading (SCACO) strategy based on a deep Q-network (DQN), the objective of which is to minimize the total cost subjecting to the risk rate constraint in mobile edge computing. We evaluate the performance of the proposed offloading scheme under various performance metrics. Our experimental results show that the SCACO strategy can effectively decrease the total cost while the risk rate constraints are satisfied. Especially, the SCACO strategy can achieve the security guard for the security-critical tasks in mobile edge computing. In our experiment, we mainly investigate that the risk rate of security service, security service, risk coefficient, edge servers’ computing capacity, tasks workload, task data size, task arrival rate, and the number of edge servers influence on the long-term cost. The extensive experiments demonstrate the effectiveness of the SCACO strategy. In future work, we will further investigate the offloading problem for multiple mobile devices which offload computation tasks to multiple edge servers.



---

*Source: 3816237-2019-12-23.xml*