# Energy-Efficient Cloudlet Management for Privacy Preservation in Wireless Metropolitan Area Networks

Authors: Xiaolong Xu; Rui Huang; Ruihan Dou; Yuancheng Li; Jie Zhang; Tao Huang; Wenbin Yu
Journal: Security and Communication Networks
 (2018)

---

## Abstract

Currently, cloudlet based wireless metropolitan area network (WMAN) is emerging as an effective paradigm to extend the performance of mobile devices, which enables the execution of computational intensive mobile applications. But the normal operation of cloudlets consumes a large amount of energy, which brings about carbon dioxide emissions, aggravation of the greenhouse effect, etc. Meanwhile the data transmission of various mobile users among cloudlets would cause leakage of personal privacy. In view of this challenge, we propose an energy-efficient cloudlet management method, named ECM, for privacy preservation in WMAN. Technically, an optimization model is designed to formalize our problem. Then, based on live virtual machine (VM) migration technique, a corresponding privacy-aware VM scheduling method for energy saving is designed to determine which VMs should be migrated and where they should be migrated. Finally, experimental data demonstrate that the proposed method is both efficient and effective.


---

## Body

## 1. Introduction

In recent years, wireless metropolitan area network (WMAN) is emerging as a public network, which covers the metropolis and provides abundant services for human beings in urban cities [1]. Mobile devices in WMAN ubiquitously access rich resources from remote data centers via access points (APs). With the popularity and development of mobile devices, the demands of mobile users for wireless broadband are constantly increasing. WMAN is useful and powerful to satisfy such increasing resource requirements [2, 3].In practice, as the service requirements of mobile users increase, mobile applications, such as interactive gaming, virtual reality, and natural language processing, are becoming more computation-intensive. But the computing capacity of mobile devices remains limited, due to the service experience of the mobile devices about weight, size, and battery life [4]. A powerful and effective way to extend the performance of mobile devices is to offload some loads on the mobile devices to the remote clouds for execution. Most existing researches in computation offloading tend to regard the cloud as the remote offloading destination, due to its abundant computing resources [5]. However, the cloud generally is located remotely and far away from the mobile users, which leads to the unneglectable and unavoidable communication delay between the mobile device and the remote cloud. To reduce such communication delay, cloudlets, which are some small mobile data centers deployed in WMAN, are introduced in the service framework of WMAN [6, 7].There are a large amount of APs colocated with cloudlets in WMAN, where mobile devices are available to access the cloudlet resources easily, provided in the form of virtual machines (VMs) [8]. The obvious advantage of the cloudlet, compared to cloud, is the close physical distance between the mobile devices and the cloudlet, which shortens the communication delay and improves the experience of mobile users. In such a cloudlet based WMAN, users can deploy computational intensive mobile applications to cloudlets nearby, such as resolving HD video, real-time large-scale online games, and real-time navigation. However, when the cloudlets deployed in WMAN are placed incorrectly or many mobile users concurrently access the cloudlet resources in a region, high communication delay which affects the experience of mobile users would be generated in WMAN [9, 10]. Therefore, it is necessary to pay attention to the communication delay for cloudlet management.On the other hand, in order to support the efficient and effective operation of each cloudlet deployed in WMAN, a certain amount of energy is necessary to be consumed by the computing nodes and the cooling systems in the cloudlets [11, 12]. However, when the user traffic of cloudlets in some areas of urban cities is low, the usage of cloudlet resources would decrease. And if these cloudlets keep operating, excessive energy consumption continues to grow, which brings about some effects in urban cities, such as increased carbon dioxide emissions and aggravating the greenhouse effect [13, 14]. As a result, it is of great importance to implement cloudlet management for WMAN in an energy-efficient way.Live VM migration technique provides potential and opportunities for cloudlet management to reduce energy consumption. Live VM migration is of great benefit to improve the usage of cloudlet resources, which does a great favor to reduce the number of running cloudlets and in turn achieve the energy efficiency. Consider the scenario that a cloudlet near a scenic spot is heavily loaded during the daytime, but this cloudlet has little load at night, so that these running VMs on this cloudlet could be migrated to another cloudlet and turn this cloudlet off to reduce energy consumption.Moreover, due to the continuous evolution of WMAN and increasing diverse set of mobile users, WMAN has significant privacy problems [1]. When using the cloudlets to extend the performance of mobile devices, multiple mobile services with the personal private information could be accessed by multiple cloudlets due to the VM migrations. This information is very sensitive, as users’ preferences, habits, and interests could be inferred from them. Especially in an open network such as WMAN, personal information is easily leaked with high probability. The more nodes data pass through in WMAN when the data about users’ privacy is transmitted, the more possibly users’ privacy could be leaked [15]. To realize privacy preservation of mobile users, it is necessary to reduce the data transmission hops due to VM migrations [16].With these observations, it is still a challenge to realize energy-efficient resource management for cloudlets, while considering the privacy preservation due to VM migrations. In view of this challenge, a VM scheduling method for cloudlets in WMAN, to save energy and protect privacy, is proposed in this paper [17]. Our main contributions are threefold. Firstly, an optimization model for energy consumption of VM migrations in the cloudlets is designed. Secondly, taking into account the transmission delay and the possibility of privacy leakage, an energy-efficient cloudlet management method, named as ECM, is proposed to achieve the goal of energy saving and privacy preservation, through VM migrations. Specifically, ECM consists of five main steps, i.e., the destination cloudlet identification, shortest time identification, VM migration scheduling strategy designing, idle-state detection for cloudlets, and global VM0scheduling. Finally, comprehensive experiments and simulations are conducted to demonstrate the validity of our proposed method.The structure of this paper is organized as follows. Section2 shows the analysis of the energy consumption of cloudlets and privacy preservation in WMAN. An effective method of VM scheduling to save energy and privacy preservation is illustrated in Section 3. Section 4 presents the experiment results. Section 5 reviews the related work. Section 6 concludes the paper and gives an outlook of the future work [18].


## 2. Preliminary Knowledge

In this section, we introduce some formalized concepts and definitions of the cloudlet management in WMAN. Besides, we analyze the potential energy consumption and privacy preservation in the paper. Key terms and descriptions used in this paper are listed in Table1.Table 1
Key terms and descriptions.
Terms Description C The set of running cloudlets, C ={c1,c2,…,cN} K The set of tasks running on the cloudlets,K={k1,k2,…,kM} A The set of APs deployed in WMAN,A={a1,a2,…,aW} E ( t ) The total energy consumption without VM migrations gn The number of VM instancescn (n={1,2,…,N}) accommodates. λ The data transmission rate between APs ε n The resource utilization ofcn (n={1,2,…,N}) Y ( t ) The data transmission delay time Θ n , i ( t ) The running time ofcn (n={1,2,…,N}) r m The requested number of VM instances. s m The VM instances in the same cloud for executing the same tasktm could be composed as a special VM


### 2.1. Basic Concepts and Definitions

In WMAN, mobile users access the computation and storage resources through the access points (APs) and offload their computing tasks to the cloudlets for resource response [19].Suppose that there areN cloudlets in the WMAN, denoted as C={c1,c2,…,cN} and there areM mobile tasks running on the cloudlets, denoted asK = {k1,k2,…,kM}. LetW be the amount of the APs deployed in the WMAN, and the AP set is denoted asA = {a1,a2,...,aW}. Figure 1 shows an example of WMAN framework with five APs (i.e.,a1,a2,a3,a4, anda5) and three cloudlets (i.e.,c1,c2, andc3). In this example,c1 is connected toa1,c2 is connected toa2, andc3 is connected toa3. Mobile users access resources from the cloudletsc1 andc2 via the APsa1 anda2. Without loss of generality, we assume that there is only one physical machine (PM) deployed in a cloudlet. Letgn be the number of VM instances accommodated in thenth (n = {1,2,..,N}) cloudletcn.Figure 1
An example of WMAN with five APs and three cloudlets.Some basic concepts for further energy-aware VM scheduling method design about cloudlet management in WMAN are presented as follows.Definition 1 (VM instance requirement oftm).
The VM instance requirement oftm is defined by the requested number of VM instances, denoted asrm.Definition 2 (composed VM fortm).
The VM instances in the same cloud for executing the same tasktm could be composed as a special VM, denoted as sm.
As the tasks are deployed on the VMs among the cloudlets, there areM running composed VMs, denoted asS = {s1,s2,…,sM}.Definition 3 (the data transmission rate between Aps).
When the tasks and VMs are migrated betweenpw and pw′, there is a delay limited by the bandwidth betweenpw and pw′. We call it data transmission rate, denoted as λw,w′.


### 2.2. Energy Consumption Analysis

The energy consumption of cloudlets contains several aspects, including the energy consumption of active VMs, the energy consumed by the idle VMs, and the baseline energy consumption of running cloudlets. These three parts of energy consumption are calculated as follows [11].The energy consumption of active VMs at the time instantt for thenth cloudletcn inC is calculated by(1)VEnactivet=∑i=1gnan,i·ϕn,i,where αn,i is the energy consumption rate of vn,i in active mode and ϕn,i is the total operating time of vn,i.In fact, the idle VMs with no task executions also have a baseline energy consumption. The idle time is determined by the running time of the longest running VM. The running time of the longest running VM is calculated by(2)θn,it=maxi=1gnϕn.i,Then the energy consumption of idle VMs incn is calculated by(3)VEnidlet=∑i=1gnβn,i·θn,it-ϕn,i,where βn,i is the energy consumption rate of vn,i in idle mode.When a cloudlet is active, there is at least one active VM running on it. It will consume the baseline energy. The energy for the all running cloudlets is calculated by(4)Ecloudlet=∑n=1Nγn·θn,it,where γn is the energy consumption rate ofcn.Through the above analysis, the total energy consumption is calculated by(5)Et=∑n=1NVEnactivet+∑n=1NVEnidlet+Ecloudlett,


### 2.3. Resource Utilization Analysis

The resource utilization is a standard for measuring whether the resources are used sufficiently. When the resource utilization ofcn is low, the VMs in this cloudlet should be migrated to other cloudlets, and then it could be shut down to save energy. The resource utilization ofcn which is denoted as εn is calculated by(6)εn=1gn·∑i=1gnvn,it,where vn,i(t) is a binary variable employed to judge whether vn,i hosts a load, defined by(7)vn,it=0,ifϕn,i=0,1,Otherwise.So, the average resource utilization of all the cloudlets is calculated by(8)η=1N′·∑n=1Nεn,where N′ is the number of all active cloudlets, calculated by(9)N′=∑n=1Nfnt,where fn(t) is a binary variable employed to judge whethercn accommodates a mobile task, defined by(10)fnt=0,if∑i=1gnϕn,i=0,1,Otherwise.


### 2.4. Delay and Privacy Preservation Analysis

Besides, when the data are transmitted betweenvi and vi′, there is always a delay due to the data transmission rate, which affects the experience of mobile users. The delay of all the tasks for data transmission among VMs is calculated by(11)Yt=∑i=1gn∑i′=1gnDi,i′λi,i′,where Di,i′ is the total size of data which are transferred fromvi to vi′ and λi,i′ is the data transmission rate which is mentioned above.When data are transmitted from one cloudlet to another, it is relayed through multiple APs. The APs in the WMAN have limited storage and computational capabilities, as well as low security, and the mobile users are vulnerable to violations while transmitting the applications and data with private personal information. An efficient and effective method to solve this problem is decreasing the number of APs that the transmitted data pass through. When there are fewer nodes which data passes through, the probability of privacy leakage is lowered; thus we could achieve the goal of privacy preservation.


## 2.1. Basic Concepts and Definitions

In WMAN, mobile users access the computation and storage resources through the access points (APs) and offload their computing tasks to the cloudlets for resource response [19].Suppose that there areN cloudlets in the WMAN, denoted as C={c1,c2,…,cN} and there areM mobile tasks running on the cloudlets, denoted asK = {k1,k2,…,kM}. LetW be the amount of the APs deployed in the WMAN, and the AP set is denoted asA = {a1,a2,...,aW}. Figure 1 shows an example of WMAN framework with five APs (i.e.,a1,a2,a3,a4, anda5) and three cloudlets (i.e.,c1,c2, andc3). In this example,c1 is connected toa1,c2 is connected toa2, andc3 is connected toa3. Mobile users access resources from the cloudletsc1 andc2 via the APsa1 anda2. Without loss of generality, we assume that there is only one physical machine (PM) deployed in a cloudlet. Letgn be the number of VM instances accommodated in thenth (n = {1,2,..,N}) cloudletcn.Figure 1
An example of WMAN with five APs and three cloudlets.Some basic concepts for further energy-aware VM scheduling method design about cloudlet management in WMAN are presented as follows.Definition 1 (VM instance requirement oftm).
The VM instance requirement oftm is defined by the requested number of VM instances, denoted asrm.Definition 2 (composed VM fortm).
The VM instances in the same cloud for executing the same tasktm could be composed as a special VM, denoted as sm.
As the tasks are deployed on the VMs among the cloudlets, there areM running composed VMs, denoted asS = {s1,s2,…,sM}.Definition 3 (the data transmission rate between Aps).
When the tasks and VMs are migrated betweenpw and pw′, there is a delay limited by the bandwidth betweenpw and pw′. We call it data transmission rate, denoted as λw,w′.


## 2.2. Energy Consumption Analysis

The energy consumption of cloudlets contains several aspects, including the energy consumption of active VMs, the energy consumed by the idle VMs, and the baseline energy consumption of running cloudlets. These three parts of energy consumption are calculated as follows [11].The energy consumption of active VMs at the time instantt for thenth cloudletcn inC is calculated by(1)VEnactivet=∑i=1gnan,i·ϕn,i,where αn,i is the energy consumption rate of vn,i in active mode and ϕn,i is the total operating time of vn,i.In fact, the idle VMs with no task executions also have a baseline energy consumption. The idle time is determined by the running time of the longest running VM. The running time of the longest running VM is calculated by(2)θn,it=maxi=1gnϕn.i,Then the energy consumption of idle VMs incn is calculated by(3)VEnidlet=∑i=1gnβn,i·θn,it-ϕn,i,where βn,i is the energy consumption rate of vn,i in idle mode.When a cloudlet is active, there is at least one active VM running on it. It will consume the baseline energy. The energy for the all running cloudlets is calculated by(4)Ecloudlet=∑n=1Nγn·θn,it,where γn is the energy consumption rate ofcn.Through the above analysis, the total energy consumption is calculated by(5)Et=∑n=1NVEnactivet+∑n=1NVEnidlet+Ecloudlett,


## 2.3. Resource Utilization Analysis

The resource utilization is a standard for measuring whether the resources are used sufficiently. When the resource utilization ofcn is low, the VMs in this cloudlet should be migrated to other cloudlets, and then it could be shut down to save energy. The resource utilization ofcn which is denoted as εn is calculated by(6)εn=1gn·∑i=1gnvn,it,where vn,i(t) is a binary variable employed to judge whether vn,i hosts a load, defined by(7)vn,it=0,ifϕn,i=0,1,Otherwise.So, the average resource utilization of all the cloudlets is calculated by(8)η=1N′·∑n=1Nεn,where N′ is the number of all active cloudlets, calculated by(9)N′=∑n=1Nfnt,where fn(t) is a binary variable employed to judge whethercn accommodates a mobile task, defined by(10)fnt=0,if∑i=1gnϕn,i=0,1,Otherwise.


## 2.4. Delay and Privacy Preservation Analysis

Besides, when the data are transmitted betweenvi and vi′, there is always a delay due to the data transmission rate, which affects the experience of mobile users. The delay of all the tasks for data transmission among VMs is calculated by(11)Yt=∑i=1gn∑i′=1gnDi,i′λi,i′,where Di,i′ is the total size of data which are transferred fromvi to vi′ and λi,i′ is the data transmission rate which is mentioned above.When data are transmitted from one cloudlet to another, it is relayed through multiple APs. The APs in the WMAN have limited storage and computational capabilities, as well as low security, and the mobile users are vulnerable to violations while transmitting the applications and data with private personal information. An efficient and effective method to solve this problem is decreasing the number of APs that the transmitted data pass through. When there are fewer nodes which data passes through, the probability of privacy leakage is lowered; thus we could achieve the goal of privacy preservation.


## 3. A Cloudlet Management Method for Energy Conservation with Privacy Preservation

Based on the analysis of energy and the possibility of VM migrations for privacy preservation in WMAN in Section2, an energy-efficient cloudlet management method is proposed in this section.


### 3.1. Method Overview

Technically, as specified in Box1, our proposed cloudlet management method for energy saving and privacy preservation consists of five steps, i.e., the destination cloudlet identification, privacy-aware cloudlet confirmation, VM scheduling strategy obtaining, idle-state detection for cloudlets and global VM scheduling. The destination cloudlet identification is designed to confirm and record the active cloudlets with idle spaces, which could be employed to host the migrated VMs. These obtained cloudlets could be served as the input for the processing of Step 2. Step 2 is developed for the VMs in the source cloudlet to select the final cloudlet with the minimum data transmission time and, at the same time, achieve the purpose of privacy preservation. Step 3 is designed to produce the available migration strategies. Step 4 detects the idle state of active cloudlets in a dynamic way. All the achieved migration strategies and records are the input of the final global VM scheduling.Box 1:Specification of VM scheduling method for energy conservation.
Step  1: Destination cloudlet identification. Select a certain range of cloudlets with free VMs, which form a primitive cloudlet collection. This collection is identified for VM migration. Step  2: Privacy-aware cloudlet confirmation. Confirm the cloudlet with the shortest data transmission time for the source cloudlet from the cloudlet collection obtained in Step1. Step  3: VM scheduling strategy obtaining. Migrating the tasks to the cloudlet obtained from Step 1 and Step 2. Update the migration policies and the task storage records. Step  4: Idle-state detection for cloudlets. Detect the idle state of each cloudlet, and calculate the resource utilization for the cloudlets, which is used as the migrated standard of dynamic scheduling. Step  5: Global VM scheduling. Based on the VM migration strategies obtained in Step 3, the dynamic VM scheduling strategies for all the running VMs are achieved according to the cloudlet utilization obtained in Step4.


### 3.2. Destination Cloudlet Identification

When tasks are being hosted on a VM, each task may have a different requirement of idle VMs. The active cloudlets can host multiple VMs for task execution. For example, if a task requiring three virtual machines is migrated from another low-load virtual machine, the destination cloudlet must have more than or equal to three idle VMs. Otherwise, the task cannot be migrated to the currently selected cloudlet. Therefore, in Algorithm1, we design the algorithm to find the migrated cloudlets which have the idle VMs and meet the requests. It plays an important role in cloudlet management.Algorithm 1:Destination cloudlet identification DCI.
Input:The acquired number q of idle VMs VM occupation record orz Output:The selected running cloudlets RS={rs1,rs2,…,rsd} ( 1 ) Get the running cloudlets lists from orz ( 2 ) Sort the PM in cloudlets in the decreasing order of idle VMs, PMs={pm1,pm2,…pmN} ( 3 )  flag = 1, H = 0, n= 1 ( 4 ) while  flag = 1  do ( 5 ) Get the number of idle VMs, i.e., rn ( 6 )  if  rn can hosts the VMs &&H < d  then ( 7 ) Add rn to rsd ( 8 )  n=n+1 ( 9 )  H = H + 1 ( 10 )  else  if  n ==N∥H== d  then ( 11 )  f l a g = 0 (12)end if (13)end if (14)end while (15)Return  RSOnce the resource requirement of a task is responded by a VM instance, it generates a VM occupation record for the resource manager. Here, each cloudlet hosts only one PM. LetOR ={or1,or2,….,orZ} be the VM occupation record collection, whereZ is the number of the records inOR, defined as follows.Definition 4 (VM occupation recordorz).
o r z records the occupation status of the task deployed in the cloudlet, which is denoted asorz= (tidz,cidz,pidz,vidz,stimz,dtimz), wheretidz,cidz,pidz,vidz,stimz, anddtimz are the allocated task, the occupied cloudlet, the relevant employed PM in the cloudlet, the applied VM instance, the service start time, and the duration time, respectively.Here we select the VMs that meet the task requirements. To better schedule the VMs in an energy-efficient way, they are classified according to VM occupation records. These records generate from the destination cloudlets once current round of migration is finished.Firstly, we get the running cloudlet list fromorz and then sort the cloudlets in the decreasing order of idle VMs; that is, we need to sort the different cloudlets according to the load distribution of the cloudlets. Suppose we want to selectd cloudlets that meet the requirements. The cloudlets with higher load, meeting the resource requirements, are selected. Once the cloudlet meets the requirement, we add it to the selected running cloudlet list, denoted as RS={rs1,rs2,…,rsd}. When we traverse all the cloudlets in the original list or find the number of qualified VMs that meet expectations, the searching processing would be stopped.Algorithm1 specifies the process of destination cloudlet identification. In this algorithm, we first sort the PMs in cloudlets in descending order of load (Lines (1) and (2)). Then we identify the cloudlets that satisfy the meets of idle VMs (Lines (3) to (14)).


### 3.3. Privacy-Aware Cloudlet Identification

Algorithm1 identifies a range of cloudlets that meet the criteria. In this section, we use Dijkstra's algorithm to sort the selected cloudlets and then select the closest one to migration tasks from source cloudlet to the destination, with shortest path [20].Besides, the processes of VM scheduling could lead to potential risks due to the privacy leakage in WMAN. In order to cater for the application scenario of WMAN, our algorithm is in demand to take privacy protection into consideration. In view of this challenge, we find that using the algorithm of shortest time identification based on Dijkstra is an appropriate method to achieve the goal of protecting users’ privacy. When identifying the shortest data transmission time between source cloudlet and destination cloudlet, the data also go through the fewest APs; in other words, due to the principle of Dijkstra, we reduce the possibility of privacy leaks.Definition 5 (data transmission timedisij).
The transmission time between the cloudletsi andj is denoted asdisij (1≤i<j ≤d) according to the transmission delay among the APs of these two cloudlets.Firstly, all the vertices should be divided into two parts: i.e., the known shortest time vertex setPd and the unknown shortest time vertex setQd. Initially, only one collection of the source cloudlet is known in the collectionPd of the shortest time. Secondly, the source cloudletCs is set by its own shortest time as 0, that is,disCs= 0, if there is a vertexi that the source point can directly reach, setdisi toysi. The cloudlet with the smallest data transmission time among the collectionQd is selected, and it should be added to the collectionPd. And all the edges starting from this cloudlet are examined. For example, if there is an edge fromCi toCj, then the transmission time fromCs toCj can be extended by adding the edgeCi →Cj to the tail. The length of this time isdisi +yij. If this value is smaller than the currently known value ofdisi, we can replace the current value indisi with the new one. If the collectionQd is empty, the algorithm ends. Then return the cloudlet with the shortest transmission time.Algorithm2 specifies the process of privacy-aware cloudlet identification. In this algorithm, we firstly divide all the vertices into two parts (Lines (1) and (2)); then we use Dijkstra algorithm to find the destination with the shortest time (Lines (3) to (11)). At last, we return the result (Lines (12) and (13)).Algorithm 2:Privacy-aware cloudlet identification.
Input:The selected running cloudlets RS={rs1,rs2,..,rsd} The known shortest time vertex setP={p1,p2,…,pd} The unknown shortest time vertex setQ={q1,q2,…,qd} The collectionsyij and disi The VM occupation recordorz Output: The cloudlet with the shortest time Cdij ( 1 ) Get the transmission time between each cloudlet in RS from orz to yij ( 2 ) Add the source cloudlet (Cs) to p and set its dis = 0 ( 3 ) if C i is directly connected to the source cloudletthen ( 4 )  disi=ysi ( 5 ) end if ( 6 ) sort Q in the increasing order of the transmission time ( 7 )  for each cloudlet in Q  do ( 8 )   if  disi + yij<di  then ( 9 )   disi=disi+yij ( 10 ) end if ( 11 )  end for ( 11 ) sort  disi  in the increasing order of the transmission time (12)Cdij=di (13)return  Cdij


### 3.4. VM Scheduling Strategy Confirmation

In order to obtain the optimal energy savings, we try to migrate the tasks on the low-load cloudlets to the high-load cloudlets as much as possible, based on the results achieved by the above analysis. Once all the tasks on a cloudlet are migrated, the cloudlet could be changed to the sleep mode to save power. At the same time, a fully loaded cloudlet can also achieve higher resource utilization. And each migration procedure is recorded in the relevant VM occupation records with the migration time instantt.Definition 6 (migration composition collectionMCd).
For all the running VMs inrsd, the synchronous VM migration strategies constitute the migration composition.
Suppose there areZ cloudlets inORZ andK migration compositions, i.e., {l1,l2,…,lK}, for each turn of migration composition generation. First, we sort the cloudlets in the decreasing order of idle VMs, which means sorting from low load to high load. Here, flag is a sign. Get cloudlets to migrate from low load to high load from the list of cloudlets.To facilitate the energy-efficient VM scheduling, the running VM list and hosting tasks in the occupation record setORz are obtained first. Then for each task in the list, we use Algorithms 1 and 2 to identify the most suitable cloudlet for migration and migrate the task from the high-load cloudlet to the low-load cloudlet. And once a task in a low-load VM cannot be all migrated, the temporal migration strategies should be removed. For example, if a cloudlet hosts two tasks, the first task takes up one VM instance and the other takes up two VM instances. The two VMs instances of the second task should be placed in the same target cloudlet. Therefore, we should migrate the second task first. If the second task can be migrated but the first one does not work, leave the two tasks on the original cloudlet. Otherwise, the migration is successful and the temporary migration recordsc1~ck should be updated to the migration record collectionMCd, respectively.Here, after the VM migrations at the time instantt, the start time and duration of the relevant VM occupation records should be updated, according to the real occupation time. Specifically, if the migration occurs, the duration of the task on the original cloudlet is necessary to be modified. For the task that is migrated, the relevant new VM occupation records are generated, the start time is updated tot, and the duration also needs to be changed as the predicted occupation time.Algorithm3 specifies the process of VM scheduling strategy confirmation. We firstly sort the cloudlets in the decreasing order according to the load (Line (1)). Then we move the tasks on the low-load cloudlets to the high-load cloudlets and store the records in the list migration recordMCd (Lines (2) to (21)).Algorithm 3:VM scheduling strategy confirmation.
Input:The running cloudlet set rsd; VM occupation record orz VM occupation recordorz Output: The migration composition collection MCd ( 1 ) Sort the cloudlets in the decreasing order of idle VMs ( 2 ) f l a g = 0, z=1, K=1 ( 3 )  whileflag=0  do ( 4 )  Get the running VM list in ORz ( 5 )   Get the hosting task list in ORz ( 6 )   for each hosted task do ( 7 )     Get the number of occupied VMs ( 8 )     Select the destination cloudlet by Algorithms 1 and 2 ( 9 )    end for ( 10 )   if all the VMs can be migrated away then ( 11 )   Generate a migration composition lK (12)   Update the cloudlet status and VM distribution (13)elseflag=1 (14)   Delete the temporal migration strategies (15)end if (16)z=z+1,K=K+1 (17)end while (18)for the reserved tasks’ records do (19)dtimk=t (20)end for (21)fork=1to K  do (22)  Addl1~lk as a migration composition to MCd (23)  Get the start time and time of each migration task (24)dtimk=dtimk- t (25)stimk=t (26)  Update the start time and duration time inMCd (27)  Update task storage recordcsx (28)end for (29)Return  MCd


### 3.5. Idle-State Detection for Cloudlets

The idle-state detection of active cloudlets is also one of the important factors for global migration considerations. It is a significant standard for the global VM scheduling. When the next migration is going to be conducted, we should evaluate the idle-state of the active cloudlets, to confirm and record the idle space before the migration timet. After detecting the idle space, the current cloudlet resource utilization before the migration time could be achieved.As the running time changes dynamically, some tasks would be competed over time. In this section, we detect the idle state of cloudlets as time progress. Therefore, to find the available destination cloudlet, the idle space and resource utilization of the all the running cloudlets should be detected first.Once the migration strategy is designed in Algorithm3, it generates one or more VM occupation records based on the migration records for the resource manager. So, we should also update the task storage records before the migration based on the latest migration strategy. Suppose there arex tasks that still need to be hosted. Let CS={cs1,cs2,…,csx} be the task occupation record collection, wherex is the number of the records inCS, defined as follows.Definition 7 (task storage recordcsx).
cs xrecords the storage status of the tasks deployed in the cloudlet, whose number isx, start time is denoted asstimx, and duration time is denoted asdtimx, which is dynamically changed.The running cloudlet list from the task storage recordcsxis gotten first as the main data. Secondly, all the active tasks are traversed and the cloudlet idle space of all the cloudlets needs to be updated accordingly. Then, use the formula (9) to calculate the number of idle VM instances and the amount of the active cloudlets. At last,RP is returned as the final results of the idle spaces of the definite cloudlet.Algorithm4 specifies the process of idle-state detection of cloudlets. Firstly, we update the cloudlet idle space (Lines (1) to (6)). Then we calculate the resource utilization and return it (Lines (7) and (8)).Algorithm 4:Cloudlet idle-state detection.
Input: The task storage record csx Migration timet Output: The current cloudlet resource utilization RP ( 1 ) Get running cloudlet list from the csx of time t ( 2 ) foreach hosted taskdo ( 3 )  Get the number of occupied VMs ( 4 ) end for ( 5 ) Update the cloudlet idle space ( 6 ) end for ( 7 ) Calculate the current cloudlet resource utilization RP by (9) ( 8 ) Return RP


### 3.6. Global VM Scheduling

To achieve the goal of energy-efficient cloudlet management in WMAN, the optimal VM migration composition should be confirmed from the migration strategies which is achieved by Algorithm3. In this section, global VM scheduling is designed to determine which VMs and where should be migrated for the whole execution period of the VMs.Due to the goal of scheduling all the running VMs from a global perspective in this section, migration operation time is necessary for monitoring the resource utilization of all the cloudlets through the calculating of the current VM occupation records. The number of idle VMs of each cloudlet can be detected dynamically. Algorithm4 is designed to acquire the idle space as the migration standard which is one of the inputs in Algorithm 5.Algorithm 5:Global VM scheduling.
Input: The VMs V={v1,v2,⋯,vz} Migration timet Total operation timeH Output: The cloudlets scheduling policy ( 1 ) Get the migration time t; ( 2 )  Get the number of idle VMs on each cloudlets ( 3 )  whilet<H  do ( 4 ) Get RP from Algorithm 4 ( 5 )   ifRP < utilization threshold level then ( 6 )   Update the migration time t as the next finish time of the VMs. ( 7 )   Get migration compositions MCh by Algorithm 3 ( 8 )   Schedule the VMs by the policy ( 9 )   Set the vacant cloudlets to the sleeping mode ( 10 )  end if ( 11 )  end whileAs we aim to schedule the VMs with the goals of energy saving and privacy preservation, the VM migration composition with highest utilization will be chosen as the ultimate VM scheduling policy. And after VM scheduling for the identified VMs, the vacant PMs should be set to the sleeping mode to save the energy. Besides, through the Algorithm2, we can also attain another purpose for reducing the possibility of privacy leaks.The migration times are necessary to be obtained for global VM scheduling according to the utilization threshold. Besides, the number of the idle VMs on each cloudlet is a key measure to decide the destination cloudlet of the migrations. During the execution period of all the mobile applications in WMAN, the resource utilizationRPof the cloudlets is obtained by Algorithm 4. Then, if the currentRP is less than utilization threshold, that is, there are enough space in the active cloudlets after the previous round of scheduling, we need to schedule the VMs by the policy and set the vacant cloudlets to the sleeping mode according toMCh by Algorithm 3. Thus, we achieve the goal of dynamic scheduling.Algorithm5 specifies the process of global VM scheduling (t). Firstly, we update theRPandMChof the migration time (Lines (1) to (6)). Then schedule the running VMs and set the vacant cloudlets to the sleeping mode (Lines (8) to (10)).


## 3.1. Method Overview

Technically, as specified in Box1, our proposed cloudlet management method for energy saving and privacy preservation consists of five steps, i.e., the destination cloudlet identification, privacy-aware cloudlet confirmation, VM scheduling strategy obtaining, idle-state detection for cloudlets and global VM scheduling. The destination cloudlet identification is designed to confirm and record the active cloudlets with idle spaces, which could be employed to host the migrated VMs. These obtained cloudlets could be served as the input for the processing of Step 2. Step 2 is developed for the VMs in the source cloudlet to select the final cloudlet with the minimum data transmission time and, at the same time, achieve the purpose of privacy preservation. Step 3 is designed to produce the available migration strategies. Step 4 detects the idle state of active cloudlets in a dynamic way. All the achieved migration strategies and records are the input of the final global VM scheduling.Box 1:Specification of VM scheduling method for energy conservation.
Step  1: Destination cloudlet identification. Select a certain range of cloudlets with free VMs, which form a primitive cloudlet collection. This collection is identified for VM migration. Step  2: Privacy-aware cloudlet confirmation. Confirm the cloudlet with the shortest data transmission time for the source cloudlet from the cloudlet collection obtained in Step1. Step  3: VM scheduling strategy obtaining. Migrating the tasks to the cloudlet obtained from Step 1 and Step 2. Update the migration policies and the task storage records. Step  4: Idle-state detection for cloudlets. Detect the idle state of each cloudlet, and calculate the resource utilization for the cloudlets, which is used as the migrated standard of dynamic scheduling. Step  5: Global VM scheduling. Based on the VM migration strategies obtained in Step 3, the dynamic VM scheduling strategies for all the running VMs are achieved according to the cloudlet utilization obtained in Step4.


## 3.2. Destination Cloudlet Identification

When tasks are being hosted on a VM, each task may have a different requirement of idle VMs. The active cloudlets can host multiple VMs for task execution. For example, if a task requiring three virtual machines is migrated from another low-load virtual machine, the destination cloudlet must have more than or equal to three idle VMs. Otherwise, the task cannot be migrated to the currently selected cloudlet. Therefore, in Algorithm1, we design the algorithm to find the migrated cloudlets which have the idle VMs and meet the requests. It plays an important role in cloudlet management.Algorithm 1:Destination cloudlet identification DCI.
Input:The acquired number q of idle VMs VM occupation record orz Output:The selected running cloudlets RS={rs1,rs2,…,rsd} ( 1 ) Get the running cloudlets lists from orz ( 2 ) Sort the PM in cloudlets in the decreasing order of idle VMs, PMs={pm1,pm2,…pmN} ( 3 )  flag = 1, H = 0, n= 1 ( 4 ) while  flag = 1  do ( 5 ) Get the number of idle VMs, i.e., rn ( 6 )  if  rn can hosts the VMs &&H < d  then ( 7 ) Add rn to rsd ( 8 )  n=n+1 ( 9 )  H = H + 1 ( 10 )  else  if  n ==N∥H== d  then ( 11 )  f l a g = 0 (12)end if (13)end if (14)end while (15)Return  RSOnce the resource requirement of a task is responded by a VM instance, it generates a VM occupation record for the resource manager. Here, each cloudlet hosts only one PM. LetOR ={or1,or2,….,orZ} be the VM occupation record collection, whereZ is the number of the records inOR, defined as follows.Definition 4 (VM occupation recordorz).
o r z records the occupation status of the task deployed in the cloudlet, which is denoted asorz= (tidz,cidz,pidz,vidz,stimz,dtimz), wheretidz,cidz,pidz,vidz,stimz, anddtimz are the allocated task, the occupied cloudlet, the relevant employed PM in the cloudlet, the applied VM instance, the service start time, and the duration time, respectively.Here we select the VMs that meet the task requirements. To better schedule the VMs in an energy-efficient way, they are classified according to VM occupation records. These records generate from the destination cloudlets once current round of migration is finished.Firstly, we get the running cloudlet list fromorz and then sort the cloudlets in the decreasing order of idle VMs; that is, we need to sort the different cloudlets according to the load distribution of the cloudlets. Suppose we want to selectd cloudlets that meet the requirements. The cloudlets with higher load, meeting the resource requirements, are selected. Once the cloudlet meets the requirement, we add it to the selected running cloudlet list, denoted as RS={rs1,rs2,…,rsd}. When we traverse all the cloudlets in the original list or find the number of qualified VMs that meet expectations, the searching processing would be stopped.Algorithm1 specifies the process of destination cloudlet identification. In this algorithm, we first sort the PMs in cloudlets in descending order of load (Lines (1) and (2)). Then we identify the cloudlets that satisfy the meets of idle VMs (Lines (3) to (14)).


## 3.3. Privacy-Aware Cloudlet Identification

Algorithm1 identifies a range of cloudlets that meet the criteria. In this section, we use Dijkstra's algorithm to sort the selected cloudlets and then select the closest one to migration tasks from source cloudlet to the destination, with shortest path [20].Besides, the processes of VM scheduling could lead to potential risks due to the privacy leakage in WMAN. In order to cater for the application scenario of WMAN, our algorithm is in demand to take privacy protection into consideration. In view of this challenge, we find that using the algorithm of shortest time identification based on Dijkstra is an appropriate method to achieve the goal of protecting users’ privacy. When identifying the shortest data transmission time between source cloudlet and destination cloudlet, the data also go through the fewest APs; in other words, due to the principle of Dijkstra, we reduce the possibility of privacy leaks.Definition 5 (data transmission timedisij).
The transmission time between the cloudletsi andj is denoted asdisij (1≤i<j ≤d) according to the transmission delay among the APs of these two cloudlets.Firstly, all the vertices should be divided into two parts: i.e., the known shortest time vertex setPd and the unknown shortest time vertex setQd. Initially, only one collection of the source cloudlet is known in the collectionPd of the shortest time. Secondly, the source cloudletCs is set by its own shortest time as 0, that is,disCs= 0, if there is a vertexi that the source point can directly reach, setdisi toysi. The cloudlet with the smallest data transmission time among the collectionQd is selected, and it should be added to the collectionPd. And all the edges starting from this cloudlet are examined. For example, if there is an edge fromCi toCj, then the transmission time fromCs toCj can be extended by adding the edgeCi →Cj to the tail. The length of this time isdisi +yij. If this value is smaller than the currently known value ofdisi, we can replace the current value indisi with the new one. If the collectionQd is empty, the algorithm ends. Then return the cloudlet with the shortest transmission time.Algorithm2 specifies the process of privacy-aware cloudlet identification. In this algorithm, we firstly divide all the vertices into two parts (Lines (1) and (2)); then we use Dijkstra algorithm to find the destination with the shortest time (Lines (3) to (11)). At last, we return the result (Lines (12) and (13)).Algorithm 2:Privacy-aware cloudlet identification.
Input:The selected running cloudlets RS={rs1,rs2,..,rsd} The known shortest time vertex setP={p1,p2,…,pd} The unknown shortest time vertex setQ={q1,q2,…,qd} The collectionsyij and disi The VM occupation recordorz Output: The cloudlet with the shortest time Cdij ( 1 ) Get the transmission time between each cloudlet in RS from orz to yij ( 2 ) Add the source cloudlet (Cs) to p and set its dis = 0 ( 3 ) if C i is directly connected to the source cloudletthen ( 4 )  disi=ysi ( 5 ) end if ( 6 ) sort Q in the increasing order of the transmission time ( 7 )  for each cloudlet in Q  do ( 8 )   if  disi + yij<di  then ( 9 )   disi=disi+yij ( 10 ) end if ( 11 )  end for ( 11 ) sort  disi  in the increasing order of the transmission time (12)Cdij=di (13)return  Cdij


## 3.4. VM Scheduling Strategy Confirmation

In order to obtain the optimal energy savings, we try to migrate the tasks on the low-load cloudlets to the high-load cloudlets as much as possible, based on the results achieved by the above analysis. Once all the tasks on a cloudlet are migrated, the cloudlet could be changed to the sleep mode to save power. At the same time, a fully loaded cloudlet can also achieve higher resource utilization. And each migration procedure is recorded in the relevant VM occupation records with the migration time instantt.Definition 6 (migration composition collectionMCd).
For all the running VMs inrsd, the synchronous VM migration strategies constitute the migration composition.
Suppose there areZ cloudlets inORZ andK migration compositions, i.e., {l1,l2,…,lK}, for each turn of migration composition generation. First, we sort the cloudlets in the decreasing order of idle VMs, which means sorting from low load to high load. Here, flag is a sign. Get cloudlets to migrate from low load to high load from the list of cloudlets.To facilitate the energy-efficient VM scheduling, the running VM list and hosting tasks in the occupation record setORz are obtained first. Then for each task in the list, we use Algorithms 1 and 2 to identify the most suitable cloudlet for migration and migrate the task from the high-load cloudlet to the low-load cloudlet. And once a task in a low-load VM cannot be all migrated, the temporal migration strategies should be removed. For example, if a cloudlet hosts two tasks, the first task takes up one VM instance and the other takes up two VM instances. The two VMs instances of the second task should be placed in the same target cloudlet. Therefore, we should migrate the second task first. If the second task can be migrated but the first one does not work, leave the two tasks on the original cloudlet. Otherwise, the migration is successful and the temporary migration recordsc1~ck should be updated to the migration record collectionMCd, respectively.Here, after the VM migrations at the time instantt, the start time and duration of the relevant VM occupation records should be updated, according to the real occupation time. Specifically, if the migration occurs, the duration of the task on the original cloudlet is necessary to be modified. For the task that is migrated, the relevant new VM occupation records are generated, the start time is updated tot, and the duration also needs to be changed as the predicted occupation time.Algorithm3 specifies the process of VM scheduling strategy confirmation. We firstly sort the cloudlets in the decreasing order according to the load (Line (1)). Then we move the tasks on the low-load cloudlets to the high-load cloudlets and store the records in the list migration recordMCd (Lines (2) to (21)).Algorithm 3:VM scheduling strategy confirmation.
Input:The running cloudlet set rsd; VM occupation record orz VM occupation recordorz Output: The migration composition collection MCd ( 1 ) Sort the cloudlets in the decreasing order of idle VMs ( 2 ) f l a g = 0, z=1, K=1 ( 3 )  whileflag=0  do ( 4 )  Get the running VM list in ORz ( 5 )   Get the hosting task list in ORz ( 6 )   for each hosted task do ( 7 )     Get the number of occupied VMs ( 8 )     Select the destination cloudlet by Algorithms 1 and 2 ( 9 )    end for ( 10 )   if all the VMs can be migrated away then ( 11 )   Generate a migration composition lK (12)   Update the cloudlet status and VM distribution (13)elseflag=1 (14)   Delete the temporal migration strategies (15)end if (16)z=z+1,K=K+1 (17)end while (18)for the reserved tasks’ records do (19)dtimk=t (20)end for (21)fork=1to K  do (22)  Addl1~lk as a migration composition to MCd (23)  Get the start time and time of each migration task (24)dtimk=dtimk- t (25)stimk=t (26)  Update the start time and duration time inMCd (27)  Update task storage recordcsx (28)end for (29)Return  MCd


## 3.5. Idle-State Detection for Cloudlets

The idle-state detection of active cloudlets is also one of the important factors for global migration considerations. It is a significant standard for the global VM scheduling. When the next migration is going to be conducted, we should evaluate the idle-state of the active cloudlets, to confirm and record the idle space before the migration timet. After detecting the idle space, the current cloudlet resource utilization before the migration time could be achieved.As the running time changes dynamically, some tasks would be competed over time. In this section, we detect the idle state of cloudlets as time progress. Therefore, to find the available destination cloudlet, the idle space and resource utilization of the all the running cloudlets should be detected first.Once the migration strategy is designed in Algorithm3, it generates one or more VM occupation records based on the migration records for the resource manager. So, we should also update the task storage records before the migration based on the latest migration strategy. Suppose there arex tasks that still need to be hosted. Let CS={cs1,cs2,…,csx} be the task occupation record collection, wherex is the number of the records inCS, defined as follows.Definition 7 (task storage recordcsx).
cs xrecords the storage status of the tasks deployed in the cloudlet, whose number isx, start time is denoted asstimx, and duration time is denoted asdtimx, which is dynamically changed.The running cloudlet list from the task storage recordcsxis gotten first as the main data. Secondly, all the active tasks are traversed and the cloudlet idle space of all the cloudlets needs to be updated accordingly. Then, use the formula (9) to calculate the number of idle VM instances and the amount of the active cloudlets. At last,RP is returned as the final results of the idle spaces of the definite cloudlet.Algorithm4 specifies the process of idle-state detection of cloudlets. Firstly, we update the cloudlet idle space (Lines (1) to (6)). Then we calculate the resource utilization and return it (Lines (7) and (8)).Algorithm 4:Cloudlet idle-state detection.
Input: The task storage record csx Migration timet Output: The current cloudlet resource utilization RP ( 1 ) Get running cloudlet list from the csx of time t ( 2 ) foreach hosted taskdo ( 3 )  Get the number of occupied VMs ( 4 ) end for ( 5 ) Update the cloudlet idle space ( 6 ) end for ( 7 ) Calculate the current cloudlet resource utilization RP by (9) ( 8 ) Return RP


## 3.6. Global VM Scheduling

To achieve the goal of energy-efficient cloudlet management in WMAN, the optimal VM migration composition should be confirmed from the migration strategies which is achieved by Algorithm3. In this section, global VM scheduling is designed to determine which VMs and where should be migrated for the whole execution period of the VMs.Due to the goal of scheduling all the running VMs from a global perspective in this section, migration operation time is necessary for monitoring the resource utilization of all the cloudlets through the calculating of the current VM occupation records. The number of idle VMs of each cloudlet can be detected dynamically. Algorithm4 is designed to acquire the idle space as the migration standard which is one of the inputs in Algorithm 5.Algorithm 5:Global VM scheduling.
Input: The VMs V={v1,v2,⋯,vz} Migration timet Total operation timeH Output: The cloudlets scheduling policy ( 1 ) Get the migration time t; ( 2 )  Get the number of idle VMs on each cloudlets ( 3 )  whilet<H  do ( 4 ) Get RP from Algorithm 4 ( 5 )   ifRP < utilization threshold level then ( 6 )   Update the migration time t as the next finish time of the VMs. ( 7 )   Get migration compositions MCh by Algorithm 3 ( 8 )   Schedule the VMs by the policy ( 9 )   Set the vacant cloudlets to the sleeping mode ( 10 )  end if ( 11 )  end whileAs we aim to schedule the VMs with the goals of energy saving and privacy preservation, the VM migration composition with highest utilization will be chosen as the ultimate VM scheduling policy. And after VM scheduling for the identified VMs, the vacant PMs should be set to the sleeping mode to save the energy. Besides, through the Algorithm2, we can also attain another purpose for reducing the possibility of privacy leaks.The migration times are necessary to be obtained for global VM scheduling according to the utilization threshold. Besides, the number of the idle VMs on each cloudlet is a key measure to decide the destination cloudlet of the migrations. During the execution period of all the mobile applications in WMAN, the resource utilizationRPof the cloudlets is obtained by Algorithm 4. Then, if the currentRP is less than utilization threshold, that is, there are enough space in the active cloudlets after the previous round of scheduling, we need to schedule the VMs by the policy and set the vacant cloudlets to the sleeping mode according toMCh by Algorithm 3. Thus, we achieve the goal of dynamic scheduling.Algorithm5 specifies the process of global VM scheduling (t). Firstly, we update theRPandMChof the migration time (Lines (1) to (6)). Then schedule the running VMs and set the vacant cloudlets to the sleeping mode (Lines (8) to (10)).


## 4. Experiment Evaluation

In this section, a series of comprehensive experiments are operated to evaluate the performance of our proposed energy-efficient cloudlet management method for privacy preservation in WMAN. For comparison analysis, the running state without migration is marked as a benchmark is employed to verify the effectiveness of our proposed ECM method.


### 4.1. Experimental Context

In our experiment, we select the specified type of cloudlet servers to create our simulated cloud infrastructure services net and the basic hardware parameter are as follows. Its configuration consists of Intel Xeon 3040, dual-Processor clocked at 1860 MHz, and 4 GB of RAM. Its baseline power consists of two parts, the PM part and the VM part. The power consumption of the PM, active VM, and idle VM are 86W, 6W, and 4W, respectively.Six basic parameters are initialized as records in our experiment. The value domain of each parameter is specified in Table2. In other words, 4 different-scale datasets are generated, respectively, for our experiment with the number of running tasks 500, 600, 700, and 800. Each data record in these datasets is a tuple with 6 attributes. For example, there is a data record (6, 18, 2, 0, 3, and 2), where “6” is the sequence number of the hosted task, “18” is the cloudlet sequence number, “2” is the number of VMs that this hosted task simultaneously occupies, “0” is the VM start time, “3” is the VM duration time, and “2” is the total amount of data for this hosted task.Table 2
Parameter settings.
Parameter Item Domain Number of running PMs { 500 ,  600 ,  700 ,  800 } Number of running VMs in each PM [ 1,3 ] Number of VMs on each PM 6 Data transmission rate (Mb/s) [ 270,540 ] VM duration time { 1,3 } VM transmission data (G) [ 0.5,0.8 ]In our experiment, there are two rounds of multiple VM migration, and we use time instants before migration to distinguish different round of VM scheduling. The two time instants are recorded as 1 and 2 in the experimental section, respectively.


### 4.2. Performance Evaluation

In this section, evaluations on energy consumption, resource utilization, and transmission delay are discussed to validate our proposed method.


#### 4.2.1. Performance Evaluation on Energy Consumption

Our energy consumption evaluation is based on the energy consumption of the running cloudlets and VMs. The number of active employed cloudlets is also significant to the total energy consumption. We used four different sets of data to compare energy consumption in the case of benchmark and ECM. And we use two time instants to compare the employed number of different time instants in the process of dynamic scheduling. From Figure2, it can be concluded that, with the ECM method, the number of running cloudlets is much less than that in the benchmark case in both two time instants. Since the idle cloudlet can be directly considered to be completely closed, its energy consumption can be ignored in our experiment when setting them to the sleeping mode. Thus, less active cloudlet can generate less energy. The energy consumption here is divided into three parts, the idle VMs, the active VMs, and the energy consumed by active cloudlets themselves.Figure 2
Comparison of the number of employed cloudlets for different time instants with Benchmark and ECM.
(a)
Time instant = 1 (b)
Time instant = 2After two rounds of VM migrations, the total energy consumption can be calculated respectively. Figure3 indicates these energy values in the case of benchmark and ECM. Table 3 shows the results of improvements of energy consumption with ECM compared to benchmark. From the analysis of Figure 3 and Table 3, using our proposed method ECM saves more energy than using benchmark.Table 3
Improvements of energy consumption with ECM compared to benchmark.
Number of servers ECM 500 600 700 800 54.4% 54.1% 41.9% 52.1%Figure 3
Comparison of total energy consumption with Benchmark, ECM.


#### 4.2.2. Performance Evaluation on Resource Utilization

Resource utilization efficiency for two different time instants is also one of the dimensions that measure the value of our proposed method. Resource utilization changes dynamically in different rounds of VM migrations because when the resource utilization of one cloudlet is low, the VMs in this cloudlet should be migrated to other cloudlets, and then it could be shut down to save energy. It refers to the relationship between VMs and active cloudlets. Similar to the energy consumption evaluated above, resource utilization is also tested by four different datasets. The ECM and benchmark are compared with these four datasets, and the simulated resource utilization result is illustrated in Figure4.Figure 4
Comparison of resource utilization for different time instants with Benchmark, ECM.
(a)
Time instant = 1 (b)
Time instant = 2As can be shown in the chart, after VM migrations, our proposed method ECM has better resource utilization than benchmark. And, with the increasing times of VM scheduling, the resource utilization for cloudlets is getting higher. Since ECM considers the energy consumption by cloudlets, some VMs would not be migrated to other cloudlets and the cloudlet hosting it still keeps in active mode, which would raise the resource utilization. It shows that, with our method for energy saving, the resource utilization becomes higher than the benchmark.


#### 4.2.3. Performance Evaluation on Transmission Delay

Figure5 illustrates the observation results of the amount of the migrations after VM scheduling. According to the system model, the transmission delay is determined based on the amount of data and the data transmission rate. At the same time, the number of VM migrations is also one of the important factors of total transmission delay. Compared to benchmark, our proposed method ECM uses a large amount times of VM migrations to achieve the goal of saving energy which would generate high transmission delay.Figure 5
Number of migrations with ECM.Furthermore, to make the evaluation on performance of our proposed ECM method more clearly, the effect of VM migrations on transmission delay is fully investigated in detail. The transmission delay of our proposed ECM method is based on four different data sets and two time instants for simulation evaluation, respectively.The result which illustrates transmission delay time caused by the VM migrations is evaluated. From Figure6, simulation results can be intuitively achieved that the transmission delay based on the ECM method fluctuates around 10 minutes in the simulation environment. The result shows that there is still a lot of room for improvement in handling challenges of high transmission delay.Figure 6
Transmission delay time of ECM.Besides, from the results of evaluations on VM migrations, we can achieve that using the shortest time identification can indeed reduce the number of migrations; in other words, we can also achieve the goal of reduces the possibility of privacy leaks.


## 4.1. Experimental Context

In our experiment, we select the specified type of cloudlet servers to create our simulated cloud infrastructure services net and the basic hardware parameter are as follows. Its configuration consists of Intel Xeon 3040, dual-Processor clocked at 1860 MHz, and 4 GB of RAM. Its baseline power consists of two parts, the PM part and the VM part. The power consumption of the PM, active VM, and idle VM are 86W, 6W, and 4W, respectively.Six basic parameters are initialized as records in our experiment. The value domain of each parameter is specified in Table2. In other words, 4 different-scale datasets are generated, respectively, for our experiment with the number of running tasks 500, 600, 700, and 800. Each data record in these datasets is a tuple with 6 attributes. For example, there is a data record (6, 18, 2, 0, 3, and 2), where “6” is the sequence number of the hosted task, “18” is the cloudlet sequence number, “2” is the number of VMs that this hosted task simultaneously occupies, “0” is the VM start time, “3” is the VM duration time, and “2” is the total amount of data for this hosted task.Table 2
Parameter settings.
Parameter Item Domain Number of running PMs { 500 ,  600 ,  700 ,  800 } Number of running VMs in each PM [ 1,3 ] Number of VMs on each PM 6 Data transmission rate (Mb/s) [ 270,540 ] VM duration time { 1,3 } VM transmission data (G) [ 0.5,0.8 ]In our experiment, there are two rounds of multiple VM migration, and we use time instants before migration to distinguish different round of VM scheduling. The two time instants are recorded as 1 and 2 in the experimental section, respectively.


## 4.2. Performance Evaluation

In this section, evaluations on energy consumption, resource utilization, and transmission delay are discussed to validate our proposed method.


### 4.2.1. Performance Evaluation on Energy Consumption

Our energy consumption evaluation is based on the energy consumption of the running cloudlets and VMs. The number of active employed cloudlets is also significant to the total energy consumption. We used four different sets of data to compare energy consumption in the case of benchmark and ECM. And we use two time instants to compare the employed number of different time instants in the process of dynamic scheduling. From Figure2, it can be concluded that, with the ECM method, the number of running cloudlets is much less than that in the benchmark case in both two time instants. Since the idle cloudlet can be directly considered to be completely closed, its energy consumption can be ignored in our experiment when setting them to the sleeping mode. Thus, less active cloudlet can generate less energy. The energy consumption here is divided into three parts, the idle VMs, the active VMs, and the energy consumed by active cloudlets themselves.Figure 2
Comparison of the number of employed cloudlets for different time instants with Benchmark and ECM.
(a)
Time instant = 1 (b)
Time instant = 2After two rounds of VM migrations, the total energy consumption can be calculated respectively. Figure3 indicates these energy values in the case of benchmark and ECM. Table 3 shows the results of improvements of energy consumption with ECM compared to benchmark. From the analysis of Figure 3 and Table 3, using our proposed method ECM saves more energy than using benchmark.Table 3
Improvements of energy consumption with ECM compared to benchmark.
Number of servers ECM 500 600 700 800 54.4% 54.1% 41.9% 52.1%Figure 3
Comparison of total energy consumption with Benchmark, ECM.


### 4.2.2. Performance Evaluation on Resource Utilization

Resource utilization efficiency for two different time instants is also one of the dimensions that measure the value of our proposed method. Resource utilization changes dynamically in different rounds of VM migrations because when the resource utilization of one cloudlet is low, the VMs in this cloudlet should be migrated to other cloudlets, and then it could be shut down to save energy. It refers to the relationship between VMs and active cloudlets. Similar to the energy consumption evaluated above, resource utilization is also tested by four different datasets. The ECM and benchmark are compared with these four datasets, and the simulated resource utilization result is illustrated in Figure4.Figure 4
Comparison of resource utilization for different time instants with Benchmark, ECM.
(a)
Time instant = 1 (b)
Time instant = 2As can be shown in the chart, after VM migrations, our proposed method ECM has better resource utilization than benchmark. And, with the increasing times of VM scheduling, the resource utilization for cloudlets is getting higher. Since ECM considers the energy consumption by cloudlets, some VMs would not be migrated to other cloudlets and the cloudlet hosting it still keeps in active mode, which would raise the resource utilization. It shows that, with our method for energy saving, the resource utilization becomes higher than the benchmark.


### 4.2.3. Performance Evaluation on Transmission Delay

Figure5 illustrates the observation results of the amount of the migrations after VM scheduling. According to the system model, the transmission delay is determined based on the amount of data and the data transmission rate. At the same time, the number of VM migrations is also one of the important factors of total transmission delay. Compared to benchmark, our proposed method ECM uses a large amount times of VM migrations to achieve the goal of saving energy which would generate high transmission delay.Figure 5
Number of migrations with ECM.Furthermore, to make the evaluation on performance of our proposed ECM method more clearly, the effect of VM migrations on transmission delay is fully investigated in detail. The transmission delay of our proposed ECM method is based on four different data sets and two time instants for simulation evaluation, respectively.The result which illustrates transmission delay time caused by the VM migrations is evaluated. From Figure6, simulation results can be intuitively achieved that the transmission delay based on the ECM method fluctuates around 10 minutes in the simulation environment. The result shows that there is still a lot of room for improvement in handling challenges of high transmission delay.Figure 6
Transmission delay time of ECM.Besides, from the results of evaluations on VM migrations, we can achieve that using the shortest time identification can indeed reduce the number of migrations; in other words, we can also achieve the goal of reduces the possibility of privacy leaks.


## 4.2.1. Performance Evaluation on Energy Consumption

Our energy consumption evaluation is based on the energy consumption of the running cloudlets and VMs. The number of active employed cloudlets is also significant to the total energy consumption. We used four different sets of data to compare energy consumption in the case of benchmark and ECM. And we use two time instants to compare the employed number of different time instants in the process of dynamic scheduling. From Figure2, it can be concluded that, with the ECM method, the number of running cloudlets is much less than that in the benchmark case in both two time instants. Since the idle cloudlet can be directly considered to be completely closed, its energy consumption can be ignored in our experiment when setting them to the sleeping mode. Thus, less active cloudlet can generate less energy. The energy consumption here is divided into three parts, the idle VMs, the active VMs, and the energy consumed by active cloudlets themselves.Figure 2
Comparison of the number of employed cloudlets for different time instants with Benchmark and ECM.
(a)
Time instant = 1 (b)
Time instant = 2After two rounds of VM migrations, the total energy consumption can be calculated respectively. Figure3 indicates these energy values in the case of benchmark and ECM. Table 3 shows the results of improvements of energy consumption with ECM compared to benchmark. From the analysis of Figure 3 and Table 3, using our proposed method ECM saves more energy than using benchmark.Table 3
Improvements of energy consumption with ECM compared to benchmark.
Number of servers ECM 500 600 700 800 54.4% 54.1% 41.9% 52.1%Figure 3
Comparison of total energy consumption with Benchmark, ECM.


## 4.2.2. Performance Evaluation on Resource Utilization

Resource utilization efficiency for two different time instants is also one of the dimensions that measure the value of our proposed method. Resource utilization changes dynamically in different rounds of VM migrations because when the resource utilization of one cloudlet is low, the VMs in this cloudlet should be migrated to other cloudlets, and then it could be shut down to save energy. It refers to the relationship between VMs and active cloudlets. Similar to the energy consumption evaluated above, resource utilization is also tested by four different datasets. The ECM and benchmark are compared with these four datasets, and the simulated resource utilization result is illustrated in Figure4.Figure 4
Comparison of resource utilization for different time instants with Benchmark, ECM.
(a)
Time instant = 1 (b)
Time instant = 2As can be shown in the chart, after VM migrations, our proposed method ECM has better resource utilization than benchmark. And, with the increasing times of VM scheduling, the resource utilization for cloudlets is getting higher. Since ECM considers the energy consumption by cloudlets, some VMs would not be migrated to other cloudlets and the cloudlet hosting it still keeps in active mode, which would raise the resource utilization. It shows that, with our method for energy saving, the resource utilization becomes higher than the benchmark.


## 4.2.3. Performance Evaluation on Transmission Delay

Figure5 illustrates the observation results of the amount of the migrations after VM scheduling. According to the system model, the transmission delay is determined based on the amount of data and the data transmission rate. At the same time, the number of VM migrations is also one of the important factors of total transmission delay. Compared to benchmark, our proposed method ECM uses a large amount times of VM migrations to achieve the goal of saving energy which would generate high transmission delay.Figure 5
Number of migrations with ECM.Furthermore, to make the evaluation on performance of our proposed ECM method more clearly, the effect of VM migrations on transmission delay is fully investigated in detail. The transmission delay of our proposed ECM method is based on four different data sets and two time instants for simulation evaluation, respectively.The result which illustrates transmission delay time caused by the VM migrations is evaluated. From Figure6, simulation results can be intuitively achieved that the transmission delay based on the ECM method fluctuates around 10 minutes in the simulation environment. The result shows that there is still a lot of room for improvement in handling challenges of high transmission delay.Figure 6
Transmission delay time of ECM.Besides, from the results of evaluations on VM migrations, we can achieve that using the shortest time identification can indeed reduce the number of migrations; in other words, we can also achieve the goal of reduces the possibility of privacy leaks.


## 5. Related Work

Recently, the emergence and development of WMAN aim to meet the growing market requirements of wireless broadband. An increasing number of mobile users in urban cities have tendency to accomplish the execution of various tasks in clouds due to the abundant resources, but one significant limitation of offloading tasks to the remote clouds is the long physical distance between mobile users and remote cloud [21–23]. To reduce such delay of offloaded tasks to remote clouds, cloudlets are deployed in WMAN to support mobile devices, by executing offloaded tasks on local cloudlets.Ma et al. [24] realized the placement of cloudlets in WMAN is important and proposed a New Heuristic Algorithm (NHA) to reduce the sorting process of APs running time to solve the delay problem. Xiang et al. [25] proposed an adaptive cloud placement method for GPS big data mobile applications that adjusts the placement of the cloud by identifying the aggregated area of mobile devices. Mike Jia et al. [26] proposed an algorithm of allocating the requirements of mobile users to suitable cloudlets deployed in WMAN to implement load balancing to solve the delay problem and discussed how their algorithm can be practically applied to WMANs with dynamic and constantly moving users.Currently, live VM migration techniques significantly make contribution to manage resource in cloudlet, thus improving energy efficiency and effectively lower the delay in cloudlets as much as possible via providing high quality of resources for execution of tasks and migrating VMs to idle cloudlets to reduce task queuing time.Decreasing the delay in cloudlets could be divided into two aspects: decreasing transmission delay between cloudlets and decreasing processing delay in cloudlets [27]. But Rodrigues et al. [28] reduced the two kinds of delay at the same time by utilizing a mathematical model with a Particle Swarm Optimization (PSO) model. Dhanoa et al. [29] quantified the energy consumption and time in the VM migration process and mathematically modeled them and then discovered that the main factors of energy consumption during VM migration are the size of the VMs and the network bandwidth. Xu et al. [30] proposed a VM scheduling method to achieve balance between energy and performance in cyberphysical cloud systems and the method determines which VMs and where should be migrated. Shaw et al. [31] proposed a method to save energy by using a VM migration dynamic merge to reduce the number of running physical machines and the authors designed an algorithm to determine if the VM really needs to continue running or if it needs to be migrated.The optimization problem of energy consumption for cloudlet management could be solved by the heuristic methods and the convex optimization technique [32, 33]. Besides, along with the other scheduling objectives, the game theory with Nash Equilibria is employed to realize the competition usage of the cloudlet resources [34, 35]. However, some cloudlet services need to access mobile users’ personal private information. When transmitting the mobile applications of the mobile users, personal privacy is easily leaked through some APs with poor security, which may cause some bad events and even hurt the personal safety of mobile users. Therefore, privacy preservation could be treated as a constraint for the optimization problem in cloudlet management [36–39].Maria et al. [40] proposed a conceptual framework in Ambient Intelligent focused in cloud computing that includes the various privacy policies and privacy issues. Ji et al. [41] implemented a hybrid privacy protection solution with KP-ABE and CP-ABE, whose advantage is balancing the performance and security in cloud platforms. Wang et al. [42] developed a method to protect privacy by dividing user’s private information into several parts and then this system saves the data in different cloud servers and user’s fog devices. Moreover, when the data itself is insecure, which may include trust spoof attacks and replicated-sink attacks, the data collection needs to identify whether such data could be received. Wang et al. [43] proposed a trust evaluation scheme to ensure the trustworthiness of sensors and mobile sinks, which protects the system from trust spoof attacks and replicated-sink attacks.The above analysis concludes that live VMs migration technique has not been well used in the cloudlets researches and few studies have considered privacy preservation as much as possible while reducing energy consumption about cloudlets deployed in WMAN.


## 6. Conclusion and Future Work

The utilization of cloudlet in WMAN is an important technology which extends the performance of mobile devices. As WMAN continues to develop, cloudlets which are public and easy to access would be vital to mobile cloud computing.In this paper, we proposed an energy-efficient cloudlet management for privacy preservation in WMAN abbreviated as ECM taking advantage of the live VM migration techniques. We implemented an effective algorithm that determines the place to which VMs should be migrated. Our goal is to reduce the energy consumption of cloudlets and privacy preservation in WMAN. Experimental evaluations have been conducted to validate the efficiency and effectiveness of our proposed method. As this paper mainly focused on the energy saving and privacy in WMAN, there are also many important issues which need to be investigated in future. For example, in the people-intensive areas, there are large amounts of mobile applications that need to be offloaded to the cloudlets for execution; how to reduce the energy consumption and achieve load balancing of the cloudlets in WMAN remains challenging. When we only pay attention to energy saving, the communication delay would be high, which affects the experience of mobile users.For future work, we try to take cloudlet load balancing and energy consumption of cloudlets into consideration at the same time. Furthermore, we will design a corresponding method for trade-offs between energy consumption and load balancing.



---

*Source: 8180451-2018-09-24.xml*