# An Adaptive Deployment Algorithm for IaaS Cloud Virtual Machines Based onQ Learning Mechanism

Authors: Shuguang Chen
Journal: Journal of Sensors
 (2021)

---

## Abstract

When deploying infrastructure as a service (IaaS) cloud virtual machines using the existing algorithms, the deployment process cannot be simplified, and the algorithm is difficult to be applied. This leads to the problems of high energy consumption, high number of migrations, and high average service-level agreement (SLA) violation rate. In order to solve the above problems, an adaptive deployment algorithm for IaaS cloud virtual machines based onQ learning mechanism is proposed in this research. Based on the deployment principle, the deployment characteristics of the IaaS cloud virtual machines are analyzed. The virtual machine scheduling problem is replaced with the Markov process. The multistep Q learning algorithm is used to schedule the virtual machines based on the Q learning mechanism to complete the adaptive deployment of the IaaS cloud virtual machines. Experimental results show that the proposed algorithm has low energy consumption, small number of migrations, and low average SLA violation rate.


---

## Body

## 1. Introduction

In recent years, the development scale of service computing model has gradually expanded and has been widely used in various fields. The rapid progress of distributed computing, virtualization, and parallel computing technology has promoted the development of cloud technology [1, 2]. At the service model level, cloud computing is usually divided into software as a service (SaaS), platform as a service (PaaS), and IaaS. IaaS belongs to the basic level in cloud computing services and can provide resources and storage resources in the network. Therefore, it is of great significance to study the algorithm of deploying IaaS cloud virtual machines [3].Lei and Wang [4] combined the best adaptation algorithm and hierarchical clustering algorithm to optimize the efficiency. On this basis, the local search algorithm was used to optimize the deployment of the virtual machines’ locations. The deployment process of the algorithm was relatively cumbersome, leading to the high energy consumption of the algorithm. Liao et al. [5] combined dynamic programming method and Markov models to predict the price of virtual machines and deployed virtual machines in a cloud environment according to the execution time limit of the workflow. However, this algorithm could not simplify the virtual machine deployment process. The number of migrations of virtual machines was relatively high. Jiliang et al. [6] established a multiservice-desk queuing model. The system energy saving rate and the average stay time of cloud user requests were obtained through the matrix geometry method and virtual birth and death process. The virtual machines were scheduled to realize deployment. The average SLA violation rate of this method was high, and the effectiveness of the algorithm was poor.In order to solve the problems in the above algorithms, an adaptive IaaS cloud virtual machine deployment algorithm based on theQ learning mechanism is proposed.The rest of this paper is organized as follows. Section2 gives the basic issues of virtual machine deployment and an adaptive deployment algorithm for IaaS cloud virtual machines. The experiment is provided in Section 3 to illustrate the effectiveness of the proposed method. Finally, Section 4 concludes this paper.


## 2. Related Works

### 2.1. Basic Issues of Virtual Machine Deployment

The problem of the self-adaptive deployment of IaaS cloud virtual machines can be described as follows.n virtual machines constitute a virtual machine set [7, 8], which is denoted as V=v1,v2,⋯,vn, where vi=pi,Qi,ri is a virtual machine. pi is a virtual machine type; Qi=ci,mi,si,bi is a virtual machine configuration. ci is the number of CPUs in the virtual machine, mi is the memory of the virtual machine, bi represents the expected bandwidth, and si represents the hard disk space; ri represents the amount of data required to build the virtual machines [9]. A host machine set is composed of m host machines, denoted as H=h1,h2,⋯,hm, where hj=pj,Qj,dj represents a host machine. pj is a virtualization platform existing in the host machine, Qj=Cj,Mj,Sj means the hardware configuration of the host machine, Cj means the number of CPUs in the host machine, Sj means the hard disk space of the host machine, and Mj means the memory of the host machine. A mapping f:V⟶H is constructed, where f:vi⟶hj is the deployment of the virtual machine vi in the host machine hj.


#### 2.1.1. Deployment Principles

The virtual machines run on the basis of the host’s hardware resources. The main hardware resources include hard disk, memory, and CPU.After fully testing the applications existing in the virtual machines, the resource usage can be set. However, it is usually difficult to predict the behaviors of IaaS users. In order to improve the quality of service, the deployment of IaaS cloud virtual machines needs to be implemented on the basis of the following principles [10, 11]:
(a)
The number of CPU cores in the host machine for all virtual machines should be lower than the number of cores in the actual host machine for CPU(b)
The memory of the host machines should be lower than the virtual memory of the virtual machines in the host machines


#### 2.1.2. Deployment Characteristics

Regarding the deployment of virtual machines as an independent issue, its deployment has the following characteristics:(a)
Resources are limited during task execution. When the traditional method is to schedule independent tasks, any number of tasks can be allocated in a certain resource, but when deploying virtual machines, it is necessary to provide sufficient resources for the host machines [12, 13](b)
When the task is scheduled to the resource, the task can be expanded and processed


#### 2.1.3. Problem Description

In the virtual machine deployment algorithm, suitable host machineshj should be found for all virtual machines vi to optimize the target. Task scheduling goals can generally be divided into the following two categories:
(a)
Optimize the period of tasks by considering the interests of resource users(b)
Optimize the specific conditions of the resources in the use process by considering the interests of resource providersIntroducing virtualization technology into the server can improve resource utilization. The main task of the IaaS cloud virtual machine deployment algorithm is to use the least host machines to complete the deployment of the virtual machines, regardless of deployment time, bandwidth, and hard disk space. There is a clear corresponding relationship between the type of virtual machines and the manager of the virtual machine of the hosts [14, 15], so the IaaS cloud virtual machine deployment problem can be simplified as follows.For the virtual machine setV=v1,v2,⋯,vn, vi=ci,mi, and the host machine set H=h1,h2,⋯,hm, hj=Cj,Mj, we obtain a subset H′=h1′,h2′,⋯,h∣H∣′′⊂H, hj′=Cj′,Mj′, of the host machine set H and establish a mapping f:V⟶H′ that satisfies
(1)f:V⟶H′s.t.∑vj⟶hj′ci≤Cj′∑vj⟶hj′mi≤Mj′,where Cj′ represents the number of CPUs in the host machine after the mapping process and Mj′ represents the memory of the host machine after the mapping process.


### 2.2. An Adaptive Deployment Algorithm for IaaS Cloud Virtual Machines

According to the above analysis of the deployment problem of IaaS cloud virtual machines, the adaptive deployment algorithm for IaaS cloud virtual machines based on theQ learning mechanism uses the Markov decision process to describe the deployment problem of IaaS cloud virtual machines. The essence of IaaS cloud virtual machine deployment is to schedule virtual machines in the cloud environment to achieve adaptive deployment.


#### 2.2.1.Q Learning Method

TheQ learning algorithm can solve the Markov decision process, which belongs to the reinforcement learning algorithm [16, 17]. The Markov model can be described with five tuples <S,A,R,P,V>, where S is the state set, A represents the action set, R is return function, P represents the state transition probability matrix, and V is the model function value.Markov’s strategy can be described by the mapping of the probability distribution in the action space and the state space [18, 19], expressed as π:S⟶∏A, where ∏A is the probability distribution corresponding to the action set A.In the state-action space, the value function in the strategy can be calculated by(2)Qπx,a=Rxa+∑y∈Spxya×Vπy,where Rxa represents the return obtained when the action a is transited under the state x, pxya represents the corresponding probability when the action a transits from state x to state y, and Vπy represents the expected total discounted return corresponding to state y under the strategy π.Using the optimalQ value to obtain the optimal strategy π∗ is the principle of the Q learning mechanism, and Q∗x,a is used to describe the optimal Q value. If a∗=argmaxaQ∗x,a, the optimal strategy at this time can be described by π∗x=a∗.TheQ learning process is usually composed of two parts, namely, plot and step. The plot is a sequence of steps between the target state and the starting state; the step is the return acquisition and action execution in the determined state.It is supposed thatxt represents the state corresponding to the agent at the time t, and at represents the execution of the action. We observe the return rt and the next state xt+1 during the learning process and update the value function:
(3)Qt+1x,a=1−αtQtx,a+αtrt+Vtxt+1,x=xt,a=at,Qtx,a,other,where the parameter Vtxt+1=maxbQtxt+1,b and αt represents the learning rate.


#### 2.2.2. Markov Description of Task Scheduling

When using reinforcement learning to solve the virtual machine scheduling problem, it is necessary to use the Markov model to replace the virtual machine scheduling problem. Markov elements can be selected according to the following principles:(a)
The state at the next moment is not affected by the historical state and is only related to the current state. The selected state needs to reflect Markov properties [20, 21](b)
When choosing an action, it is necessary to ensure that there is a law of state transition, all states can be reached, and the goal of the agent is completed by performing a series of actions(c)
The selected return function requires the least decision-making steps to make the agent reach the target stateThe Markov process is described on the basis of the above principles:(a)
The unit-task matching matrix is used to describe the Markov state. In the virtual machine scheduling process, the state of the matrix is the scheduling’s feasible solution, and there is a state transition of reinforcement learning in the feasible solution space [22, 23](b)
In all states, there aren tasks and n actions. The jth (1<j≤n) task can be transited to the next design unit through the jth action. At this time, the result obtained by executing the jth action can be expressed by dkj=1, and the parameter k can be calculated by(4)k=i+1,i<m,1,i=m.(c)
Determine the return value throughrx,a,x′=1/tx′ in the reinforcement learning


#### 2.2.3. MultistepQ Learning Algorithm

In theQ learning process, the value function Qxt,at is updated through rt+γVtxt+1, and the return rt is determined by
(5)rtn=rt+γrt+1+⋯+γn−1rt+n−1+γnVt+n−1xt+n,where n represents the number of iterations, γ represents the learning coefficient, and rtn represents the truncated return corresponding to the correction step n at a time t.The weighted average of the truncated returnrtn is used to update, and the learning steps are simplified. The weighted average rrλ,k of truncated return rtn can be calculated by k step time difference TDλ.
(6)rrλ,k=1−λ∑i=1k−1λi−1rti+λk−1rtk,where λ represents the time difference.rtn is substituted in the weighted average rrλ,k to simplify the processing:
(7)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i.Equation (7) is converted to
(8)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i+∑i=1k−1λγiVt+ixt+i−Vt+i−1xt+i.When the learning rateα is low, the adjustment speed of the Q value is slow. At this time, the speed of the V value is also reduced. Let et=rt+γVtxt+1−Vtxt, et′=rt+γVtxt+1−Qtxt,at, and we have
(9)rtλ,k=et′+∑i=1k−1λγiet+i+Qtxt,at.Equation (9) is converted to
(10)rtλ,k−Qtxt,at=et′+∑i=1k−1λγiet+i.Through the above process, the update rules of theQ learning mechanism are obtained:
(11)Qt+1x,a=Qtx,a+αtet′+∑i=1k−1λγiet+i,x=xt,a=at,Qt=x,a,other.The virtual machines are scheduled according to the above rules to realize the adaptive deployment of IaaS cloud virtual machines.


## 2.1. Basic Issues of Virtual Machine Deployment

The problem of the self-adaptive deployment of IaaS cloud virtual machines can be described as follows.n virtual machines constitute a virtual machine set [7, 8], which is denoted as V=v1,v2,⋯,vn, where vi=pi,Qi,ri is a virtual machine. pi is a virtual machine type; Qi=ci,mi,si,bi is a virtual machine configuration. ci is the number of CPUs in the virtual machine, mi is the memory of the virtual machine, bi represents the expected bandwidth, and si represents the hard disk space; ri represents the amount of data required to build the virtual machines [9]. A host machine set is composed of m host machines, denoted as H=h1,h2,⋯,hm, where hj=pj,Qj,dj represents a host machine. pj is a virtualization platform existing in the host machine, Qj=Cj,Mj,Sj means the hardware configuration of the host machine, Cj means the number of CPUs in the host machine, Sj means the hard disk space of the host machine, and Mj means the memory of the host machine. A mapping f:V⟶H is constructed, where f:vi⟶hj is the deployment of the virtual machine vi in the host machine hj.


### 2.1.1. Deployment Principles

The virtual machines run on the basis of the host’s hardware resources. The main hardware resources include hard disk, memory, and CPU.After fully testing the applications existing in the virtual machines, the resource usage can be set. However, it is usually difficult to predict the behaviors of IaaS users. In order to improve the quality of service, the deployment of IaaS cloud virtual machines needs to be implemented on the basis of the following principles [10, 11]:
(a)
The number of CPU cores in the host machine for all virtual machines should be lower than the number of cores in the actual host machine for CPU(b)
The memory of the host machines should be lower than the virtual memory of the virtual machines in the host machines


### 2.1.2. Deployment Characteristics

Regarding the deployment of virtual machines as an independent issue, its deployment has the following characteristics:(a)
Resources are limited during task execution. When the traditional method is to schedule independent tasks, any number of tasks can be allocated in a certain resource, but when deploying virtual machines, it is necessary to provide sufficient resources for the host machines [12, 13](b)
When the task is scheduled to the resource, the task can be expanded and processed


### 2.1.3. Problem Description

In the virtual machine deployment algorithm, suitable host machineshj should be found for all virtual machines vi to optimize the target. Task scheduling goals can generally be divided into the following two categories:
(a)
Optimize the period of tasks by considering the interests of resource users(b)
Optimize the specific conditions of the resources in the use process by considering the interests of resource providersIntroducing virtualization technology into the server can improve resource utilization. The main task of the IaaS cloud virtual machine deployment algorithm is to use the least host machines to complete the deployment of the virtual machines, regardless of deployment time, bandwidth, and hard disk space. There is a clear corresponding relationship between the type of virtual machines and the manager of the virtual machine of the hosts [14, 15], so the IaaS cloud virtual machine deployment problem can be simplified as follows.For the virtual machine setV=v1,v2,⋯,vn, vi=ci,mi, and the host machine set H=h1,h2,⋯,hm, hj=Cj,Mj, we obtain a subset H′=h1′,h2′,⋯,h∣H∣′′⊂H, hj′=Cj′,Mj′, of the host machine set H and establish a mapping f:V⟶H′ that satisfies
(1)f:V⟶H′s.t.∑vj⟶hj′ci≤Cj′∑vj⟶hj′mi≤Mj′,where Cj′ represents the number of CPUs in the host machine after the mapping process and Mj′ represents the memory of the host machine after the mapping process.


## 2.1.1. Deployment Principles

The virtual machines run on the basis of the host’s hardware resources. The main hardware resources include hard disk, memory, and CPU.After fully testing the applications existing in the virtual machines, the resource usage can be set. However, it is usually difficult to predict the behaviors of IaaS users. In order to improve the quality of service, the deployment of IaaS cloud virtual machines needs to be implemented on the basis of the following principles [10, 11]:
(a)
The number of CPU cores in the host machine for all virtual machines should be lower than the number of cores in the actual host machine for CPU(b)
The memory of the host machines should be lower than the virtual memory of the virtual machines in the host machines


## 2.1.2. Deployment Characteristics

Regarding the deployment of virtual machines as an independent issue, its deployment has the following characteristics:(a)
Resources are limited during task execution. When the traditional method is to schedule independent tasks, any number of tasks can be allocated in a certain resource, but when deploying virtual machines, it is necessary to provide sufficient resources for the host machines [12, 13](b)
When the task is scheduled to the resource, the task can be expanded and processed


## 2.1.3. Problem Description

In the virtual machine deployment algorithm, suitable host machineshj should be found for all virtual machines vi to optimize the target. Task scheduling goals can generally be divided into the following two categories:
(a)
Optimize the period of tasks by considering the interests of resource users(b)
Optimize the specific conditions of the resources in the use process by considering the interests of resource providersIntroducing virtualization technology into the server can improve resource utilization. The main task of the IaaS cloud virtual machine deployment algorithm is to use the least host machines to complete the deployment of the virtual machines, regardless of deployment time, bandwidth, and hard disk space. There is a clear corresponding relationship between the type of virtual machines and the manager of the virtual machine of the hosts [14, 15], so the IaaS cloud virtual machine deployment problem can be simplified as follows.For the virtual machine setV=v1,v2,⋯,vn, vi=ci,mi, and the host machine set H=h1,h2,⋯,hm, hj=Cj,Mj, we obtain a subset H′=h1′,h2′,⋯,h∣H∣′′⊂H, hj′=Cj′,Mj′, of the host machine set H and establish a mapping f:V⟶H′ that satisfies
(1)f:V⟶H′s.t.∑vj⟶hj′ci≤Cj′∑vj⟶hj′mi≤Mj′,where Cj′ represents the number of CPUs in the host machine after the mapping process and Mj′ represents the memory of the host machine after the mapping process.


## 2.2. An Adaptive Deployment Algorithm for IaaS Cloud Virtual Machines

According to the above analysis of the deployment problem of IaaS cloud virtual machines, the adaptive deployment algorithm for IaaS cloud virtual machines based on theQ learning mechanism uses the Markov decision process to describe the deployment problem of IaaS cloud virtual machines. The essence of IaaS cloud virtual machine deployment is to schedule virtual machines in the cloud environment to achieve adaptive deployment.


### 2.2.1.Q Learning Method

TheQ learning algorithm can solve the Markov decision process, which belongs to the reinforcement learning algorithm [16, 17]. The Markov model can be described with five tuples <S,A,R,P,V>, where S is the state set, A represents the action set, R is return function, P represents the state transition probability matrix, and V is the model function value.Markov’s strategy can be described by the mapping of the probability distribution in the action space and the state space [18, 19], expressed as π:S⟶∏A, where ∏A is the probability distribution corresponding to the action set A.In the state-action space, the value function in the strategy can be calculated by(2)Qπx,a=Rxa+∑y∈Spxya×Vπy,where Rxa represents the return obtained when the action a is transited under the state x, pxya represents the corresponding probability when the action a transits from state x to state y, and Vπy represents the expected total discounted return corresponding to state y under the strategy π.Using the optimalQ value to obtain the optimal strategy π∗ is the principle of the Q learning mechanism, and Q∗x,a is used to describe the optimal Q value. If a∗=argmaxaQ∗x,a, the optimal strategy at this time can be described by π∗x=a∗.TheQ learning process is usually composed of two parts, namely, plot and step. The plot is a sequence of steps between the target state and the starting state; the step is the return acquisition and action execution in the determined state.It is supposed thatxt represents the state corresponding to the agent at the time t, and at represents the execution of the action. We observe the return rt and the next state xt+1 during the learning process and update the value function:
(3)Qt+1x,a=1−αtQtx,a+αtrt+Vtxt+1,x=xt,a=at,Qtx,a,other,where the parameter Vtxt+1=maxbQtxt+1,b and αt represents the learning rate.


### 2.2.2. Markov Description of Task Scheduling

When using reinforcement learning to solve the virtual machine scheduling problem, it is necessary to use the Markov model to replace the virtual machine scheduling problem. Markov elements can be selected according to the following principles:(a)
The state at the next moment is not affected by the historical state and is only related to the current state. The selected state needs to reflect Markov properties [20, 21](b)
When choosing an action, it is necessary to ensure that there is a law of state transition, all states can be reached, and the goal of the agent is completed by performing a series of actions(c)
The selected return function requires the least decision-making steps to make the agent reach the target stateThe Markov process is described on the basis of the above principles:(a)
The unit-task matching matrix is used to describe the Markov state. In the virtual machine scheduling process, the state of the matrix is the scheduling’s feasible solution, and there is a state transition of reinforcement learning in the feasible solution space [22, 23](b)
In all states, there aren tasks and n actions. The jth (1<j≤n) task can be transited to the next design unit through the jth action. At this time, the result obtained by executing the jth action can be expressed by dkj=1, and the parameter k can be calculated by(4)k=i+1,i<m,1,i=m.(c)
Determine the return value throughrx,a,x′=1/tx′ in the reinforcement learning


### 2.2.3. MultistepQ Learning Algorithm

In theQ learning process, the value function Qxt,at is updated through rt+γVtxt+1, and the return rt is determined by
(5)rtn=rt+γrt+1+⋯+γn−1rt+n−1+γnVt+n−1xt+n,where n represents the number of iterations, γ represents the learning coefficient, and rtn represents the truncated return corresponding to the correction step n at a time t.The weighted average of the truncated returnrtn is used to update, and the learning steps are simplified. The weighted average rrλ,k of truncated return rtn can be calculated by k step time difference TDλ.
(6)rrλ,k=1−λ∑i=1k−1λi−1rti+λk−1rtk,where λ represents the time difference.rtn is substituted in the weighted average rrλ,k to simplify the processing:
(7)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i.Equation (7) is converted to
(8)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i+∑i=1k−1λγiVt+ixt+i−Vt+i−1xt+i.When the learning rateα is low, the adjustment speed of the Q value is slow. At this time, the speed of the V value is also reduced. Let et=rt+γVtxt+1−Vtxt, et′=rt+γVtxt+1−Qtxt,at, and we have
(9)rtλ,k=et′+∑i=1k−1λγiet+i+Qtxt,at.Equation (9) is converted to
(10)rtλ,k−Qtxt,at=et′+∑i=1k−1λγiet+i.Through the above process, the update rules of theQ learning mechanism are obtained:
(11)Qt+1x,a=Qtx,a+αtet′+∑i=1k−1λγiet+i,x=xt,a=at,Qt=x,a,other.The virtual machines are scheduled according to the above rules to realize the adaptive deployment of IaaS cloud virtual machines.


## 2.2.1.Q Learning Method

TheQ learning algorithm can solve the Markov decision process, which belongs to the reinforcement learning algorithm [16, 17]. The Markov model can be described with five tuples <S,A,R,P,V>, where S is the state set, A represents the action set, R is return function, P represents the state transition probability matrix, and V is the model function value.Markov’s strategy can be described by the mapping of the probability distribution in the action space and the state space [18, 19], expressed as π:S⟶∏A, where ∏A is the probability distribution corresponding to the action set A.In the state-action space, the value function in the strategy can be calculated by(2)Qπx,a=Rxa+∑y∈Spxya×Vπy,where Rxa represents the return obtained when the action a is transited under the state x, pxya represents the corresponding probability when the action a transits from state x to state y, and Vπy represents the expected total discounted return corresponding to state y under the strategy π.Using the optimalQ value to obtain the optimal strategy π∗ is the principle of the Q learning mechanism, and Q∗x,a is used to describe the optimal Q value. If a∗=argmaxaQ∗x,a, the optimal strategy at this time can be described by π∗x=a∗.TheQ learning process is usually composed of two parts, namely, plot and step. The plot is a sequence of steps between the target state and the starting state; the step is the return acquisition and action execution in the determined state.It is supposed thatxt represents the state corresponding to the agent at the time t, and at represents the execution of the action. We observe the return rt and the next state xt+1 during the learning process and update the value function:
(3)Qt+1x,a=1−αtQtx,a+αtrt+Vtxt+1,x=xt,a=at,Qtx,a,other,where the parameter Vtxt+1=maxbQtxt+1,b and αt represents the learning rate.


## 2.2.2. Markov Description of Task Scheduling

When using reinforcement learning to solve the virtual machine scheduling problem, it is necessary to use the Markov model to replace the virtual machine scheduling problem. Markov elements can be selected according to the following principles:(a)
The state at the next moment is not affected by the historical state and is only related to the current state. The selected state needs to reflect Markov properties [20, 21](b)
When choosing an action, it is necessary to ensure that there is a law of state transition, all states can be reached, and the goal of the agent is completed by performing a series of actions(c)
The selected return function requires the least decision-making steps to make the agent reach the target stateThe Markov process is described on the basis of the above principles:(a)
The unit-task matching matrix is used to describe the Markov state. In the virtual machine scheduling process, the state of the matrix is the scheduling’s feasible solution, and there is a state transition of reinforcement learning in the feasible solution space [22, 23](b)
In all states, there aren tasks and n actions. The jth (1<j≤n) task can be transited to the next design unit through the jth action. At this time, the result obtained by executing the jth action can be expressed by dkj=1, and the parameter k can be calculated by(4)k=i+1,i<m,1,i=m.(c)
Determine the return value throughrx,a,x′=1/tx′ in the reinforcement learning


## 2.2.3. MultistepQ Learning Algorithm

In theQ learning process, the value function Qxt,at is updated through rt+γVtxt+1, and the return rt is determined by
(5)rtn=rt+γrt+1+⋯+γn−1rt+n−1+γnVt+n−1xt+n,where n represents the number of iterations, γ represents the learning coefficient, and rtn represents the truncated return corresponding to the correction step n at a time t.The weighted average of the truncated returnrtn is used to update, and the learning steps are simplified. The weighted average rrλ,k of truncated return rtn can be calculated by k step time difference TDλ.
(6)rrλ,k=1−λ∑i=1k−1λi−1rti+λk−1rtk,where λ represents the time difference.rtn is substituted in the weighted average rrλ,k to simplify the processing:
(7)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i.Equation (7) is converted to
(8)rrλ,k=rt+γVtxt+1+∑i=1k−1λγirt+1+γVt+ixt+i+1−Vt+i−1xt+i+∑i=1k−1λγiVt+ixt+i−Vt+i−1xt+i.When the learning rateα is low, the adjustment speed of the Q value is slow. At this time, the speed of the V value is also reduced. Let et=rt+γVtxt+1−Vtxt, et′=rt+γVtxt+1−Qtxt,at, and we have
(9)rtλ,k=et′+∑i=1k−1λγiet+i+Qtxt,at.Equation (9) is converted to
(10)rtλ,k−Qtxt,at=et′+∑i=1k−1λγiet+i.Through the above process, the update rules of theQ learning mechanism are obtained:
(11)Qt+1x,a=Qtx,a+αtet′+∑i=1k−1λγiet+i,x=xt,a=at,Qt=x,a,other.The virtual machines are scheduled according to the above rules to realize the adaptive deployment of IaaS cloud virtual machines.


## 3. Experiments and Results

In order to verify the overall effectiveness of the adaptive deployment algorithm for IaaS cloud virtual machines based on theQ learning mechanism, it is necessary to conduct related tests on the proposed algorithm. The switch in this test was a three-layer switch, with a power consumption of 80 W, physical machine power consumption of 750 W, and a network bandwidth capacity of 100 M.The proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm were used for comparison in the following tests.


### 3.1. Energy Consumption Test

The energy consumption of different algorithms in network congestion and noncongestion conditions was tested. The test results are shown in Figure1.Figure 1
Comparison of energy consumption by different algorithms. (a) Comparison of energy consumption without network congestion. (b) Comparison of energy consumption with network congestion.
(a)(b)From Figure1, the energy consumption of the proposed algorithm did not change in the states with and without network congestion and was lower than that of Reference [4] algorithm and Reference [5] algorithm. The energy consumption of Reference [4] algorithm and Reference [5] algorithm in the state with network congestion was higher than that in the state without network congestion. Through the above analysis, it could be seen that the energy consumption of the proposed algorithm was not affected by network congestion. The reasons why the proposed algorithm simplified the learning process through a multistep Q learning algorithm based on the Q learning mechanism, and scheduling virtual machines improved the rationality of virtual machine deployment and reduced the energy consumption.


### 3.2. Analysis of the Number of Migrations

The number of migrations of the virtual machines of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm was compared under different resource utilization efficiency thresholds. The analysis results are shown in Figure 2.Figure 2
Comparison of the number of migrations by different algorithms.From Figure2, it shows that when increasing the resource utilization efficiency threshold, the number of migrations of the three algorithms gradually increased. Through comparison, it could be seen that the number of migrations of the proposed algorithm was the smallest, indicating that the proposed algorithm completed the deployment of the IaaS cloud virtual machines through the least migrations.


### 3.3. Average SLA Violation Rate

The average SLA violation rate was used as a test indicator to compare the deployment effectiveness of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm. The lower the average SLA violation rate, the better the deployment effect of the algorithm. The average SLA violation rate can be calculated by
(12)SLAviolation=Nq−NfNq,where Nq represents the processing capacity of all requests and Nf represents the processing capacity corresponding to the allocated requests.The average SLA violation rates of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm are shown in Table 1.Table 1
The average SLA violation rates by different algorithms.
Resource utilization efficiency threshold (%)Average SLA violation rate (%)Proposed algorithmReference [4] algorithmReference [3] algorithm1004.23.92005.64.83005.95.74006.56.75006.87.5605.07.58.8705.08.09.4805.08.49.8905.08.910.41005.09.511.7From Table1, it indicates that the average SLA violation rate for the proposed algorithm did not change after the resource utilization efficiency threshold reached 60%. For Reference [4] algorithm and Reference [5] algorithm, the obtained average SLA violation rate increased with the resource utilization efficiency threshold increasing. Through analysis, it could be seen that the average SLA violation rate of the proposed algorithm was the lowest, indicating the best deployment effect by the proposed algorithm.


## 3.1. Energy Consumption Test

The energy consumption of different algorithms in network congestion and noncongestion conditions was tested. The test results are shown in Figure1.Figure 1
Comparison of energy consumption by different algorithms. (a) Comparison of energy consumption without network congestion. (b) Comparison of energy consumption with network congestion.
(a)(b)From Figure1, the energy consumption of the proposed algorithm did not change in the states with and without network congestion and was lower than that of Reference [4] algorithm and Reference [5] algorithm. The energy consumption of Reference [4] algorithm and Reference [5] algorithm in the state with network congestion was higher than that in the state without network congestion. Through the above analysis, it could be seen that the energy consumption of the proposed algorithm was not affected by network congestion. The reasons why the proposed algorithm simplified the learning process through a multistep Q learning algorithm based on the Q learning mechanism, and scheduling virtual machines improved the rationality of virtual machine deployment and reduced the energy consumption.


## 3.2. Analysis of the Number of Migrations

The number of migrations of the virtual machines of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm was compared under different resource utilization efficiency thresholds. The analysis results are shown in Figure 2.Figure 2
Comparison of the number of migrations by different algorithms.From Figure2, it shows that when increasing the resource utilization efficiency threshold, the number of migrations of the three algorithms gradually increased. Through comparison, it could be seen that the number of migrations of the proposed algorithm was the smallest, indicating that the proposed algorithm completed the deployment of the IaaS cloud virtual machines through the least migrations.


## 3.3. Average SLA Violation Rate

The average SLA violation rate was used as a test indicator to compare the deployment effectiveness of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm. The lower the average SLA violation rate, the better the deployment effect of the algorithm. The average SLA violation rate can be calculated by
(12)SLAviolation=Nq−NfNq,where Nq represents the processing capacity of all requests and Nf represents the processing capacity corresponding to the allocated requests.The average SLA violation rates of the proposed algorithm, Reference [4] algorithm, and Reference [5] algorithm are shown in Table 1.Table 1
The average SLA violation rates by different algorithms.
Resource utilization efficiency threshold (%)Average SLA violation rate (%)Proposed algorithmReference [4] algorithmReference [3] algorithm1004.23.92005.64.83005.95.74006.56.75006.87.5605.07.58.8705.08.09.4805.08.49.8905.08.910.41005.09.511.7From Table1, it indicates that the average SLA violation rate for the proposed algorithm did not change after the resource utilization efficiency threshold reached 60%. For Reference [4] algorithm and Reference [5] algorithm, the obtained average SLA violation rate increased with the resource utilization efficiency threshold increasing. Through analysis, it could be seen that the average SLA violation rate of the proposed algorithm was the lowest, indicating the best deployment effect by the proposed algorithm.


## 4. Conclusion

Various cloud service providers have begun to establish big data centers in the context of the continuous reduction of communication traffic costs and the gradual increase of network bandwidth. The energy consumption of big data centers is high. Therefore, enterprises and governments need to construct environmentally friendly and energy-saving cloud data centers. Deploying virtual machines can improve service quality and reduce energy consumption.At present, the virtual machine deployment algorithm has the problems of high energy consumption, high number of migrations, and high average SLA violation rate. An adaptive deployment algorithm for IaaS cloud virtual machines based on theQ learning mechanism is proposed. First, the deployment problem of the IaaS cloud virtual machines is analyzed. Then, the Q learning mechanism is used to realize the adaptive deployment of IaaS cloud virtual machines, reducing the energy consumption, number of migrations, and average SLA violation rate. The future work is to improve the running speed of the algorithm while ensuring the accuracy.



---

*Source: 2119993-2021-10-31.xml*