## 7 Conclusions  
In this paper, we solve the problem of task offloading in order to decrease the delay for offloaded tasks and increase the utilization rate of physical machine in the cloud. Different from the traditional \(Q\)-learning algorithm, the DRL can be suited to the problem with high-dimension state space. Moreover, in order to improve the speed of learning optimal policy, we propose a novel TLRL algorithm for task offloading, where the k-NN algorithm is applied to divide the PMs into several clusters. With the reduced dimension of state space and action space, the DRL layer aims at learning the optimal policy to choose a cluster. Then, the \(Q\)-learning layer learns an optimal policy to select the optimal PM to execute the current offloaded task. The experiments show that the TLRL algorithm is faster than the DRL algorithm when learning the optimal policy for task offloading. By adjusting the weight factor \(\beta\), our proposed algorithms for task offloading can trade off between utilization rate of physical machine and delay effectively.  
Figure 10: (**a–e**) shows the change for the utilization rate of physical machines (PMs) with the weight \(\beta\) increasing from 0.1 to 0.9. (**f–j**) shows the change of delay for the current offloaded task with the weight \(1-\beta\) decreasing from 0.9 to 0.1 correspondingly.  
Our proposed algorithms intend to find the optimal PM for executing offloaded tasks by considering utilization rate of PM and delay for task offloading simultaneously. We simulate 100 PMs with different bandwidths and six different target applications to verify our proposed algorithms. Moreover, this paper is mainly focused on developing novel RL-based algorithms to solve the offloading problem. We verify the effectiveness of the proposed algorithms in theory, which can trade off between utilization rate of PM and delay effectively by designing a weighted reward. In the future study, we will verify our proposed algorithms in a real environment where the running environments of offloaded tasks are deployed on PMs. Besides, simulated tasks will be replaced by the real applications running on smartphones. Moreover, Different tasks have different attributes, which also cloud have different delay requirement. However, we mainly focus on the effective tradeoff between the utilization rate of PM and delay and try to decrease the delay as possible ignoring the delay requirement for different tasks. Take task 1 for example, if it requires that the delay cannot exceed a given time \(t\), then the algorithm could choose a \(\beta\) that makes the utilization rate of PM larger before meeting its delay requirement. Therefore, we can find the relation between the value of \(\beta\) and the delay requirement for different tasks through analyzing the obtained real data. Therefore, we will research on choosing the adaptive value \(\beta\) according to offloading tasks conditions by using the prediction model in our future work.  
The work presented in this paper represents a collaborative effort by all authors, whereas [PERSON] wrote the main paper. [PERSON] and [PERSON] discussed the proposed main problem and algorithms. Both authors have read and approved the final manuscript.  
This paper is supported by National Natural Science Foundation of China (Key Project, No. 61432004): Mental Health Cognition and Computing Based on Emotional Interactions. National Key Research and Development Program of China (No. 2016 YFB1001404): Multimodal Data Interaction Intention Understanding in Cloud Fusion. National Key Research and Development Program of China (No. 2017 YFB1002804) and National Key Research and Development Program of China (No. 2017 YFB1401200).  
The authors declare no conflict of interest.