## 6 Simulation Study
### The Comparison of Different Algorithms  
In this experiment, we compare our proposed TLRL algorithm and DRL algorithm with the random policy for task offloading. The random algorithm arranges the offloaded tasks to the PM randomly and ignores the utilization rate of PMs and delay. The beginning state is initialized randomly, and \(\beta=0.5\) is selected for the weight factor of reward. We observe the change of utilization rate of PMs and delay through simulating offloading tasks, where the number of offloaded tasks increases from 1 to 200. We will compare the obtained optimal policy after learning process for DRL and TLRL with a random policy. We track the utilization rate of PMs and delay for 200 offloaded tasks by following the learned optimal policy for task offloading. These results are shown in Figure 8. It is shown that the random algorithm results in severe fluctuation, in terms of the utilization rate of PMs and delay, compared to the cases of DRL and TLRL. First, in Figure 7(a), both the DRL and TLRL, the utilization rate of PMs is maintained at the level around 60%, whereas the random algorithm leads to an inferior level around 23% for the utilization rate of PMs, which may lead to the waste of cloud resources. Second, in Figure 7(b), it can be seen that based on DRL and TLRL, a lower delay for each offloaded task can be obtained. It changes between 9 s and 15 s. However, frequent changes in the delay for the offloaded task by using the random algorithm, where the max delay is around 27 s. This is because the random approach selects a PM to execute the offloaded task randomly. According to the definition of delay in Formula (3), if \(T_{W}\) is larger and \(BW\) is smaller, it will lead to a higher delay \(TD\) correspondingly. On the contrary, if \(T_{W}\) is smaller and \(BW\) is larger, a lower delay \(TD\) is obtained. Therefore, this random selection approach for PMs can lead to a \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Offloaded Tasks** & **Amount of Data** & **Time of Executing Task** \\ \hline \(Task_{1}\) & 1 MB & 2 s \\ \(Task_{2}\) & 3 MB & 5 s \\ \(Task_{3}\) & 5 MB & 3 s \\ \(Task_{4}\) & 7 MB & 4 s \\ \(Task_{5}\) & 9 MB & 6 s \\ \(Task_{6}\) & 10 MB & 5 s \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The parameters of offloaded tasks in following experiments. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Algorithms** & **Number of Input** & \begin{tabular}{c} **Number of Neurons in** \\ **First Hidden Layer** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Neurons in** \\ **Second Hidden Layer** \\ \end{tabular} &
\begin{tabular}{c} **Number of** \\ **Output** \\ \end{tabular} \\ \hline DRL & 200 & 114 & 36 & 100 \\ TLRL & 40 & 18 & 4 & 20 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The setting of CNN in following experiments. random delay in each decision step, which may cause the high jitter in the delay for offloaded tasks. We can conclude that our proposed algorithms based on DRL can get a better performance for increasing the utilization rate of PMs and decreasing the delay caused by task offloading than random policy. From the above results, one can observe that the policy learned by DRL can achieve better results than the policy learned by TLRL. This is because we reduce both the dimension of state and action for decreasing the computational complex by proposing the TLRL algorithm, where we assume the PMs from the same cluster are similar. Then we assume the waiting time and bandwidth of PMs from the same cluster are equal to the cluster according to Section 5, which may lead to a bias of learning optimal policy. However, our proposed TLRL algorithm could find the optimal policy than other methods when facing to high-dimension state space and action space. To discuss above problem, we start from a randomly initialized state for DRL, TLRL, and \(Q\)-learning. We compare convergence speed of the three algorithms for task offloading.