## 5 Proposed Methods
### Two-Layered Reinforcement Learning for Task Offloading  
In this subsection, we propose a two-layered reinforcement learning (TLRL) algorithm for task offloading to address above problem. First, we divide the PMs into \(K\) clusters according to the bandwidth between PMs and devices using the k-NN algorithm where the \(k\)th cluster contains \(P_{k}\) PMs. The structure diagram of proposed TLRL algorithm is shown as Figure 6.  
Our proposed two-layered reinforcement learning structure comprises of DRL and RL. Different from DRL for task offloading, TLRL gives a current optimal cluster of PMs \(C_{k}^{p}\) according to the current state first by using DRL first. Then the optimal PM \(PM_{p}\) is obtained by using \(Q\)-Learning for the cluster \(C_{k}^{p}\). The detailed flow of task offloading algorithm based on TLRL is shown in Figure 7.  
The first layer of TLRL is implemented by DRL, which is used to get an optimal cluster for offloaded tasks. We define the waiting time of the \(k\)th cluster as:  
\[T_{cw}^{k}=\min_{p_{k\in\{1,\ldots,P_{k}\}}}\left(T_{iw}^{p_{k}}\right) \tag{11}\]  
where \(T_{iw}^{p_{k}}\) is the waiting time of the \(p_{k}\)th PM. The total number of VMs that run in the \(k\)th cluster is \(N_{cc}^{k}\) at decision \(t\). The state for clusters is denoted as follow:  
\[\text{state}:\ S_{t}^{c}<T_{cw}^{1},N_{cc}^{1},T_{cw}^{2},N_{cc}^{2}\cdots T_{ cw}^{K},N_{cc}^{K}> \tag{12}\]  
and  
\[\text{action}:\ A^{c}=\left\{a^{c}\Big{|}a^{c}\in\left\{C_{1}^{p},C_{2}^{p},C_{ 3}^{p}\ldots C_{K}^{p}\right\}\right\} \tag{13}\]  
where \(S_{t}^{c}\) represents the state of layer one and A\({}^{c}\) is its action space. Then we define the reward value when the action \(C_{k}^{p}\) is selected and the task is offloaded to the \(k\)th cluster:  
\[ \begin{split}\mathbb{R}^{c}&=\beta*UR+(1-\beta)TD= \beta(N_{cv}/N_{tv}*100\%)\\ &\quad\quad+(1-\beta)Normal(1/T_{c}+T_{cw}^{4}+S/\text{SW}_{t}^{4}) \end{split}  \tag{14}\]  
Figure 6: The structure diagram of two-layered reinforcement learning (TLRL) algorithm. where the bandwidth of the \(k\)th cluster is denoted as \(BW_{c}^{k}\), which is equal to the bandwidth of center PM of cluster. The evaluation function corresponds to \(R^{c}\) is denoted as \(Q^{c}(s_{t}^{c},a^{c})\), which can be updated according to Formula (5). The second layer of TLRL is implemented by \(Q\)-learning, which is used to get an optimal PM for offloaded tasks. We need to build a \(Q\)-learning model for each cluster separately. After the DRL of TLRL selects the \(thk\) cluster, the RL of TLRL will select a PM for executing current offloaded task further by using \(Q\)-learning in the \(thk\) cluster. In the \(thk\) cluster, we define the total number of VMs that run in the \(p_{k}\)th PM is \(N_{c}^{p_{k}}\) at episode \(t\). The state for clusters is denoted as follow:  
\[\text{state}:\ S_{t}^{k}<T_{w}^{1},N_{c}^{1},T_{w}^{2},N_{c}^{2}\cdots T_{w}^{ p_{k}},N_{c}^{p_{k}}> \tag{15}\]  
Figure 7: Flow diagram of task offloading algorithm based on TLRL. \[\text{action}:A^{k}=\left\{a^{k}\Big{|}a^{k}\in\left\{PM_{1}^{k},PM_{2}^{k},PM_{3}^{k }\ldots PM_{P_{k}}^{k}\right\}\right\} \tag{16}\]  
where \(S_{t}^{k}\) represents the state of layer two and \(A^{k}\) is its action space. The reward is denoted as:  
\[R^{k}=\alpha*UR+(1-\alpha)TD=\alpha\big{(}N_{c}^{k}/N_{c}^{k}*100\%\big{)}+(1 -\alpha)(T_{c}+T_{w}^{p_{k}}+S/BW_{k}^{p_{k}}) \tag{17}\]  
where \(N_{c}^{k}\) is max number of VMs that run in the \(k\)th cluster. Similarly, the evaluation function corresponds to \(R^{k}\) is denoted as \(Q^{k}\big{(}s_{t}^{k},a^{k}\big{)}\) which can be updated according to Formula (5). Besides, we define the global \(Q\)-function for TLRL algorithm given by Formula (18), which indicates the convergence of Algorithm 2.