## 5 Proposed Methods  
### Deep Reinforcement Learning for Task Offloading  
Task offloading in MCC environment tries to offload computing-intensive or energy-consuming tasks to cloud servers. As described in Section 3, we should select the optimal PM to execute offloaded tasks by trading off the utilization rate of PM and delay. Existing machine learning methods [15; 16; 17] for task offloading belong to supervised learning. They need to learn the optimal policy through amount of historical data, in which the selection of historical data will affect the effectiveness of the corresponding algorithm. In order to learn the optimal task offloading policy from real-time data, we mode this problem using DRL in which action space is the set of PMs in the cloud. This method can update policy constantly according to real-time data until it converges. Usually, these PMs are distributed geographically and large scale in order to meet the needs of a large number of devices. This can lead to a large action space. We define state space according to the waiting time and the number of VMs that run in PMs that are introduced in Section 3. Suppose that, there are \(P\) PMs in cloud for task offloading and the \(p\)th PM is denoted as \(PM_{p}\). The number of VMs that run in the \(p\)th PM is \(N_{c}^{p}\) at current decision step \(t\). We use \(T_{w}^{p}\) to denote the waiting time that a task is offloaded to the \(p\)th PM in the cloud. Therefore, we could define the state for task offloading as follow:  
\[\text{state}:\ S_{t}<T_{w}^{1},N_{c}^{1},T_{w}^{2},N_{c}^{2}\cdots T_{w}^{p},N _{c}^{P}> \tag{9}\]  
The dimension of state in our proposed problem is \(2*P\), where \(P\) is always lager in real cloud infrastructure. Therefore, our proposed problem has a high-dimension state space. We define the reward value according to the utilization rate of PM and delay which are also the goal of optimization. However, the highest utilization rate of PM and the lowest delay often cannot be met at the same time. If the action \(PM_{p}\) is selected and the task is offloaded to the \(p\)th PM correspondingly, then the reward is defined as follow:  
\[R=\beta*UR+(1-\beta)TD=\beta\big{(}N_{CV}/N_{TV}*100\%\big{)}+(1-\beta)Normal (1/T_{c}+T_{W}+S/BW) \tag{10}\]The first part is current utilization rate of PM in the cloud, and the second part is the delay caused by task offloading. Then \(\beta(0\leq\ \beta\leq 1)\) is a weight factor that is used to trade off the utilization rate of PM and delay. If \(\beta\) is larger, it means that the utilization rate of PM is preferred than the delay and vice versa. In order to trade off between \(UR\) and \(TD\) efficiently by adjusting the weight factor \(\beta\), we need to normalize the two metric. The Min-Max normalization is used to the two metric. Because the value of \(UR\) is between 0 and 1, we only normalize the \(TD\) in general. In Formula (10), function \(Normal\) represents the normalization of the \(TD\). In DRL, before the process of online learning, we construct the DNN by learning from the related training data or initializing parameters randomly. The construction and updating of the DNN are based on experience replay [26; 27]. The experience M consists of a tuple \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\) at each time step \(t\) in this paper. The input of DNN is the state of PMs in cloud according to Formula (9) and the output of DNN is the corresponding \(Q\) value for selecting each PMs. Therefore, the dimension of input is \(2*P\), and the dimension of output is \(P\). The Figure 5 shows the process of our proposed task offloading algorithm based on DRL that focuses on the online \(Q\) learning. When the offloaded task arrives at time step \(t\), the algorithm obtains current states \(s_{t}\) and decides how to select the action \(a_{t}\) that represents the PM in cloud that the offloaded task executes on.