# A Novel Deep Reinforcement Learning Approach for Task Offloading in MEC Systems
## 4 Tasking Offloading Algorithm Based on DRL
### DRL-Based Task Offloading  
In the task-offloading decision, the agent interacts with the environment to estimate the value function through the resulting sample and learn an optimal strategy \(\pi^{*}\). Both the state and action space of the system in our paper have high dimensionality and are dynamic and discrete; they have to be optimized while generating sequences. Based on the above factors, we choose the DQN-based method to optimize our decision-making algorithm. Generally speaking, DRL uses the \(\epsilon\)-greedy strategy to balance exploration and exploitation, where \(\epsilon\) refers to the probability for exploring randomly and exploiting with a chance of \(1-\epsilon\). This stochastic policy can ensure the tradeoff between exploration and exploitation, but it has to explore for a long-term extension and causes low resource utilization. In this paper, we use a novel DQN-based approach by taking advantage of a parameterized indexed function for efficient exploration. In the proposed framework, we parameterize the state-action value function as:  
\[Q(s,a)=v(s,a)+m(s,a)Z, \tag{12}\]  
where \(Z\sim N(0,1)\). Note that there are two parts for Q value function, so we utilize a dual-network consisting of a mean network for learning, \(v(s,a)\), and an uncertainty network for learning, \(m(s,a)\). Let \(\theta=(\psi,\omega)\) be the parameters of the trainable networks, where \(\psi\omega\) are the parameters of the mean and the uncertainty network. Let \(\tilde{\theta}=(\tilde{\psi},\tilde{\omega})\) be the target networks' parameters. The overall training progress and task-offloading strategy are shown in Figure 2 and Algorithm 1. A prior mechanism is also incorporated, where each trainable network is paired with an additive prior network, and it is suited to sequential decision problems. The prior networks share the same architecture as the trainable networks, let \(\overline{\theta}=(\overline{\psi},\overline{\omega})\) be its parameters and \(\beta\) is the prior scale of the network. Thus, the mean network parameters are updated through the target mean network and prior network, as follows:  
\[\
abla_{\psi}L_{v}(\psi,\tilde{\psi},\overline{\psi},D)= \tag{13}\] \[\frac{1}{|D|}\sum_{(s_{i},a_{i},r_{i},s^{{}^{\prime}}_{i},done) \in D}\
abla_{\psi}((v_{\psi}+\beta_{1}\overline{\eta}_{\overline{\psi}})(s_{ i},a_{i})-\tilde{Q}_{\psi,\tilde{\psi},\overline{\psi}})^{2},\]  
where \(\tilde{Q}_{\psi,\tilde{\psi},\overline{\psi}}\) is \(r_{i}+(1-done)\gamma(v_{\tilde{\psi}}+\beta_{1}\overline{\eta}_{\overline{ \psi}})(s_{i}^{{}^{\prime}},\overline{a}_{i})\). Furthermore, the uncertainty network parameters are updated as follows:  
\[ \begin{split}&\
abla_{\omega}L_{m}(\omega,\tilde{\omega}, \overline{\omega},D)=\\ &\frac{1}{|D|}\sum_{(s_{i},a_{i},r_{i},s^{{}^{\prime}}_{i},done) \in D}\
abla_{\omega}((m_{\omega}+\beta_{1}\overline{\eta}_{\overline{\omega} })(s_{i},a_{i})-\tilde{Q}_{\omega,\tilde{\omega},\overline{\omega}})^{2}, \end{split}  \tag{14}\]  
where \(\tilde{Q}_{\omega,\tilde{\omega},\overline{\omega}}\) is \(\sigma+(1-done)\gamma(m_{\tilde{\omega}}+\beta_{1}\overline{\eta}_{\overline{ \omega}})(s_{i}^{{}^{\prime}},\overline{a}_{i})\).