## 4 Model Solving
### Algorithm Design  
In this paper, we use the deep reinforcement learning algorithm DQN (Deep Q Network) and its variants based on the combination of reinforcement learning and deep learning to design a task offloading decision algorithm. A typical DQN model is composed of agent, state, action, and reward, and the policy is generated as a mapping \(\pi:S\to A\) of states to actions to obtain a reward \(R\), \(r_{t}(s_{t},a_{t})\) denotes the reward that can be obtained by choosing action \(a_{t}\) in state \(s_{t}\), and \(R_{0}^{\gamma}=\sum_{(t=0)}^{T}\gamma^{t}r_{t}(s_{t},a_{t})\) is the long-term reward, where \(T\rightarrow\infty\), \(\gamma\in[0,1]\), \(r_{t}\in R\), \(s_{t}\in S\), \(a_{t}\in A\). In traditional Q-learning, each state-action pair can be easily stored into the Q-table because the state space and action space are discrete and have small dimensions. However, when the state space and action space dimensions are large, as described in this paper, it is difficult to put all state-action pairs into Q-table. To solve this problem, the DQN model in DRL combines deep neural networks and Q-learning algorithms, and it transforms the Q-table tables into the Q-networks and uses neural networks to fit the optimal Q-functions. DQN relies mainly on the following key techniques. Deep Q Network (DQN): There are two neural networks with the same structure but different parameters in DQN, i.e., the target network \(\tilde{Q}(s^{\prime},a^{\prime};\tilde{\theta})\) and the main network \(Q(s,a;\theta)\). When iteratively updating the network, the algorithm first uses the target network to generate the target Q-value as the label \(f(t)\), and uses the loss function \(\text{Loss}(\mathbf{\theta})\) to update the parameters of the main network. After the introduction of the target network, the target Q value generated by the target network remains constant in time \(j\), which can reduce the correlation between the current Q value and the target Q value and improve the stability of the algorithm. The target network will update the parameters \(\tilde{\theta}\) with the Q network parameters \(\theta\) every certain step. \[f(t)=\left\{ \begin{array}{l}r(t),\text{ if episode terminates at step }j+1\\ r(t)+\gamma\max_{a^{\prime}}\{\tilde{Q}(s^{\prime},a^{\prime};\tilde{\theta}) \},\text{ otherwise}\end{array} \right. \tag{14}\]  
\[\text{Loss}(\mathbf{\theta})=(f(t)-Q(s,a;\theta))^{2} \tag{15}\]  
Replay Memory: In order to break the correlation within the data, DQN uses the experience replay method to solve this problem. After interacting with the environment, the agent is stored in the replay buffer in the form of \((s_{t},a_{t},r_{t},s_{t+1})\). When executing valuation updates, the agent randomly selects a small set of experience tuples \((s_{t},a_{t},r_{t},s_{t+1})\) from the replay buffer at each time step, and then the algorithm updates the network parameters by optimizing the loss function, using experience replay can not only make training more efficient, but also reduce the problem overfitting that generated by the training process. Double DQN: Double DQN is proposed to solve the overestimation problem. DQN takes the maximum value with max each time, and the difference between this maximum  
Figure 5: Illustration of offloading decision phase. value and the weighted average value introduces an error, which will lead to overestimation after a long time accumulation. The Double DQN is composed of two networks, \(Q^{A}\) and \(Q^{B}\), and it utilizes these two networks to proceed the state valuation and the action output alternatively. That is, one network is used to select out the action, and the other network is used to update the Q value according to the selected action. The Double DQN makes the learning process more stable and reliable by separating the two steps of selecting the action corresponding to the Q value and evaluating the Q value corresponding to the action, which eliminates the overestimation brought by the greedy algorithm and obtains a more accurate Q estimation.