## 5 Two-Stage Iterative Algorithm Design
### Stage 1: Optimizing Task Offloading Decision
#### 5.1.2 DQN Update  
DQN consists of an experience replay memory \(\mathcal{D}\) and two deep neural networks (DNNs). The replay memory buffer \(\phi\) is used to store observed experiences \(\left(st^{l},a^{l},r^{l},st^{l+1}\right)\). The DNN includes an evaluation network and a target network, where \(Q^{real}\) and \(Q^{lat}\) represent their Q-values under state \(st^{l}\), respectively. These two networks are trained using experiences from the replay memory with parameter vectors \(\omega\) and \(\omega^{-}\), respectively.  
At each step, the current state \(st^{l}\) is fed into the evaluation network to obtain \(Q^{real}(st^{l},a^{l};\omega)\). The action can be selected using the \(\epsilon\)-greedy algorithm, either choosing the optimal action given by \(a^{l}=\arg\max_{a^{l}}Q^{real}(st^{l},a^{l},\omega)\) or a random action with probability \(\epsilon\). After choosing the action \(a^{l}\), we obtain the next state \(st^{l+1}\). However, due to the queuing process in the computational model, the reward \(r^{l}\) for action \(a^{l}\) taken in state \(st^{l}\) can only be obtained after actions have been selected for all states. These experiences are stored in the replay memory, from which random batches of data will be sampled to train the DNN. After training, the \(Q^{real}(st^{l},a^{l};\omega)\) will approach the \(Q^{lat}(st^{l},a^{l};\omega^{-})\), which is given by  
\[Q^{lat}(st^{l},a^{l};\omega^{-})=r+\gamma\max_{a^{l+1}}Q^{\prime}(st^{l+1},a^{ l+1};\omega^{-}), \tag{10}\]  
where \(r\) is the reward for choosing \(a^{l}\) in the state \(st^{l}\), \(Q^{\prime}(st^{l+1},a^{l+1};\omega^{-})\) is used to approximate the Q-value of taking action \(a^{l+1}\) in state \(st^{l+1}\), and \(\gamma\) is the discount factor. The loss function is established according to the mean squared error (MSE) of the \(Q^{real}\) and the \(Q^{lat}\), which is given by  
\[L(\omega)=\mathbb{E}[(Q^{lat}(st^{l},a^{l};\omega^{-})-Q^{real}(st^{l},a^{l}; \omega))^{2}]. \tag{11}\]  
To minimize the difference between the \(Q^{real}\) and the \(Q^{lat}\), the network weight parameter \(\omega\) is updated using the gradient descent method, which is represented as  
\[\
abla_{\omega}L(\omega)=\mathbb{E}\Big{[}(Q^{lat}(st^{l},a^{l};\omega^{-})- Q^{real}(st^{l},a^{l};\omega))\Big{]}\times\
abla_{\omega}Q^{real}(s^{l},a^{l}; \omega). \tag{12}\]  
In addition, the initial parameters of the evaluation network and the target network are the same value. The parameters \(\omega\) of the evaluation network are updated at each step, while the parameters \(\omega^{-}\) of the target network are updated every fixed \(\kappa\) step. Figure 3 shows the detailed DQN algorithm.  
In summary, the details of the proposed DQN-based task offloading algorithm are described in Algorithm 1.  
Figure 3: Framework of the DQN algorithm.  
```
1: Initialize replay memory \(\mathcal{D}\) to capacity \(\phi\);
2: Initialize evaluation network with random weights \(\omega\);
3: Initialize target network with weights \(\omega^{-}=\omega\);
4:forepisode \(=1,2,3\ldots\)do
5:for\(l=1,2,3\ldots\)do
6: Observe current system state \(st^{l}\);
7: With probability \(\varepsilon\) select random action \(a^{l}_{j}\);
8: Otherwise, select \(a^{l}=\text{argmax}_{a}Q^{equil}(st^{l},a^{l};\omega)\);
9:endfor
10:for\(l=1,2,3\ldots\)do
11: Execute action \(a^{l}\) and observe reward \(r^{l}\) and \(st^{l+1}\);
12: Store transition \((st^{l},a^{l},r^{l},st^{l+1})\) in \(\mathcal{D}\);
13: Sample random minibatch of transitions \((st^{j},a^{l},r^{l},st^{l+1})\) form \(\mathcal{D}\);
14: Set \(Q^{tar}(st^{l+1},a^{l+1};\omega^{-})=r+\gamma\max\limits_{a^{(l+1)}}Q^{\prime }(st^{l+1},a^{l+1};\omega^{-})\);
15: Perform gradient descent to update the network parameters \(\omega\) by (12);
16: The target network parameter \(\omega^{-}\) is periodically replaced by the evaluation network parameter \(\omega\) when \(l\) is a multiple of \(\kappa\);
17:endfor
18:endfor
```  
**Algorithm 1** DQN-Based Task Offloading Algorithm