# Dynamic Computation Offloading with Deep Reinforcement Learning in Edge Network
## 4 Algorithm
### DQN-Based Computation Offloading Algorithm
#### 4.2.2 Q-Network Update  
Traditional Q-learning is a model-free reinforcement learning that solves the Bellman equation using asynchronous iterations. In Q-learning, \(Q(s,a)\) represents the Q-value of a state-action pair \((s,a)\). The value of \(Q(s,a)\) can be considered as the expected payoff of the task, which takes action at state \(s\), i.e., an edge server is chosen as the execution location for that task. At state \(s\), the optimal policy can be formulated as:  
\[\pi^{*}(s)=\arg\max_{a\in\mathbb{A}}Q^{*}(s,a) \tag{17}\]  
where \(Q^{*}(s,a)\) is the optimal Q-value of the state-action pair \((s,a)\). Therefore, to obtain the optimal policy for current state \(s\), we need to calculate the Q-value for all actions \(a\in\mathbb{A}\). In the Q-learning algorithm, each \(Q(s,a)\) can be updated with the following learning rule:  
\[Q(s,a)=Q(s,a)+\alpha\big{(}R(s,a)+\gamma\max_{a^{\prime}\in\mathbb{A}}Q\big{(} s^{\prime},a^{\prime}\big{)}-Q(s,a)\big{)} \tag{18}\]  
where \(\alpha\in(0,1]\) is the learning rate and \(\gamma\) is the discount parameter. Meanwhile, in the deep Q-learning algorithm, the Q-values are abstractly stored in the DNN. The system's state changes in each time slot, and the DNN must be adaptively updated.  
Therefore, an essential part of the deep Q-learning algorithm is the DNN training based on the historical information stored in the experience replay memory \(\mathbb{M}\)[47]. In the \(t\)-th time slot, we randomly selected a batch of training data samples from \(\mathbb{M}\). The DNN with weights \(\theta\) was updated by applying the Adam algorithm [48] to reduce the averaged cross-entropy loss, as \[L(\theta_{t})=-\frac{1}{|\mathbb{M}|}\sum_{\tau}^{\mathbb{M}}\left((a_{\tau}^{*})^ {\intercal}\log f_{\theta_{t}}(s_{\tau})+(1-a_{\tau}^{*})^{\intercal}\log\left(1 -f_{\theta_{t}}(s_{\tau})\right)\right) \tag{19}\]  
where \(|\mathbb{M}|\) denotes the size of \(\mathbb{M}\). For simplicity, the detailed update procedure of the Adam algorithm is omitted. The use of historical data can reduce the variance in \(\theta_{t}\), while the random sampling approach reduces the correlation among the training data, thus accelerating the convergence of \(L(\theta)\). Meanwhile, as size of \(|\mathbb{M}|\) is fixed and the memory space \(\mathbb{M}\) is updated in each iteration, the DNN learns from the latest data samples generated by the recent strategies, which constantly improve the algorithm's decision. Overall, the detailed algorithm steps for the Q-network update are shown in Algorithm 2.  
```
0:\(s_{\tau},a_{\tau},r_{\tau},s_{\tau+1},\mathbb{M}\)
0:\(\theta\)
1: Store \((s_{\tau},a_{\tau},r_{\tau},s_{\tau+1})\) in \(\mathbb{M}[i\;mod\;|\mathbb{M}|]\)
2: Sample \(\mathbb{M}_{\infty}\subset\mathbb{M}\)
3:for\((s_{\tau-1}^{(j)},a_{\tau-1}^{(j)},r_{\tau-1}^{(j)},s_{\tau}^{(j)})\) in \(\mathbb{M}_{\infty}\)do
4: Calculate \(Q(s_{\tau}^{(j)},a_{\tau}^{(j)};\theta^{\prime})\)
5: Calculate target network's Q-value \(Q_{target}\)
6: Compute the error between \(Q(s_{\tau}^{(j)},a_{\tau}^{(j)};\theta^{\prime})\) and \(Q_{target}\)
7:endfor
8:\(\theta=argmin_{\theta}L(\theta)\)
9: Occasionally reset \(\theta^{{}^{\prime}}=\theta\)
10: Return \(\theta\)
```  
**Algorithm 2** Q-Network Update  
Based on the reinforcement settings and Q-network update, we present our deep reinforcement-learning-based dynamic computation-offloading strategy (DCOS) in the next section.