# Intelligent Computation Offloading Based on Digital Twin-Enabled 6G Industrial IoT
## 4 DRL-Driven Offloading Scheme Based on DT
### Solution Based on DDPG
\[L(\theta)=\mathbb{E}[(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{m+1})-Q_{\theta}(s _{m},a_{m}))^{2}] \tag{25}\]  
Next, we obtain the gradient for the loss function  
\[\
abla_{\theta}f(\theta)=\mathbb{E}[2(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{ m+1})-Q_{\theta}(s_{m},a_{m}))\
abla Q_{\theta}(s_{m},a_{m})] \tag{26}\]  
With the gradient, we can use gradient descent to solve the above optimization problem. The TD target \(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{m+1})\) changes after each network parameter update, which is not conducive to the stability of training. Therefore, it is common to use two sets of networks: an original network (parameter \(\theta\)) and a target network (parameter \(\theta^{\prime}\)) to estimate \(Q_{\theta}(s_{m},a_{m})\) and \(Q_{\theta^{\prime}}(s_{m+1},a_{m+1})\), respectively. That is, the parameters of the target network are not updated immediately with each update but are synchronized with the original network parameters after a certain interval. \[\
abla_{\theta}J(\theta)=\frac{2}{M}\underset{m=1}{\sum}^{M}[r(s_{m},a_{m})+ \gamma Q_{\theta^{\prime}}(s_{m+1},a_{m+1})-Q_{\theta}(s_{m},a_{m})]\
abla Q_{ \theta}(s_{m},a_{m}) \tag{27}\]  
Soft updates to the parameters of the Target Critic network are given by  
\[\theta^{\prime}\leftarrow\rho\theta+(1-\rho)\theta^{\prime} \tag{28}\]  
The workflow of the proposed algorithm is presented in Algorithm 1. ```
Input: The original Actor parameter \(\mu\). The target Actor parameter \(\mu^{\prime}\). The original Critic parameter \(\theta\). The target Critic parameter \(\theta^{\prime}\). Discount factor \(\gamma\). Soft update step factor \(\rho\). Mini-batch size \(M\). Output: Computation offloading scheme and network resource allocation. 1. Initialize Actor networks and Critic networks. 2.for\(epi=1\) tothreshold\({}_{E}\)do
3. Digital twin observes its \(DT_{n}\). 4. Normalize the state \(\mathcal{S}\) and receive the initial state \(S_{1}\). 5.for\(step=1\) tothreshold\({}_{T}\)do
6. Choose and execute action \(a_{t}\) and map it onto \(DT_{n}\). 7. Calculate reward \(r_{t}\) and receive next station \(s_{t+1}\). 8.if the experience replay buffer is not full then
9. Store transition \((a_{t},s_{t},s_{t+1},r_{t})\) in the buffer. 10.else
11. Randomly replace a tuple in the buffer. 12. Sample a minibatch of \(M\) transition \((a_{m},s_{m},s_{m+1},r_{m})\). 13.for\(m\)=1 to \(M\)do
14. Calculate the target value based on \(Q^{*}(s_{m},a_{m})=\mathbb{E}(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{m+1}))\). 15. Update the Original Critic network by \(L(\theta)=\mathbb{E}[(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{m+1})-Q_{\theta} (s_{m},a_{m}))^{2}]\). 16. Update the Original Actor network by \(L(\mu)=\frac{1}{M}\underset{m=1}{\sum}^{M}(\
abla_{a_{m}}Q_{\theta}(s_{m},a_{ m})\
abla_{\mu}\
abla_{\mu}\
abla_{\mu}(s_{m}))\). 17.endfor
18.endif
19. Soft update the Target Actor network by \(\mu^{\prime}\leftarrow\rho\mu+(1-\rho)\mu^{\prime}\). 20. Soft update the Target Critic network by \(\theta^{\prime}\leftarrow\rho\theta+(1-\rho)\theta^{\prime}\). 21.endfor
22.endfor
```  
**Algorithm 1**DRL-driven computation offloading algorithm with DT-enabled architecture