# Intelligent Computation Offloading Based on Digital Twin-Enabled 6G Industrial IoT
## 4 DRL-Driven Offloading Scheme Based on DT
### Solution Based on DDPG  
In this section, the DT-enabled network is considered as a DRL agent to resolve tasks and make decisions. At each time step \(t\), the agent is given a set of states and learns the policy that allows the agent to maximize its rewards in this environment. After executing the action, the agent obtains feedback from the environment and accesses the value of the policy to optimize the parameters of the neural network. The commonly used DDPG algorithm is an actor-critic algorithm that supports continuous action spaces. It uses two networks to generate actions and evaluate the current policy. The structure of the DRL agent is shown in Figure 3. Both the actor and the critic networks contain an original network and a target network. Let us start with the actor network. At time step \(t\), the agent executes \(a_{t}\) in the DT virtual environment and then obtains the next state \(s_{t+1}\) and immediate reward \(r_{t}\). Existing known tuples \((s_{t},a_{t},s_{t+1},r_{t})\) are stored in the experience replay buffer for further training. When the experience buffer overflows, DDPG constructs a mini-batch by randomly selecting \(M\) samples in the buffer. Let \((s_{m},a_{m},s_{m+1},r_{m})\) be the \(m\) sample in the mini-batch. At the same time, the actor takes as input \(s_{m}\) and outputs a deterministic action, and the network that produces this action is defined as \(a_{m}=\pi_{\mu}(s_{m})\). Here, \(\pi\) denotes the deterministic policy and \(\mu\) represents the parameters of the Original Actor network. When this action is input to the Critic network, it can obtain the maximum value \(Q_{\theta}(s_{m},a_{m})\). The ideal output of the network is an accurate evaluation of the policy, so that the Actor network can adjust the policy accordingly and discard the actions with poor \(Q\) value feedback. The Actor network is designed to maximize the output \(Q\) value of the Critic network, so it is updated based on gradient boosting. The loss function is given a minus sign to minimize the error. \[L(\mu)=\mathbb{E}\big{[}-Q_{\theta}(s_{m},\pi_{\mu}(s_{m}))\big{]} \tag{21}\]  
The gradient expression of \(Q\) with \(\mu\) can be obtained by the chain derivation rule. \[\
abla_{\mu}J(\mu)=\frac{1}{M}\sum_{m=1}^{M}\left(\
abla_{a_{m}}Q_{\theta}(s_{ m},a_{m})\
abla_{\mu}\pi_{\mu}(s_{m})\right) \tag{22}\]  
Moreover, it is the Target Actor network that updates the value network Critic, which has the same structure as the Original Actor network except that its parameter is \(\mu^{\prime}\). Both Actor networks output actions, which soft update the parameters of the Target Actor network. \[\mu^{\prime}\leftarrow\rho\mu+(1-\rho)\mu^{\prime} \tag{23}\]  
Here \(\rho\) is a small step factor that controls the updating speed. Now consider the Critic network; its role is to fit the value function \(Q_{\theta}(s_{m},a_{m})\). Similarly, the Critic network has an original network and a target network with parameters \(\theta\) and \(\theta^{\prime}\), respectively. The input sides of the two Critic networks are from different sources, and both output sides are the \(Q\) value of the current state. The input of the Original Critic network is the action output from the current Original Actor network. There are two Target Critic network inputs, the observed value of the current state and the action output from the Target Actor network. The optimal value of the state-action pair is obtained according to Bellman equation:  
\[Q^{*}(s_{m},a_{m})=\mathbb{E}(r(s_{m},a_{m})+\gamma Q^{*}(s_{m+1},a_{m+1})) \tag{24}\]  
where \(Q^{*}\) denotes the optimal value. In DDPG, we directly approximate \(Q^{*}\) with a neural network \(Q_{\theta}\) with parameter \(\theta\). Through training, the neural network can output an accurate estimate of \(Q_{\theta}\) for a given state-action pair. Then, we build a loss function measured by mean squared error through temporal difference (TD) learning, which can be used to measure the estimated deviation of \(Q_{\theta}(s_{m},a_{m})\) to the Bellman equation.