## 4 DRL-Based Joint Task Offloading and Resource Allocation Scheme
### An Improved DDPG-Based Joint Optimization Algorithm  
Considering that the transmit power, phase shift, and the offloading scale factor are continuous variables, we are resorting to the policy-based scheme. The DDPG algorithm has been proven as an effective solution for the continuous control problem [23]. Thus, the DDPG-based scheme is developed in this work. Figure 2 depicts the developed framework. The proposed DRL model adopted the evaluate network and target network with identical structures but differing parameters. Both evaluate and target networks contain a set of actor-critic neural networks. At each time slot, the evaluate network obtains environmental state \(s_{t}\) and then outputs the action \(a_{t}\). The \(Q\) value is adopted to describe the long-term reward of executing \(a_{t}\), which can be calculated by the Bellman equation [38]  
\[Q^{\mu}(s_{t},a_{t})=\mathbb{E}[r_{t}(s_{t},a_{t})+\gamma Q^{\mu}(s_{t+1},a_{t+ 1})], \tag{33}\]  
Figure 2: Proposed task offloading and resource allocation framework based on DDPG. where \(\mu:\mathcal{S}\leftarrow\mathcal{A}\) is the deterministic policy function, and the actor function \(\mu(s|\omega^{\mu})\) works by mapping a state to an action to specific current policy. The DRL agent interacts with the environment to find the optimal action corresponding to the maximum \(Q\) value  
\[Q^{*}(s_{t},a_{t})=\mathbb{E}\bigg{[}r_{t}(s_{t},a_{t})+\gamma\max_{a_{t+1}}Q^{* }(s_{t+1},a_{t+1})\bigg{]}. \tag{34}\]  
The experience replay mechanism is leveraged to break the correlation between experience tuples [39]. Applying \(J\) tuples sampled from the experience buffer, the critic network is trained by minimizing the loss function  
\[L\Big{(}\omega^{Q}\Big{)}=\frac{1}{J}\sum_{i\in J}\Big{(}y_{i}-Q\Big{(}s_{i},a_ {i}|\omega^{Q}\Big{)}\Big{)}^{2}, \tag{35}\]  
where  
\[y_{i}=r(s_{i},a_{i})+\gamma Q^{\prime}\Big{(}s_{i+1},a_{i+1}|\omega^{Q^{\prime }}\Big{)}, \tag{36}\]  
denotes the target value. \(\omega^{Q^{\prime}}\) represents the parameters of the function approximator. The actor network updating following the policy gradient rule and the loss function can be expressed as  
\[\
abla_{\omega^{\mu}}J=\frac{1}{J}\sum_{i\in J}\Big{[}\
abla_{a}Q\Big{(}s,a| \omega^{Q}\Big{)}|s=s_{i},a=\mu(s_{i})\
abla_{\omega^{\mu}}\mu(s|\omega^{\mu}) |s=s_{i}\Big{]}. \tag{37}\]  
To address the unstable issue in the learning process, the soft target is leveraged for the updating of target actor-critic networks, which can be formulated by  
\[\omega^{\mu^{\prime}} \leftarrow\tau\omega^{\mu}+(1-\tau)\omega^{\mu^{\prime}}, \tag{38}\] \[\omega^{Q^{\prime}} \leftarrow\tau\omega^{Q}+(1-\tau)\omega^{\mu^{\prime}}, \tag{39}\]  
with the soft update factor \(\tau\ll 0\). The experience replay mechanism overcomes the problem prone to divergence in the training process. Since the conventional experience replay mechanism replayed the transition tuples uniformly, the importance of different experiences is ignored. The prioritized experience replay (PER) assigns priorities based on the importance of the experience samples, which is adopted to speed up the training convergence [39]. The internal logic of the PER mechanism is to replay extremely good or bad experiences more frequently. The temporal difference error (TD-error) is usually leveraged as the measurement of the experiences' value [40]. The absolute TD-error is proportional to the correction to the expected action value. The TD-error of transition tuple \(i\) can be formulated by  
\[\delta_{i}=y_{i}-Q\Big{(}s_{i},a_{i}|\omega^{Q}\Big{)}. \tag{40}\]  
The probability of the transition \(i\) is given by  
\[P(i)=\frac{p_{i}^{\varrho}}{\sum_{k}p_{k}^{\varrho}}, \tag{41}\]  
where \(\frac{1}{\text{rank}(i)}\), \(\text{rank}(i)\) represents the ranking of transition \(i\) when sorted according to the absolute TD-error. \(\varrho\) is the degree of priority adopted. However, PER changes the state access frequency, introduces the bias, and may cause oscillation and divergence.