## 4 DRL-Based Joint Task Offloading and Resource Allocation Scheme
### Twin Delayed DDPG (TD3)-Based Joint Optimization Algorithm  
The TD3 algorithm is considered as an improver of DDPG, which solves a series of issues caused by overestimation in the process of the \(Q\) value estimate in DDPG [41]. We depicts the TD3-based joint optimization framework in Figure 3. Although the overestimated values are small in each update, they may accumulate after every update, creating a significant bias. Furthermore, the inaccurate \(Q\) value leads to the deterioration of the policy network. This process forms a feedback loop in which suboptimal behavior is continuously reinforced. The TD3 algorithm addresses the above-mentioned challenge through the following technologies.  
Figure 3: Proposed task offloading and resource allocation framework based on TD3.  
Firstly, clipped double \(Q\) learning. The TD3 leverages twin critic networks to estimate two \(Q\) function, and choose the smaller one as the target \(Q\) value to compute loss in the Bellman equation. The target update in the double critic networks framework is formulated as  
\[y_{i}=r(s_{i},a_{i})+\gamma\min_{n=1,2}Q_{n}^{\prime}\Big{(}s_{i+1},a_{i+1}| \omega^{Q_{n}^{\prime}}\Big{)}, \tag{42}\]  
where \(\omega^{Q_{n}^{\prime}}(n=1,2)\) denote weight parameters of two target critic networks, respectively. Critic networks are updated by using the loss function, which are given by  
\[L\Big{(}\omega^{Q_{1}}\Big{)}=\frac{1}{\gamma}\sum_{i\in J}\Big{(}y_{i}-Q_{1} \Big{(}s_{i},a_{i}|\omega^{Q_{1}}\Big{)}\Big{)}^{2}, \tag{43}\]  
\[L\Big{(}\omega^{Q_{2}}\Big{)}=\frac{1}{\gamma}\sum_{i\in J}\Big{(}y_{i}-Q_{2} \Big{(}s_{i},a_{i}|\omega^{Q_{2}}\Big{)}\Big{)}^{2}, \tag{44}\]  
where \(\omega^{Q_{1}}\) and \(\omega^{Q_{2}}\) indicate weight parameters of two estimate critic networks, respectively. The smaller value is adopted for the Bellman error function. Secondly, delayed policy updates. The actor and its target network reduce the update frequency compared to critic networks, to avoid the divergent behavior caused by the policy updates under inaccurate value estimate. Thirdly, target policy smoothing. A regularization strategy is leveraged in TD3 to address the overfit at high peaks and \(Q\) value error. In practice, a random noise is added in the action selection process to enforce the generalization of similar actions as given by  
\[\tilde{a}_{t+1}\leftarrow\mu(s_{t+1}|\omega^{\mu})+\epsilon, \tag{45}\]  
where the added noise \(\epsilon\sim clip(\mathcal{N}(0,\sigma_{a}),-c,c)\) is clipped by the constant \(c\) to ensure the proximity between the target action and the original. The TD3-based joint optimization algorithm is similar to the processing process of Algorithm 1, with improvements in the following aspects:  
* In the Input Step, input two pairs of critic networks \(Q_{1}(s,a|\omega^{Q_{1}})\) and \(Q_{2}(s,a|\omega^{Q_{2}})\), respectively. In Step 1, initialize parameters of two estimate critics and two target critics with \(\omega^{Q_{1}}\), \(\omega^{Q_{2}}\), \(\omega^{Q_{1}}\), and \(\omega^{Q_{2}}\).
* Target policy smoothing is realized by (45). Then, the agent updates the target value using (42). In Step 16, the loss is computed by (43) and (44).
* Before turning to Step 17, the agent adopts a delayed update strategy to keep policy networks updated less frequently than value networks.