## 4 Design of Proposed Algorithm
### Algorithm Design
#### 4.3.2 MTD3 CO  
(Meta TD3 Computation Offloading) Implementation  
In order to make the DAG task offloading adaptable to different tasks and, thus, achieve generalization, this chapter combines meta-learning and the TD3 algorithm [24] to improve the algorithm's adaptability to the environment. After the initial training is complete, the algorithm can set up the outer-loop learner of the mobile device with a meta-strategy. The strategy can then be fine-tuned with a small number of specific tasks, and then the outer-loop learner strategy can be changed to fit the specific tasks of the mobile device. For the actor-critic method, there is an unavoidable problem of overestimation due to cumulative errors. First, the TD3 algorithm uses the idea of double Q-learning by using two independent value functions, and using the smaller value between them in the update to reduce the deviation caused by overestimation. Second, when performing TD updates, the errors made at each step add up and make the estimation variance too high; thus, the TD3 algorithm uses the target network and delayed updates to solve this problem. Figure 4: Training process of MTD3 CO.  
Lastly, the TD3 algorithm enhances the exploratory nature by adding noise to the target actions, which smoothens the values in the region near the value network actions and reduces the generation of errors. On the basis of the above improvements, assuming that the policy network in the TD3 algorithm is \(\pi_{\phi}(s)\), and the two value networks are \(Q_{\theta_{1}}(s,a)\) and \(Q_{\theta_{2}}(s,a)\) with parameters \(\phi\), \(\theta_{1}\), and \(\theta_{2}\), respectively, then the actions and expected rewards of the task unloading policy at each training can be expressed as  
\[\left\{ \begin{aligned} \widetilde{a}&=\ \pi_{\phi^{\prime}}(s^{\prime})+\text{clip}\left( \mathcal{N}\left(0,\widetilde{\sigma}\right),-c,c\right)\\ &\ y\gets r+\gamma_{i}\min_{i=1,2}Q_{\theta_{i}^{\prime}} \left(s^{\prime},\widetilde{a}\right)\end{aligned} \right., \tag{19}\]  
where \(\phi^{\prime}\), \(\theta_{1}^{\prime}\), and \(\theta_{2}^{\prime}\) denote the parameters of the target policy network and the two target value networks, respectively, and \(\text{clip}\left(\mathcal{N}\left(0,\widetilde{\sigma}\right),-c,c\right)\) is the clipped noise function that makes the action fluctuate in a small range. \(\gamma\in[0,1]\) denotes the discount factor of learning. According to the expected reward, two value networks can use the same expected reward \(y\) and the respective rewards of the current value network \(Q_{\theta_{1}}(s,a)\) and \(Q_{\theta_{2}}(s,a)\) to determine the TD error and update the current value network parameters, respectively, by minimizing the error as the objective, whose objective function formula can be defined as  
\[J_{critic}^{TD3}(\theta_{i})\ =\ N^{-1}\sum\left(y-Q_{\theta_{i}}(s,a)\right)^{2},i\ =\ \{1,2\}. \tag{20}\]  
For the update of the strategy network, the objective is to find the strategy parameter with the largest expected return, which can be updated in the actor-critic method using the deterministic strategy gradient algorithm, which uses gradient ascent to find the parameter with the largest return \(\phi\). Therefore, for the actor network in this section, the objective function can be expressed as  
\[J_{actor}^{TD3}(\phi)\ =\ N^{-1}\sum Q_{\theta_{1}}(s,a)\Big{|}_{a}\pi_{\phi}(s). \tag{21}\]  
According to the model established in Section 3.2.2, we model the offloading tasks corresponding to different types of applications as multiple MDPs, each of which is responsible for generating an offloading policy for the same class of tasks. Formally, the task distribution is defined as \(\rho(T)\), and each task follows the task distribution \(T_{i}\sim\rho(T)\). MTD3 CO and gradient-based meta reinforcement learning share a similar structure and also have two parts: inner-loop learning and outer-loop learning.