## 4 Design of Proposed Algorithm
### Algorithm Design  
#### 4.3.1 Seq2 Seq  
The strategy for offloading task \(t_{i}\) is defined as \(\pi(a_{i}|s_{i})\) on the basis of the MDP setting in Section 3.2.2. This means that, when task \(t_{i}\) arrives at the decision module, the decision module makes the task offloading decision \(a_{i}\) according to the current state \(s_{i}\). Assuming that a particular DAG task G can be represented by n subtasks, the strategy for these n subtasks can be expressed as \(\pi(A_{1:n}|\text{G})\). Because each task's offloading decision is linked to the task before it, the chain rule can be used to represent \(\pi(A_{1:n}|\text{G})\) in terms of \(\pi(a_{i}|s_{i})\).  
\[\pi(A_{1:n}|\text{G})\ =\ \prod_{i\ =\ 1}^{n}\pi(a_{i}|s_{i}). \tag{14}\]  
The decisions taken by each of a task's n subtasks affect how that task will be executed, and the number of subtasks differs from task to task. This difference can cause training difficulties for traditional neural networks. The Seq2 Seq deep network is able to support the input of a different number of decisions, and it accepts and processes this chained strategy with recurrent neurons, whose structure is shown in Figures 3 and 4, which contains an encoder and a decoder. Both parts are implemented by recurrent neural networks, where the encoder compresses the subtask sequences into uniform context vectors, and then the decoder decodes these vectors to output the policy.  
To keep the performance from going down because the context vector is too long, an attention mechanism is used. This makes the decoder focus on the parts that are closer to it when it is making the output. In the process of decoding and output, the context vector element is given greater weight the closer it is to the source. If the encoder's input is the subtask state sequence \([s_{1},s_{2},\ldots,s_{n}]\), the decoder's output is the corresponding policy \([a_{1},a_{2},\ldots,a_{n}]\), the encoder function is \(f_{encoder}\), and the decoder function is \(f_{decoder}\); then, the encoder of the \(i\)-th task hidden output may be represented as  
Figure 3: Task unloading network based on seq2 seq.  
\[e_{i}\ =\ f_{encoder}(s_{i},h_{i-1}). \tag{15}\]  
The context vector can be expressed as \(c\ =\ [e_{1},e_{2},\ldots,e_{n}]\), and then the message is decoded by the decoder according to the last action \(a_{j-1}\) in the context vector and the last decoder output \(d_{j-1}\); its decoding formula can be expressed as  
\[d_{j}\ =\ f_{decoder}\big{(}c_{j},d_{j-1},a_{j-1}\big{)}, \tag{16}\]  
where \(c_{j}\) is the weighted sum of partial context vectors by applying the attention mechanism, which is calculated as follows:  
\[c_{j}\ =\ \sum_{i\ =\ 1}^{n}\mu_{ji}h_{i}, \tag{17}\]  
where \(\mu_{ji}\) is a probability distribution, which is calculated as follows:  
\[\mu_{ji}\ =\ \text{softmax}\big{(}score\big{(}h_{i},d_{j-1}\big{)}\big{)}, \tag{18}\]  
where \(score(h_{i},d_{j-1})\) is a function to determine the degree of matching between \(h_{i}\) and \(d_{j-1}\), and this function is defined as a trainable feedforward neural network in the literature. Lastly, the TD3 algorithm is improved by transforming the decoder output \(d\ =\ [d_{1},d_{2},\ldots,d_{n}]\) into a policy and value network using two fully connected layers to make it applicable to the model of task offloading policy.