## 5 Algorithm Design for \(\mathcal{OP}\)
### Problem Conversion
#### 5.3.2 Proposed Task Splitting Algorithm Based on DDPG  
The algorithm for the optimization of task splitting in this paper is based on the DDPG method, as shown in Figure 2. In this algorithm, four neural networks are included, namely an actor network, a critic network, a target actor network, and a target critic network. Below, we first introduce the actor and critic networks. Figure 2: The schematics of the proposed intelligent algorithm. (1) Actor and Critic Networks:  
The output of the actor network is a continuous action value, expressed as \(a=\phi(s|\eta)\), where \(\eta\) represents the parameters of the actor network. The output of the critic network is the evaluation of the action a output by the actor network, which can be expressed as \(Q(s,a|\theta)\), where \(\theta\) represents the parameters of the critic network. During each round of training, the actor network will output an action decision \(a\) based on the current environment state \(s\), and then the critic network will evaluate the decision; that is, obtain \(Q(s,a)\). Our constructed actor network consists of two parts: a deep neural network (DNN) module and an action validity assurance module based on a normalization function, as shown in Figure 3. The DNN module is responsible for making optimized task partitioning decisions based on the input state. However, the output of the DNN cannot guarantee compliance with constraints (13b) and (13c). Therefore, we designed the action validity assurance module to ensure that the actor network's output meets these constraints. First, we restructure the DNN's output into a two-dimensional tensor (denoted as \(\delta\)), with dimensions of \(J\) rows and four columns. We represent the \(j\)-th row of \(\delta\) as \(\delta_{j}=\{\delta_{1,j},\delta_{2,i},\delta_{3,j},\delta_{4,j}\}\). Then, we normalize each \(\delta_{j},j\in\mathcal{J}\), and the processed result is expressed as \(\epsilon_{j}\) and \(\epsilon_{j}\) satisfies the following:  
\[\epsilon_{ij}=\frac{\delta_{i,j}}{\sum_{k=1}^{4}\delta_{k,j}},i\in\{1,2,3,4\}, j\in\mathcal{J}. \tag{21}\]  
Obviously, each \(\epsilon_{j}\) satisfies \(\epsilon_{1,j}+\epsilon_{2,j}+\epsilon_{3,j}+\epsilon_{4,j}=1\); that is, \(\epsilon_{j}\) satisfies constraints (13b) and (13c). In this way, we can ensure that the output of the actor network (i.e., the task splitting decision) is valid. DDPG introduces an experience buffer mechanism to store states, actions, and rewards information, represented as \(\{s_{t},a_{t},r_{t},s_{t+1}\}\). During training, to update the critic and actor networks, a batch of \(\Xi\) samples is first drawn from the replay buffer. For the sample \(s_{t},a_{t},r_{t},s_{t+1}\), the Critic network first computes the target value, satisfying the following:  
\[y_{t}=r_{t}+\gamma Q^{\prime}(s_{t+1},\phi(s_{t+1}|\eta^{\prime})|\theta^{ \prime}). \tag{22}\]  
Then, by minimizing the loss function \(L(\theta)=\sum_{t\in\Xi}(y_{t}-Q(s_{t},a_{t}|\theta))^{2}\), we can update the critic network by one step of gradient descent using \(\frac{1}{|\Xi|}\
abla_{\theta}\sum L(\theta)\). In the same way, we can update the actor network parameters by one step of gradient descent using \(\frac{1}{|\Xi|}\sum(\
abla_{\eta}\phi(s_{t})\
abla_{a_{t}}Q(s_{t},a_{t}|\theta))\). (2) Target Actor and Target Critic Networks:  
In order to improve the stability of the training process, reduce the fluctuation of the training process, and reduce the variance of the training process, the DDPG method introduces the target actor network and the target critic network. The target actor network is used to output the best action of the next state, and the target critic network is used to evaluate the Q value of the next state. The structures of the target actor network and the target critic network are the same as the actor network and critic network. Moreover, the  
Figure 3: The framework of the actor and target actor networks.