# MADDPG-Based Offloading Strategy for Timing-Dependent Tasks in Edge Computing
## 4 User-Prediction-Based Timing-Dependent Task Offloading Algorithm
### Timing-Dependent Task Offloading Algorithm
At the moment \(t\)+1, because the GRU module in the deep neural network helps to memorize the historical offloading decisions and the task information, the agent selects the current offloading action based on the information of the user's behavioral prediction, the reward \(r_{t}\) from the previous moment, and the current moment \(t\)+1 observation information, to select the offloading action for the current round. In complex scenarios with multiple concurrent users, task offloading using multi-agent reinforcement learning has the problem of poor timeliness of the offloading strategy due to its long training time. Therefore, the user-behavior-prediction-based dependent task offloading (UBP_DTO) algorithm is proposed. The proposed algorithm considers historical offloading decision information and task resource occupancy to predict the computing resource usage of the task at the next moment, improving the training efficiency of reinforcement learning and the timeliness of offloading strategy. The specific optimization process is shown in Algorithm 1. Figure 4: Training process based on the UBP-DTO algorithm. The main content of the UBP_DTO algorithm optimization is shown in line 9 of Algorithm 1, which changes the mechanism for obtaining instant rewards during the reinforcement learning training process. When training and selecting actions, the agent focuses on historical user behaviour information and selects actions based on effective task offloading decisions in historical information, the current environment computing resource status, and the CPU resources required for precursor tasks, narrowing the action selection range and improving training efficiency. ```
1:Initialize the environment, including the resources, quantity, and location of MEC nodes and users
2:Agent \(d\) observes the state of the environment \(o_{t}\) and selects the action \(a_{t}\)
3:Obtain the environment state feedback reward \(r_{t}\)
4:for episode = 2 to max_episode do
5:Reset the MEC network
6:Get the task scheduling sequence
7:while the episode do not end do
8:for agent \(d\) in the set of agents do
9:Observe the \(o_{t+1}\), \(r_{t}\) and select the action \(a_{t+1}=\mu_{d}(o_{t+1},r_{t};\theta_{d})\)
10:endfor
11:Executing \(a_{t+1}\);(\(o_{t+1}\), \(r\)) = \(Actions[a_{d}]\) in parallel
12:for agent \(d\) in the set of agents do
13:Record the experience (\(o_{t}^{d}+1\), \(a_{t}^{d}+1\), \(r_{t}^{d}+1\), \(o_{t}^{d}+2\)) of agent \(d\) and put it into the experience replay pool
14:endfor
15:for agent \(d\) in the set of agents do
16:Update policy network (agent \(d\))
17:Update value network (agent \(d\))
18:endfor
19:\(\theta_{d}\leftarrow\tau\theta_{d}+(1-\tau)\theta_{d}\)
20:endwhile
21:endfor
22:Each agent autonomously decides the task offloading mode and generates the offloading policy \(\pi_{d}\)
23:Return \(\pi\)
```  
**Algorithm 1**MADDPG-based timing-dependent task offloading algorithm  
In a multi-user concurrent-timing-dependent task offloading system, each UE is configured with an independent agent that can initiate service requests at the same time. All user terminal agents need to make task offloading and location selection decisions based on changing environmental information. For this process, this article makes the following environment configuration for the edge-computing task offloading solution based on the MADDPG algorithm:  
**State**: \(x_{t}\) denotes the distributed state \(o_{t}\) observed by all local UEs at moment \(t\). The overall environmental state can be denoted as \(o_{t}=[Z_{n},R_{n},T_{sum},RL,D_{j}]\). \(Z_{n}\) denotes the location of the MEC node observed by each agent, which is used to compute the transmission time for task offloading. \(R_{n}\) denotes the computing resources of the MEC node observed by each agent and can be used to determine whether the node has sufficient computing resource to support task offloading. It can also be used to evaluate the potential benefit of the current task offloading. \(T_{sum}\) denotes the task offloading time.