## 4 Model Solving
### Algorithm Design
#### 4.2.2 Design of the Reward Function  
The reward function is generally designed according to the objective function of the system model, which minimizes the total cost of processing delay and energy consumption by making optimal offloading decisions. In the model of this paper, tasks may exceed the limits of tolerated delay regardless of whether they are offloaded or not. These tasks are considered as unsuccessful tasks, but also consume the corresponding resources. In order to reduce the number of tasks thrown in the whole system and to minimize the cost of the system, a reasonable reward function must be designed.  
For MDs that only consider saving local energy consumption, offloading is preferred. If all MDs offload the tasks, the edge server will be overloaded and the idle state of MDs is also a waste of computational resources, so the execution delay and energy consumption of tasks need to be considered at the same time. When a task completes successfully without exceeding a given tolerance delay, a corresponding reward is given; when a task exceeds its tolerance delay, the task is considered as discarded and penalized. With such a design, the action is rewarded whether the task is executed locally or on the edge server, as long as it is completed within the specified time. In this paper, for a task to be executed only if it is completed within the specified time and consumes the least amount of energy, the shorter the execution delay, the greater the reward the task receives, and the least cost of the whole system. In order to be consistent with the objective in the model of this paper, we use negative rewards. When the system objective function is at the minimum level, the DRL can get the maximum reward. Based on the above considerations, the reward function is designed as follows.  
\[r_{t}(s_{t},a_{t})=-[(\alpha T+\beta E)-p_{t}] \tag{23}\]  
where \(p_{t}\) is the penalty for the corresponding task dropped.  
The structure of the DRL algorithm in this paper is based on the combination of DQN parameter update improved Double DQN and neural network structure improved Dueling DQN algorithm, and then the LSTM prediction algorithm is migrated to DRL decision algorithm. In this paper, we propose an Online Predictive Offloading (OPO) algorithm based on the deep reinforcement learning algorithm to solve the modeling problem, and the specific process of model training is shown in Algorithm 1.  
In the algorithm, lines 8-14 with probability \(1-\epsilon\) select a random action or LSTM predict action \(a\), and lines 21-25 are network parameter updates referencing the Double DQN update method.  
```
1:Input: Input different tasks in each time slots
2:Output: Optimal offloading decision and total cost
3:Initialize \(Q^{A}\),\(Q^{B}\) and \(s\)
4:Initialize replay memory D to capacity N;
5:for episode = 1, M do
6: Initialize sequence s, and preprocessed sequence
7:for t = 1, T do
8: With probability \(1-\varepsilon\) select a random action or LSTM predict action
9: Generate another random number \(\sigma\)
10:if\(\sigma>\varepsilon\)then
11:\(a_{t}\) = Random Action Selection(\(s_{t}\))
12:endif
13:if\(\sigma<\varepsilon\)then
14:\(a_{t}\) = Prediction Action Selection(\(s_{t}\))
15:endif
16: Otherwise select \(a\) by \(a^{*}\)=\(argmax_{a}Q^{A}(s,a)\) or \(b^{*}\)=\(argmax_{a}Q^{B}(s,a)\)
17: Execute action \(a_{t}\) and receive \(r^{t}\) and \(s_{t+1}\)
18: Store \((s_{t},a_{t},r_{t},s_{t+1})\) into D
19: Randomly sample \(a\) mini-batch of experience from D
20: Preform a gradient descent step on \(Loss(\theta)\) with respect to the network parameters
21: Choose \(a\), based on \(Q^{A}(s,\bullet)\) and \(Q^{B}(s,\bullet)\), observe \(r,s^{\prime}\)
22:ifUPDATE(A) then
23:\(Q^{A}(s,a)\gets Q^{A}(s,a)+\rho\big{[}r+\gamma\max_{a^{\prime}}Q^{B}(s^{ \prime},a^{*})-Q^{A}(s,a)\big{]}\)
24:else ifUPDATE(B) then
25:\(Q^{B}(s,a)\gets Q^{B}(s,a)+\rho\big{[}r+\gamma\max_{a^{\prime}}Q^{A}(s^{ \prime},b^{*})-Q^{B}(s,a)\big{]}\)
26:endif
27:endfor
28:endfor
29:Repeat
```  
**Algorithm 1** Online Predictive Offloading Algorithm.