## 4 Methodology
### Dynamic Resource Scheduling and Task Offloading Based on Value Iteration
#### 4.1.3 Optimization Strategy
Once convergence is achieved, the optimal policy is extracted by selecting the action that maximizes the following expression:\[a_{t}^{*}=\arg\max_{a_{t}\in\mathbb{A}}\left[r_{t}(s_{t},a_{t})+\gamma\sum_{s_{t+1 }\in\mathbb{S}}P(s_{t+1}|s_{t},a_{t})V^{*}(s_{t+1})\right] \tag{24}\]  
This policy represents the optimal strategy for task offloading and resource allocation in the dynamic environment, ensuring maximum long-term resource utilization and minimized task delays. Finally, we design Algorithm 1 to clear the expression of the main idea above. ```
0: Initial state \(s_{0}\), learning rate \(\eta\), discount factor \(\gamma\), maximum number of iterations \(T\), reward function parameters \(\alpha_{1},\alpha_{2}\), state transition probabilities \(P(s_{t+1}|s_{t},a_{t})\). 0: Optimal task offloading and resource scheduling policy \(\pi^{*}(s_{t})\). 1:Initialize value function \(V_{0}(s_{t})\) to zero or initial estimate. 2:for each iteration \(k=0,1,\ldots,T-1\)do
3:for each time step \(t\)do
4: Perceive current state \(s_{t}\);
5: Select optimal action \(a_{t}^{*}\) using Equation (24);
6: Execute action \(a_{t}^{*}\), update system state \(s_{t+1}\) using Equation (18);
7: Observe reward \(r_{t}(s_{t},a_{t})\);
8: Update value function using Equation (23);
9:endfor
10:endfor
11:if\(\|V_{k+1}^{*}(s_{t})-V_{k}^{*}(s_{t})\|<\epsilon\)then
12: terminate
13:for each state \(s_{t}\)do
14: Compute optimal policy \(\pi^{*}(s_{t})\) using Equation (22);
15:endfor
16:return\(\pi^{*}(s_{t})\)
17:endif
```  
**Algorithm 1** Dynamic resource scheduling and task offloading algorithm (RSTO)  
The input parameters include the initial state \(s_{0}\), learning rate \(\eta\), discount factor \(\gamma\), maximum number of iterations \(T\), reward function parameters \(\alpha_{1}\) and \(\alpha_{2}\), and state transition probabilities \(P(s_{t+1}|s_{t},a_{t})\). The output is the optimal task offloading and resource scheduling policy \(\pi^{*}(s_{t})\). The algorithm begins by initializing the value function \(V_{0}(s_{t})\) to zero or another initial estimate (line 1), which serves as the baseline for iteration. The iterative process runs from iteration 0 to \(T-1\), traversing all time steps \(t\) (lines 2-3). At each time step, the current system state \(s_{t}\) is perceived for decision-making (line 4), representing the system's understanding of its current situation, including resource availability and task statuses. The optimal action \(a_{t}^{*}\) is selected based on the optimal policy equation (line 5), ensuring long-term performance optimization by balancing immediate rewards with future rewards. The action is executed, and the system state is updated to \(s_{t+1}\) (line 6), with the transition governed by \(P(s_{t+1}|s_{t},a_{t})\). The immediate reward \(r_{t}(s_{t},a_{t})\) is observed (line 7), reflecting the feedback from the action taken. The value function is adjusted using the value function update formula (line 8), incorporating both immediate rewards and expected future rewards to iteratively improve the system's decision-making process. After completing all time steps, the change in the value function is checked to determine if it is less than the preset threshold \(\epsilon\) (lines 11 and 12). This check ensures convergence, meaning that further iterations will not significantly improve the value estimates. If the convergence condition is met, the iteration terminates. The optimal policy \(\pi^{*}(s_{t})\) is computed for each state using the optimal policy equation (lines 13-16), representing the best strategy for task offloading and resource scheduling to achieve maximum resource utilization and minimal latency. This policy is output to guide the system's decision-making process, ensuring that the system optimizes its operations based on the learned strategy.