## 4 Methodology
### Dynamic Resource Scheduling and Task Offloading Based on Value Iteration
#### 4.1.3 Optimization Strategy  
To effectively address the issues of low resource utilization and high system latency in satellite networks, this study proposes a collaborative optimization framework based on cooperative dynamic programming. This framework constructs latency optimization and resource efficiency models, employing dynamic programming and value iteration techniques to enable real-time updates of system states and decision variables. Specifically, task offloading decisions are represented by a discrete matrix \(\mathbf{X}(t)\), which considers not only task priorities and deadlines but also the computational, storage, and bandwidth resources of each node. The state transition is influenced by the current task processing status, node resource availability, and link delays. The state transition probability is defined as \(P(s_{t+1}|s_{t},a_{t})\), where \(s_{t+1}\) is the system state at the next time step. The state transition is governed by  
\[s_{t+1}=f(s_{t},a_{t}) \tag{18}\]  
During state transitions, the system must adhere to the following constraint:  
\[y_{i}(t+1)\geq y_{i}(t)-\Delta y_{i}(t) \tag{19}\]  
where \(\Delta y_{i}(t)\) represents the state change of node \(i\) at time \(t\). The optimal decision problem for dynamic resource scheduling and task offloading is formulated as a constrained optimization problem. By incorporating the state transition equations and reward function, the problem is expressed as  
\[V^{*}(s_{t})=\max_{a_{t}\in\mathbb{A}}\Biggl{[}r_{t}(s_{t},a_{t})+\gamma\sum_ {s_{t+1}\in\mathbb{S}}P(s_{t+1}|s_{t},a_{t})V^{*}(s_{t+1})\Biggr{]} \tag{20}\]  
where \(V^{*}(s_{t})\) is the optimal value function for state \(s_{t}\), and \(\gamma\) is the discount factor representing the importance of future rewards. We iteratively update the value function until convergence:  
\[V^{*}_{t+1}(s_{t})=\max_{a_{t}\in\mathbb{A}}\Biggl{[}r_{t}(s_{t},a_{t})+\gamma \sum_{s_{t+1}\in\mathbb{S}}P(s_{t+1}|s_{t},a_{t})V^{*}_{t}(s_{t+1})\Biggr{]} \tag{21}\]  
where \(V^{*}_{t}(s_{t})\) is the value function at iteration \(t\). The optimal policy \(\pi^{*}(s_{t})\) is a mapping from each possible state \(s_{t}\) to an optimal action \(a_{t}\in\mathbb{A}\). Its goal is to maximize the long-term cumulative reward, typically achieved by solving the Bellman equation. Specifically, the optimal policy \(\pi^{*}(s_{t})\) is defined as  
\[\pi^{*}(s_{t})=\arg\max_{a_{t}\in\mathbb{A}}\Biggl{[}r_{t}(s_{t},a_{t})+\gamma \sum_{s_{t+1}\in\mathbb{S}}P(s_{t+1}\mid s_{t},a_{t})V^{*}(s_{t+1})\Biggr{]} \tag{22}\]  
where \(r_{t}(s_{t},a_{t})\) is the immediate reward obtained by taking action \(a_{t}\) in state \(s_{t}\). \(\gamma\) is the discount factor controlling the weight of future rewards. \(P(s_{t+1}\mid s_{t},a_{t})\) is the probability of transitioning from state \(s_{t}\) to state \(s_{t+1}\) after taking action \(a_{t}\). \(V^{*}(s_{t+1})\) is the optimal value function at state \(s_{t+1}\). We design an iterative algorithm based on value function updates. Initially, the value function \(V_{0}(s_{t})\) is set to zero or an initial estimate, and the learning rate \(\eta\), discount factor \(\gamma\), and maximum number of iterations \(T\) are specified. During each iteration, the Bellman update is computed using the current value function and the reward associated with each action. This update provides an estimation of the optimal value for the next time step. The value function is then updated using the difference between the estimated and current values, adjusted by the learning rate:  
\[V^{*}_{t+1}(s_{t})\gets V^{*}_{t}(s_{t})+\eta\left(r_{t}(s_{t},a_{t})+ \gamma\sum_{s_{t+1}\in\mathbb{S}}P(s_{t+1}|s_{t},a_{t})V^{*}_{t}(s_{t+1})-V^{* }_{t}(s_{t})\right) \tag{23}\]  
The iteration continues until the value function converges, i.e., the change in the value function falls below a predefined threshold \(\epsilon\).