## 4 Optimization Problem Modeling
### Dynamic Task Offloading and Computation Scheduling Problem Modeling and Solving
#### 5.2.2 Determining Satellite Dynamic Link Strategy Based on DQN
Herein, \(\theta\) and \(\theta^{\prime}\) are the parameters of the prediction network and target network. The parameters \(\theta\) are updated by reducing the loss function:  
\[\theta\leftarrow\theta+\
u\
abla F_{t}(Q(s,a|*)) \tag{37}\]  
where \(\
u\) signifies the step size, and \(F_{t}(Q(s,a|*))\) denotes the Q-value loss function, which is expressed as follows:  
\[F_{t}(Q(s,a|*)))=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim U(D)}\Big{[}(y_{ t}-Q(s,a;\theta))^{2}\Big{]} \tag{38}\]  
In Equation (38), \((s_{t},a_{t},r_{t},s_{t+1})\sim U(D)\) symbolizes samples randomly extracted from the experience replay buffer \(D\), and \(y_{t}\) is the output generated by the target network at time \(t\), which is given by the following:  
\[y_{t}=r_{t}+\gamma\max_{a^{\prime}}Q^{\prime}(s^{\prime},a^{\prime};\theta^{ \prime}) \tag{39}\]  
The detailed procedures for the computation task offloading and scheduling algorithm, rooted in the DQN learning algorithm for interactions between satellite and IoT devices, are outlined in Algorithm 1. Begin by initializing the prediction and target networks alongside the state, action, and parameters within the experience replay buffer. Within the state \(s_{t}\), implement the \(\epsilon\)-greedy policy to identify the optimal action \(a_{t}\) for the target network, facilitating the transition of the system state from \(s_{t}\) to \(s_{t+1}\). Record the experience tuple \((s_{t},a_{t},r_{t},s_{t+1})\) in the experience replay buffer, whose maximum capacity is denoted as \(D^{\text{max}}\). Select a random batch of samples with size \(D^{\text{batch}}\) from this buffer and feed them into the prediction network to generate predicted Q-values. The target Q-values are computed in line with Equation (36). Then, determine the discrepancy between the predicted and target Q-values to update the parameters of the prediction network via Equation (37). The target network parameters are periodically revised utilizing the prediction network's parameters. Iterate these steps until convergence is reached. We define'resource scheduling' in the context of our study as the strategic scheduling and management of computational and communication resources within the satellite network. This encompasses both data scheduling, which prioritizes data packet transmission, and task offloading, which involves distributing computational tasks among various network nodes. Although our primary focus is on optimizing data scheduling paths and task offloading strategies, these actions are intrinsically linked to efficient resource utilization, ensuring that the network operates optimally in conjunction with its available resources. ```
1:Initialize prediction network \(Q(s,a;\theta)\) and target network \(Q^{\prime}(s,a;\theta^{\prime})\)
2:Initialize experience replay buffer \(D\) with capacity \(D^{\text{max}}\)
3:for each episode do
4: Initialize state \(s_{t}\)
5:for each step in episode do
6: Select action \(a_{t}\) using \(\varepsilon\)-greedy policy
7: Execute action \(a_{t}\) and observe reward \(r_{t}\) and next state \(s_{t+1}\)
8: Store \((s_{t},a_{t},r_{t},s_{t+1})\) in \(D\)
9: Sample random batch of size \(D^{\text{batch}}\) from \(D\)
10: Compute target \(y_{t}=r_{t}+\gamma\max_{a^{\prime}}Q^{\prime}(s_{t+1},a^{\prime};\theta^{ \prime})\)
11: Compute loss \(F_{t}=\mathbb{E}[(y_{t}-Q(s_{t},a_{t};\theta))^{2}]\)
12: Update prediction network parameters \(\theta\leftarrow\theta+\
u\
abla_{\theta}F_{t}\)
13:if step mod update_frequency == 0 then
14: Update target network parameters \(\theta^{\prime}\leftarrow\theta\)
15:endif
16:\(s_{t}\gets s_{t+1}\)
17:endfor
18:endfor
```  
**Algorithm 1** DQN-Based Task Offloading and Scheduling