## 4 Optimization Problem Modeling
### Dynamic Task Offloading and Computation Scheduling Problem Modeling and Solving
#### 5.2.2 Determining Satellite Dynamic Link Strategy Based on DQN  
Figure 3 illustrates the developed framework for task offloading and scheduling. The algorithm of DQN, introduced by the DeepMind group, is a reinforcement learning algorithm integrating deep learning principles widely used in current complex satellite networks design [19; 30; 52; 53; 54]. Unlike traditional Q-learning, DQN is adept at managing complex state spaces characterized by high dimensionality. In the realm of Q-learning, such state spaces often pose dimensionality challenges, impeding effective learning. DQN circumvents these limitations by employing deep neural networks as function approximators, which allows for the efficient handling of high-dimensional spaces by representing states as continuous variables, thus enhancing learning efficacy. Moreover, conventional Q-learning mandates a discrete action space, thereby restricting its utility in scenarios necessitating continuous action spaces. DQN addresses this limitation by leveraging deep neural networks to estimate the Q-values for each possible action, thus enabling the handling of continuous action spaces and the selection of the optimal action via optimization techniques [55]. The integration of an experience replay mechanism permits the storage of the agent's experiences in a replay buffer, from which samples are randomly extracted for training. This approach alleviates the correlation between successive training data, thereby mitigating the sample correlation issue and bolstering training efficiency and stability. Consequently, the DQN method is proficient at maximizing interactions between intelligent agents and stochastic environments. The 'Environment' encapsulates the temporal dynamics that influence the state, allowing the model to maintain a distinction between the current conditions and the actions taken, until such action results in a transition to a 'Next State'. Figure 3 illustrates the iterative process of the Deep Q-Network (DQN) learning cycle. Although the figure does not explicitly show an entry point, the process naturally begins with the first observation of the environment, denoted as the initial state \(s_{0}\). This initial state is used to determine the first action \(a_{0}\), initiating the sequence of interactions between the agent and the environment. From this point forward, the system continuously updates states and actions based on learned policies. This design aligns with conventional DQN representations, where the emphasis is on the continuous reinforcement learning process rather than a discrete starting point. During the training phase of the DQN (Deep Q-Network) algorithm, the strengths of both convolutional and recurrent neural networks are harnessed to proficiently estimate the state-action value function, known as the Q-function. The architecture of DQN comprises  
Figure 3: Proposed DQN-based task offloading algorithm framework. two distinct networks: the prediction network tasked with generating experience replay data, and the target network, which computes the state-action value, referred to as the Q-value. To expedite convergence, DQN employs an experience replay strategy, which involves the random sampling of accumulated data for training, alongside a smoothing technique for managing the data effectively. When IoT devices execute actions during the state \(s_{t}\), the associated action value function \(Q\) is updated as follows:  
\[Q_{t+1}(s,a)=Q(s,a)+\alpha\bigg{[}r(a_{t}|s_{t})+\gamma\max_{a^{\prime}}Q_{t}( s^{\prime},a^{\prime})-Q_{t}(s,a)\bigg{]} \tag{36}\]  
In Equation (36), \(\alpha\) denotes the learning rate, and \(\gamma\) represents the discount factor. To tackle the computational intricacies of Equation (36), and prevent divergence of \(Q\)-values, DQN introduces a prediction network \(Q(s,a;\theta)\) and a target network \(Q(s,a;\theta^{\prime})\), which are responsible for predicting and assessing the Q-values, respectively.