## 3 The Proposed DRL + GNN-Based Handover Scheme
### DQN Framework  
Deep reinforcement learning (DRL) is an approach that integrates reinforcement learning with deep learning techniques. Through learning via interactions with the environment, AI can adaptively make decisions to maximize an objective function in an optimization problem. The DRL framework defines state spaces and action spaces, represented by sets of states (S) and actions (A), respectively, along with reward functions. DRL agents then seek the optimal policy by iteratively exploring the state and action spaces. The objective of Q-learning [38] is to facilitate the learning of a policy \(\pi:\)S\(\rightarrow\)A by an intelligent agent. The algorithm generates a Q-table comprising all feasible combinations of states and actions, and throughout the training process, the intelligent agent updates the Q-values in the table according to the rewards obtained from action selection. The formula for updating Q values during the training process in Q-learning is obtained from the Bellman equation:  
\[Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha(R(s_{t},a_{t})+\gamma\text{max}_{a_{t+1}}Q(s _{t+1},a_{t+1})-Q(s_{t},a_{t})) \tag{17}\]  
where Q(\(s_{t}\), \(a_{t}\)) is the Q-value function at time \(t\), \(\alpha\) For the learning rate, \(R(s_{t}\), \(a_{t})\) is the reward value for taking action \(a_{t}\) in state \(s_{t}\). Deep Q-Network (DQN) [39] is an enhanced algorithm derived from Q-learning, integrating a deep neural network (DNN) to approximate the \(q\)-value function. The Q-table in q-learning stores the associations between \(q\)-values, states, and actions. As the Q-table grows in size, q-learning encounters challenges in learning optimal policies within high-dimensional state and action spaces. Consequently, DQN employs Q-networks in lieu of Q-tables and depends on the generalization capability of DNNs to estimate the Q-values for states and actions that are not pre-defined. The implementation process of the DQN algorithm is shown in Figure 6.  
DQN comprises two \(Q\) networks with identical structures: one is the Q estimation network employed for action selection, denoted as \(Q(s,a;\theta)\); the other is the \(Q\) target network utilized for training, denoted as \(Q(s,a;\theta^{-})\). \(\theta\) and \(\theta^{-}\) represent the parameters of the Q estimation network and the Q-target network, respectively. Furthermore, throughout DQN training, the intelligent agents utilize an experience replay buffer \(D\) to retain past experiences (i.e., tuples containing \(s_{t}\), \(a_{t}\), \(r_{t}\), \(s_{t+1}\)). For each training process, a certain number of experiences are randomly selected from \(D\) as a small batch, and the loss function for training DQN is defined as:  
\[L(\theta)=E_{<s_{t}a_{t}r_{t}s_{t+1}>\in D}[(r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+ 1},a_{t+1};\theta^{-})-Q(s_{t},a_{t};\theta))^{2}] \tag{18}\]  
where \(r_{t}\) denotes the immediate reward value and \(\gamma\) denotes the discount factor. Combined with the research scenario of this paper, there are three key elements in our proposed DQN algorithm, which are state space, action space, and reward function, and the specific descriptions of the three elements are shown below. * State space  
The system state space comprises information regarding the environment perceived by the intelligent agent, encompassing the changes in the environment following the execution of the agent's actions. From a reinforcement learning perspective, the system state space serves as the foundation for intelligent agents to make decisions and assess their long-term rewards. Consequently, the design of the system state space directly influences the final performance and convergence speed of the DQN algorithm. The entire communication duration is partitioned into \(k\) equal time slots, and the satellite network state at different time slots is denoted as \(S(t)\), aiming to define the state  
Figure 6: Process of DQN algorithm. space as finite.