## 5 Problem Solving and Algorithm Designing
In deep Q-learning, the high-dimensional Q-table can be approximated by a deep Q network with low-dimensional parameters, thereby significantly reducing the storage cost. In addition, the Q-values of unvisited state-action pairs can be calculated through value function approximation, giving it strong generalization ability. In addition, the aforementioned algorithm is fully decentralized, in which each satellite calculates its Q-values according to its own local states, local actions, and local rewards. However, LEO satellites are not completely independent, but influence each other. For example, if the \(i\)-th HAP drone is connected to the \(j\)-th LEO satellite at the current moment, other LEO satellites cannot provide service for this HAP drone. Therefore, the aforementioned fully decentralized reinforcement learning algorithm cannot obtain high performance and may not even converge in some cases. An alternative solution is to use a fully centralized reinforcement learning algorithm. In each time slot, each LEO satellite sends its experience obtained from its interaction with the environment to the terrestrial traffic center. Then, both value network training and strategy making are performed at the center based on global experiences. Nevertheless, the experience of each satellite must pass through multiple ISLs, an LEO-ground link, and an optical fiber link to be transmitted back to the terrestrial traffic center, facing high propagation latency. The terrestrial traffic center is unable to obtain the latest status of the space-air-ground integrated network, so it is unable to make timely LEO-drone matching strategies. To address these issues, we employ multi-agent deep reinforcement learning with centralized training and decentralized execution. The value network of each LEO satellite is trained in a centralized manner at the terrestrial traffic center. Then, the trained value networks are distributed to the corresponding LEO satellites [33]. Each satellite distributively trains its policy network based on the received value network and the latest local observations, thus it can devise LEO-drone matching strategies in a timely manner. Specifically, when training the value network, each LEO satellite sends its local experience \(\left\{\left(s_{j},a_{j},r_{j},s^{\prime}_{j}\right)\right\}\) obtained from its interaction with the environment to the terrestrial traffic center, where \(s^{\prime}_{j}\) is the state reached after taking action \(a_{j}\) in state \(s_{j}\). Based on the collected local experiences of various LEO satellites, the terrestrial traffic center forms the global experience, including the global state \(\mathbf{s}=\left[s_{1},s_{2},\cdots,s_{I}\right]\), the global action \(\mathbf{a}=\left[a_{1},a_{2},\cdots,a_{I}\right]\), and the global reached state \(\mathbf{s}^{\prime}=\left[s^{\prime}_{1},s^{\prime}_{2},\cdots,s^{\prime}_{I}\right]\), and stores \(\left\{\left(\mathbf{s},\mathbf{a},r_{j},\mathbf{s}^{\prime}\right)\right\}\) in the replay buffer \(D_{j}\). Afterwards, the terrestrial traffic center trains the value network of the \(j\)-th LEO satellite based on \(\left\{\left(\mathbf{s},\mathbf{a},r_{j},\mathbf{s}^{\prime}\right)\right\}\) to evaluate the quality of the matching approach. As previously mentioned, the deep Q-learning algorithm is adopted, where the true Q-values of the optimal strategy are approximated by the Q-values calculated by the trained value network, which can be obtained through the quasi-static target network scheme [34]. Specifically, two networks need to be defined: the target network \(\widehat{q}_{j}\big{(}S,A,\omega_{j,target}\big{)}\) and the main network \(\widehat{q}_{j}\big{(}S,A,\omega_{j,main}\big{)}\) described by parameters \(\omega_{j,target}\) and \(\omega_{j,main}\), respectively, where \(S\) and \(A\) are global states and global actions collected by the terrestrial traffic center in the form of random variables.