## 5 Problem Solving and Algorithm Designing  
Typically, conventional optimization solvers are employed to solve problems with deterministic variables [29]. Problems with random variables are difficult to be solved using these solvers. Nevertheless, the tripartite matching problem that this paper focuses on is a mixture of stochasticity and determinism. Therefore, we adopt reinforcement learning to dynamically solve the LEO-drone matching sub-problem (19a). Specifically, the matching between each LEO satellite and HAP drones is modeled as a Markov decision process [30], where each LEO satellite is treated as an agent. The state, action, and reward of the \(j\)-th LEO satellite are defined as follows:  
* State: \(s_{j}[m]=\big{\{}\big{\{}T_{i,j}[m]|i\in\mathcal{I}\big{\}},\big{\{}R_{i,j}[m ]|i\in\mathcal{I}\big{\}},\) \(\big{\{}S_{i,q}[m-1]|i\in\mathcal{I},q\in\mathcal{Q}\big{\}},\big{\{}G_{i,q}[m -1]|i\in\mathcal{I},q\in\mathcal{Q}\big{\}}\big{\}}\). In the \(m\)-th time slot, the \(j\)-th LEO satellite obtains the state of each HAP drone within its visible range, which includes the available traffic transmission time \(T_{i,j}[m]\) and the traffic transmission rate \(R_{i,j}[m]\) of the current \(m\)-th time slot, as well as the stored traffic volume \(S_{i,q}[m-1]\) and the traffic generation rate \(G_{i,q}[m-1]\) of each traffic type in the previous \((m-1)\)-th time slot. For the HAP drone \(i_{0}\), which is not within the visible range of the \(j\)-th LEO satellite, i.e., \(i_{0}\
otin\mathcal{I}_{j}[m]\), there are \(T_{i_{0},j}[m]=0\), \(R_{i_{0},j}[m]=0\), \(S_{i_{0},q}[m-1]=0,\forall q\in\mathcal{Q}\), and \(G_{i_{0},q}[m-1]=0,\forall q\in\mathcal{Q}\). * Action: \(a_{j}[m]=\big{\{}x_{i,j}[m]|\sum_{i=1}^{l}x_{i,j}[m]=L\big{\}}\). In the \(m\)-th time slot, the action of the \(j\)-th LEO satellite is to determine which \(L\) HAP drones to provide services for. If multiple LEO satellites decide to provide services to the same HAP drone, this HAP drone will actively choose to connect to the LEO satellite, transmitting the highest traffic value. * Reward: \(r_{j}\big{(}s_{j}[m],a_{j}[m]\big{)}=\sum_{i=1}^{l}\sum_{q=1}^{Q}\mu_{q}U_{i,q _{j}}[m]\). In the \(m\)-th time slot, the reward obtained by the \(j\)-th LEO satellite after taking action \(a_{j}[m]\) in state  
where \(t\in\{1,2,\cdots,T_{\max}\}\). \(s_{j}^{t}\) represents the state at the \(t\)-th step of the episode, and \(a_{j}^{t}\) denotes the action taken in state \(s_{j}^{t}\). \(\theta_{j}^{t}\) represents the learning rate, and \(\mathcal{A}_{j}\) denotes the action space of the \(j\)-th LEO satellite. \(\overline{r}_{j}^{t}\big{(}s_{j}^{t},a_{j}^{t}\big{)}\) denotes the average one-step immediate reward acquired after taking action \(a_{j}^{t}\) in state \(s_{j}^{t}\), which can be represented as follows:  
\[\overline{r}_{j}^{t}\Big{(}s_{j}^{t},a_{j}^{t}\Big{)}=\mathbb{E}\Big{[}r_{j}( s,a)|s=s_{j}^{t},a=a_{j}^{t}\Big{]}. \tag{23}\]  
Supposing that the proposed approach converges after \(C\) iterations, the optimal policy can be expressed as follows [32]:  
\[\pi_{j}^{*}\big{(}a|s_{j}\big{)}= \begin{cases}1,a=\operatorname*{arg\,max}_{a \in A_{j}}q_{j}^{C}\big{(}s_{j},a\big{)}\\ 0,\text{otherwise}\end{cases} . \tag{24}\]  
The aforementioned conventional Q-learning algorithm stores the calculated Q-values \(q_{j}^{t}\Big{(}s_{j}^{t},a_{j}^{t}\Big{)}\) in the form of tables, known as a Q-table, which has the advantages of being intuitive and easy to analyze. However, due to the continuous state space of (19a), using the conventional tabular Q-learning algorithm requires storing a large volume of data, thereby increasing storage costs. Furthermore, the generalization ability of the conventional Q-learning algorithm is poor. To address these issues, a deep Q-learning algorithm is employed in this paper, which is one of the earliest and most successful algorithms that introduces deep neural networks into reinforcement learning [32].