## 3 Intelligent Integrated Navigation Q-Learning FEKF Algorithm
### Design of the \(Q\)-Learning-Based FKF for Intelligent Integrated Navigation
\(\alpha\) represents the learning rate of the \(Q\)-learning, \(\gamma\) represents the learning rate of the \(Q\)-learning, and \(\varepsilon\) represents the action selection probability of the \(Q\)-learning, which can be applied to regulate the performance of the learning for the reinforcement \(Q\)-learning. The specific time of the navigation in step 3 refers to the navigation time for the cruise phase. In step 7, the state process (including PVSE) and observation (including the X-ray pulsar detector and two Doppler velocity-measuring spectrometers) noise error covariance matrix are randomly generated within the set range corresponding to the initial values of the noise error covariance matrix. \(\underline{q}_{rk}\) and \(\overline{q}_{rk}\) are the upper edge and lower edge bounds for the position state noise covariance matrix of the system error, respectively. \(\underline{q}_{pk}\) and \(\overline{q}_{rk}\) are the upper edge and lower edge bounds for the velocity state noise covariance matrix of the system error, respectively. The \(\varepsilon\)-greed strategy is involved in step 8, which is determined by the cumulative rewards \([R^{(p)}(s_{k},a_{k}),R^{(d)}(s_{k},a_{k})]\) and \(Q\)-values \([Q^{(p)}(s_{k},a_{k}),Q^{(d)}(s_{k},a_{k})]\) calculated from different state \(s_{k}\) and action \(a_{k}\). The PA selects the random action according to the probability of choice action \(\varepsilon\) and selects the action to maximize the \(Q\)-value with the probability of the \((1-\varepsilon)\). The learning strategy has good stability for the PA to select the most suitable noise covariance matrix. \(ItCy\) is the period of time for the calculation of cumulative rewards. If \(ItCy\) is small, the weight and cumulative rewards for each update will be large, so the design of the \(ItCy\) should not be too small. In step 12, \(\hat{\mathbf{X}}_{ps,k}^{Tr(p)}\) and \(\hat{\mathbf{X}}_{ps,k}^{Tr(d)}\) represent the state estimate values corresponding the sub-filter in the benchmark EKF for the X-ray pulsar and Doppler, respectively. \(\mathbf{P}_{ps,k}^{Tr(p)}\) and \(\mathbf{P}_{ps,k}^{Tr(d)}\) represent the state transition matrix corresponding to the benchmark EKF, respectively. \(\widetilde{\mathbf{y}}_{ps,k}^{Tr(p)}\) and \(\widetilde{\mathbf{y}}_{ps,k}^{Tr(d)}\) represent the measured innovation information corresponding to the benchmark EKF, respectively. Innovation information in the benchmark EKF and search EKF are important parameters of the \(Q\)-learning reward evaluation. \(\hat{\mathbf{Q}}_{rk}^{(i)}\), \(\hat{\mathbf{Q}}_{vk}^{(i)}\), \(\hat{\mathbf{R}}_{pk}^{(j)}\), and \(\hat{\mathbf{R}}_{dk}^{(j)}\) are determined according to the current state \(s_{k}\). In step 14, \(\hat{\mathbf{X}}_{ps,k}^{Ex(p)}\) and \(\hat{\mathbf{X}}_{ps,k}^{Ex(d)}\) are state estimates for the searching EKF in the X-ray pulsars sub-filters and Doppler sub-filters, respectively. \(\mathbf{P}_{ps,k}^{Ex(p)}\) and \(\mathbf{P}_{ps,k}^{Ex(d)}\) represent the state transition matrix corresponding to the searching EKF, respectively. \(\widetilde{\mathbf{y}}_{ps,k}^{Ex(p)}\) and \(\widetilde{\mathbf{y}}_{ps,k}^{Ex(d)}\) represent the measurement innovation information corresponding to the searching EKF, respectively. In step 15, the cumulative reward of the QLEKF based on the X-ray pulsar and Doppler during the iteration period is calculated, and the \(Q\)-value is determined according to the sizes of the reward \(R^{(p)}(s_{k},a_{k})\) and \(R^{(d)}(s_{k},a_{k})\). A positive reward is received for suitable \(\hat{\mathbf{Q}}_{rk}^{(i)}\), \(\hat{\mathbf{Q}}_{vk}^{(i)}\



a negative reward is received for unsuitable \(\hat{Q}_{rk}^{(i)}\), \(\hat{Q}_{rk}^{(i)}\), \(\hat{R}_{pk}^{(j)}\), and \(\hat{R}_{dk}^{(j)}\). In step 16, the weights of the X-ray pulsar, Doppler sub-filter, and the main filter are designed and distributed information. In steps 17 and 18, the corresponding time information update and measurement information update of federated filtering are mainly carried out.