## 3 Intelligent Integrated Navigation Q-Learning FEKF Algorithm
### Design of the \(Q\)-Learning-Based FKF for Intelligent Integrated Navigation
\(\hat{\mathbf{X}}_{ps,k}^{Se(p)}\) and \(\hat{\mathbf{X}}_{ps,k}^{Se(d)}\) are the state estimates for the measurement updated EKF for the X-ray pulsars sub-filters and Doppler sub-filters, respectively. \(\mathbf{P}_{ps,k}^{Se(p)}\) and \(\mathbf{P}_{ps,k}^{Se(d)}\) are the state transition matrices corresponding to the measurement update EKF, respectively. \(\widehat{\mathbf{y}}_{ps,k}^{Se(p)}\) and \(\widehat{\mathbf{y}}_{ps,k}^{Se(d)}\) are the measurement innovation information corresponding to the measurement update EKF, respectively. In step 19, information fusion is performed on the state estimation of the sub-filter based on the X-ray pulsar, the sub-filter based on Doppler and the main filter. The detailed design process of the federated filtering algorithm in steps 16-19 are given below. The cumulative rewards are gained at the end of the iteration cycle. The value of the \(Q\)-value is calculated according to the cumulative reward from the innovation information of filtering, and the corresponding \(Q\)-value for each set of the state-action pair (\(s_{k},a_{k}\)) is stored in the \(Q\)-table. According to the \(Q\)-values corresponding to different actions in the current state, the best action is selected to reach the next best state, and the corresponding selection locates the state and observation noise variance matrices. The structure diagram of the \(Q\)-learning-based FEKF intelligent integrated navigation system is shown in Figure 5. The observation information of the X-ray pulsar and Doppler velocity-measuring is fused by the QLFEKF. The reinforcement \(Q\)-learning-based EKF is used as the sub-filter for the X-ray pulsar and Doppler velocity-measuring navigation system, and each sub-filter is composed of the benchmark filter, searching filter, and state estimation learning filter. The two sub-filters work in parallel, the output locally optimal state estimates \(\hat{\mathbf{X}}_{ps,k}^{Se(p)}\) and \(\hat{\mathbf{X}}_{ps,k}^{Se(d)}\) as well as the error covariance matrices \(\mathbf{P}_{ps,k}^{Se(p)}\) and \(\mathbf{P}_{ps,k}^{Se(d)}\), which are used as inputs to the global master-filter for information fusion periodically. Finally, the global optimal state estimation \(\hat{\mathbf{X}}_{ps,k}^{Se}\) and error covariance \(\mathbf{P}_{ps,k}^{S}\) are obtained. The global primary filter system performs fusion estimation according to the estimation information of the sub-filter system and assigns the fusion estimation information to the sub-filter according to the design weight by using the information distribution principle. The single-step intelligent federated filtering process is realized to obtain the current state estimation information. The information distribution process is that the global main filter and two sub-filters allocate state estimation information according to a certain weight. The information distribution principle between the main filter and each sub-filter is as follows:

\[\left\{ \begin{array}{l}\hat{\mathbf{Q}}_{nk}^{(i)}=\beta_{n}^{-1}\hat{\mathbf{Q}}_ {k}^{(i)},\hat{\mathbf{Q}}_{k}^{m}=\beta_{m}^{-1}\hat{\mathbf{Q}}_{k}\\ \mathbf{P}_{ps,k}^{Se(n)}=\beta_{n}^{-1}\mathbf{P}_{ps,k}^{s},\mathbf{P}_{ps,k}^{m}=\beta_ {m}^{-1}\mathbf{P}_{ps,k}^{g}\\ \hat{\mathbf{X}}_{ps,k}^{Se(n)}=\hat{\mathbf{X}}_{ps,k}^{g},\hat{\mathbf{X}}_{ps,k}^{m}=\hat {\mathbf{X}}_{ps,k}^{g}\end{array} ,n=p,d\right. \tag{23}\]

where \(\beta_{m}\) and \(\beta_{n}\) are the information distribution coefficients of the parallel sub-filter and primary filter, respectively, \(\beta_{m},\beta_{n}\in(0,1]\), \(n=1,2,\cdots\). Here the principle of conservation of the information distribution is satisfied, \(\sum\limits_{i}^{n}\beta_{m}+\beta_{n}=1\).