## 3 Intelligent Integrated Navigation Q-Learning FEKF Algorithm
### Design of the \(Q\)-Learning-Based FKF for Intelligent Integrated Navigation
According to the approximate calculation formula of the pulsar observation noise [53; 54], the standard deviations of the measurement noise for three X-ray pulsars are \(p_{1}\) = 0.42060 km, \(p_{2}\) = 0.14070 km, and \(p_{3}\) = 0.44488 km, respectively. The initial conditions of the relevant parameters for the X-ray pulsars and solar/target planetary Doppler measurement model that would be adopted in the simulation are listed in Table 3 and mainly include the relevant initial measurement parameter characteristics of the X-ray pulsar detectors and Doppler spectrometers. \begin{table}
\begin{tabular}{c c c} \hline
**Simulation Conditions** & **Parameters Value** \\ \hline Initial position values & [53,107,871.00559; 137,626,318.36608; \(-\)10,143.24541] km \\ Initial velocity values & [\(-\)14.01219; 35,86413; 0.42128] km/s \\  & \(\delta X_{0}=[\delta r_{x},\delta r_{y},\delta r_{z},\delta v_{x},\delta v_{y}, \delta v_{z}]^{T}\) \\ Initial state errors & \(\delta r_{x}=\delta r_{y}=\delta r_{z}=1.0\times 10^{2}\) km, \\  & \(\delta v_{x}=\delta v_{y}=\delta v_{z}=2.0\times 10^{-3}\) km/s \\ Initial state estimation covariance & \(P_{0}=diag\Big{(}\left|\delta r_{x}^{2},\delta r_{y}^{2},\delta r_{z}^{2}, \delta v_{x}^{2},\delta v_{y}^{2},\delta v_{z}^{2}\right|\Big{)}\Big{)}\) \\ Initial state process noise covariance & \(Q_{0}=diag\big{(}[q_{1}^{2},q_{1}^{2},q_{2}^{2},q_{2}^{2},q_{2}^{2}]\big{)},\)\(q_{1}=2.0\times 10^{-3}\) km; \\  & \(q_{2}=3.0\times 10^{-6}\) km/s \\ \hline \end{tabular}
\end{table} 
Table 1: Initial state orbit parameters for the probe. \begin{table}
\begin{tabular}{c c c c c c c c} \hline
**Pulsars Name** & **RAD (\({}^{\circ}\))** & **DEC (\({}^{\circ}\))** & **D\({}_{\mathbf{0}}\) (kpc)** & **P (10\({}^{-3}\) s)** & **W (10\({}^{-3}\) s)** & **Fx (ph/cm\({}^{2}\)/s)** & **Pf (\%)** \\ \hline PSR B1937+21 & 294.92 & 21.580 & 3.6 & 1.56 & 0.021 & \(4.99\times 10^{-5}\) & 86 \\ PSR B0531+21 & 83.63 & 22.014 & 2.0 & 33.5 & 1.670 & 1.54 & 70 \\ PSR J1821-24 & 276.13 & \(-\)24.870 & 5.5 & 3.05 & 0.055 & \(1.93\times 10^{-4}\) & 98 \\ \hline \end{tabular}
\end{table} 
Table 2: The corresponding parameters of the selected X-ray pulsars. In the QLFEKF algorithm, reinforcement \(Q\)-learning is adopted to tune the parameters of the state and observation noise error covariance matrices and to obtain the optimum state and observation noise error covariance matrices. The parameters tuned by the reinforcement \(Q\)-learning include \(\alpha\), \(\gamma\), and \(\varepsilon\). Set the size of the appropriate \(Q\)-learning grid to \(M\times N\) = 20 \(\times\) 20. In the case of the specific range of the \(\hat{\textbf{Q}}_{rk}^{(i)}\), \(\hat{\textbf{Q}}_{ck}^{(i)}\), \(\hat{\textbf{R}}_{pk}^{(j)}\), and \(\hat{\textbf{R}}_{dk}^{(j)}\), the larger scale grid could search more fully and cover more area. The root mean squared error (RMSE) [54] is adopted as the evaluation indicator to implement precision analysis of the PVSE for the probe's cruise phase. The relevant design parameters for the QLFEKF algorithm are summarized in Table 4 as follows. ### Simulation and Analysis

The X-ray pulsar/solar and target planetary Doppler velocity measurement based the \(Q\)-learning federation EKF (STD/XP-QLFEKF) is compared with the navigation state estimates of precision of the X-ray pulsar/solar and target planetary Doppler velocity measurement-based EKF (STD/XP-EKF) and the X-ray pulsar/solar and target planetary Doppler velocity measurement-based federated EKF (STD/XP-FEKF). The RMSE of the PVES of the cruise phase for the probe is used as the evaluation indicator of the navigation accuracy. Figures 6 and 7 show the RMSE curves of the position estimation error and the three-axis position estimation error of the three integrated navigation algorithms.