## 3 Intelligent Integrated Navigation Q-Learning FEKF Algorithm  
### Navigation Information Fusion with the Q-Learning  
In the integrated navigation scheme of the SSBE based on pulsar and solar/target planetary objects presented in the previous section, the state equation and the observational equation of the system are both nonlinear. The orbital dynamics model of the cruise phase for the SSBE shown as Equation (2) is adopted as the state transition model to forecast the next state of the probe, and it can be seen that both the X-ray pulse and solar/target planetary object Doppler velocity measurements are adopted as the measurement model for the navigation system. The EKF algorithm has a good performance for optimal state estimation for nonlinear navigation systems. The FEKF based on RL is adopted to evaluate the position and velocity for the navigation system integrating X-ray pulsar and two-dimensional Doppler velocity measurements in this research. However, the state estimation performance of the conventional federated EKF depends on the suitable state and observation noise error covariance matrices. Based on this, we attempt to combine the reinforcement \(Q\)-learning with the federated EKF, which searches for the best state and observation noise error covariance through continuous interactive trial-and-error with the environment and optimizes the error of state estimation by the QLFEKF algorithm. It can improve the navigation error precision for the probe cruise phase during long orbit run. One of the RL methods that is most frequently applied is \(Q\)-learning [50; 51; 52]. With the development of intelligent autonomous navigation research, \(Q\)-learning has been applied more and more in the navigation system design [39; 41]. Since the \(Q\)-learning is essential for adaptive trial-and-error optimization of the state and observation noise error covariance matrices, we briefly describe the fusion of enhanced \(Q\)-learning and navigation information. The intelligent information interaction with the operating environment for the probe agent (PA) is presented in Figure 3. The real-time in-orbit flight environment status for the probe is characterized by the state \(s\). The various measuring instruments have their own state in the flight environment. The environment-based state of the pulsar detector is \(s_{1}(t)\), and the environment-based state of Doppler velocimetry spectrometer is \(s_{2}(t)\). The on-orbit state prediction of the PAs is characterized by the Doppler velocity spectrometer and the X-ray pulsar detector. The PA moves from the current states \(s_{1}(t)\) and \(s_{2}(t)\) to the next state and takes the corresponding actions \(a_{1}(t)\) and \(a_{2}(t)\), while receiving the corresponding rewards \(R_{1}(t)\) and \(R_{2}(t)\) from the environmental information as feedback for performing the corresponding actions in that state. In the current operational environment state in-orbit, the reward represents the good or bad action performed by the PA to choose the next state. The PA updates its state, selects the best action in real-time according to the \(\varepsilon\)-greedy strategy, and calculates the cumulative reward. The state and observation noise matrices \(\mathbf{Q}_{k}\left(t_{1}\right)\) and \(\mathbf{R}_{k}\left(t_{1}\right)\), respectively, are used as the observed state \(s_{k}\left(t_{1}\right)\) for the integrated navigation sub-filter EKF in the SSBE cruise phase. When the PA enters the next operational environment state (performing action \(a_{k}\left(t_{2}\right)\)), the \(\mathbf{Q}_{k}\left(t_{1}\right)\) and \(\mathbf{R}_{k}\left(t_{1}\right)\) are transferred to \(\mathbf{Q}_{k}\left(t_{2}\right)\) and \(\mathbf{R}_{k}\left(t_{2}\right)\) (current state \(s_{k}\left(t_{2}\right)\)). The reward could be generated during the process of state transition, evaluating the PVSE errors during the cruise phase for the probe. The positive reward could reinforce the action to choose the more proper observation noise covariance error matrix.