## 3 Intelligent Integrated Navigation Q-Learning FEKF Algorithm
### Navigation Information Fusion with the Q-Learning
The cumulative update rules for the reward \(R(s_{k},a_{k})\) are as follows:  
\[R(s_{k},a_{k})=R(s_{k-1},a_{k-1})+\frac{1}{TtC\gamma}\Bigg{\{}\left[\left( \tilde{\mathbf{y}}_{ps,k}^{Tr}\right)^{T}\left(\tilde{\mathbf{R}}_{k}\right)^{-1} \tilde{\mathbf{y}}_{ps,k}^{Ex}\right]^{-1}-R(s_{k-1},a_{k-1})\Bigg{\}} \tag{21}\]  
where \(R(s_{k-1},a_{k-1})\) is the cumulative reward after the selection action performed in the previous state. \(ItC\gamma\) is the learning reward iteration cycle. \(\tilde{\mathbf{R}}_{k}\) is the observation noise error covariance matrix corresponding to the current state. \(\left(\tilde{\mathbf{y}}_{ps,k}^{Tr}\right)^{T}\) and \(\tilde{\mathbf{y}}_{ps,k}^{Ex}\) are the deviation (measurement residual) between the actual observed value and observed update value obtained by the stateful estimation calculation for the conventional EKF and searching EKF, respectively, and are closely associated with the observation noise error covariance adopted for the QLFEKF. The accumulation of rewards is assessed using the \(Q\)-value. The reward is updated in each iteration cycle in the \(Q\)-learning algorithm, and the \(Q\)-value also is updated accordingly. The updating rules of the \(Q\)-value are expressed as follows:  
\[Q(s_{k},a_{k})=(1-\alpha)Q(s_{k},a_{k})+\alpha\bigg{[}R(s_{k},a_{k})+\gamma \underset{a_{k+1}}{\text{max}}Q(s_{k+1},a_{k+1})\bigg{]} \tag{22}\]  
where \(Q\left(s_{k},a_{k}\right)\) is the performance of performing action \(a_{k}\) based on flight environment state \(s_{k}\) for the agent. \(\alpha\) is the learning rate, \(\alpha\in(0,1]\). \(\gamma\) is the discount factor, \(\gamma\in(0,1]\). Each time the state \(s\) changes, the corresponding action a is selected to obtain the corresponding cumulative reward \(R\), and a corresponding \(Q\)-value is calculated. The \(Q\)-value represents the superiority of choosing action \(a\) in state \(s\). When the \(Q\)-value corresponding to action a is greater than the \(Q\)-value corresponding to other actions that can be selected in this state, action \(a\) is the optimum selection for the current state. The \(Q\)-table is introduced to combine \(Q\)-learning with the traditional navigation filtering method. By constructing collections for state \(\mathbf{S}\) and collections for actions \(\mathbf{A}\), the \(Q\)-learning table with the grid specification \(M\times N\) is established. The \(Q\)-table is utilized to determine the appropriate noise covariance matrix to enhance the efficiency of the in  
Figure 3: Intelligent information interaction with the flight environment for the PA.  
telligent integrated navigation system during the cruise phase. The PA tries to search the optimal state and observation noise covariance based on the \(Q\)-value calculated by the cumulative reward to optimize the federated filter. The set of states and corresponding action sets for the QLFEEF are shown in Figure 4. The process noise error covariance matrix is denoted as \(\mathbf{Q}_{k}^{(i)}=\left\{\mathbf{Q}_{rk}^{(i)},\mathbf{Q}_{sk}^{(i)}\right\}\), \(i=1,\,2,\,\ldots,\,m\). \(\mathbf{Q}_{rk}^{(i)}\) and \(\mathbf{Q}_{pk}^{(i)}\) represent the position and velocity state noise covariance information, respectively. The measurement noise error covariance matrix is denoted as \(\mathbf{R}_{k}^{(j)}=\left\{\mathbf{R}_{pk}^{(j)},\mathbf{R}_{dk}^{(j)}\right\}\)\(j=1,\,2,\,\ldots,\,n\). \(\mathbf{R}_{pk}^{(j)}\) and \(\mathbf{R}_{dk}^{(j)}\) represent the measurement error covariance information of the pulsar detectors and Doppler spectrometers, respectively. The state set of the \(Q\)-table is represented as \(\mathbf{S}^{(ij)}=\left\{\mathbf{Q}_{rk}^{(i)},\mathbf{Q}_{pk}^{(i)},\mathbf{R}_{pk}^{(j)},\bm {R}_{dk}^{(j)}\right\}\), which is distributed in a grid of \(M\) rows and \(N\) columns.