# Q-Learning-Based Noise Covariance Matrices Adaptation in Kalman Filter for Inertial Navigation

Ghadeer Shaaban, Hassen Fourati, Christophe Prieur, Alain Kibangou

► To cite this version:

Ghadeer Shaaban, Hassen Fourati, Christophe Prieur, Alain Kibangou. Q-Learning-Based Noise Covariance Matrices Adaptation in Kalman Filter for Inertial Navigation. MICNON 2024 - 4th IFAC Conference on Modelling, Identification and Control of Nonlinear Systems, Sep 2024, Lyon, France. pp.96-101, 10.1016/j.ifacol.2024.10.150. hal-04766404

HAL Id: hal-04766404

https://hal.science/hal-04766404v1

Submitted on 4 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Q-Learning-Based Noise Covariance
Matrices Adaptation in Kalman Filter for
Inertial Navigation

Ghadeer Shaaban, Hassen Fourati, Christophe Prieur,
Alain Kibangou

Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, GIPSA-lab,
Grenoble, France (e-mails: ghadeer.shaaban@gipsa-lab.fr,
hassen.fourati@gipsa-lab.fr, christophe.prieur@gipsa-lab.fr,
alain.kibangou@gipsa-lab.fr).

**Abstract:** Velocity estimation of a rigid body using measurements from low-cost inertial and magnetic sensors plays an important role in various applications. The extended Kalman filter (EKF) is widely used for this purpose. However, EKF's estimation performance relies on the knowledge of process and measurement noise covariance matrices, and this information is generally unavailable. In this work, we introduce a solution that combines two techniques: the generation of velocity pseudo-measurements using a Bidirectional Long Short-Term Memory (BiLSTM) network, and the Q-learning method for online adaptation of noise covariance matrices. The performance of the proposed solution is validated using real experimental datasets, demonstrating that Q-learning can select appropriate noise covariance matrices to enhance velocity estimation.

**Keywords:** Q-learning, velocity estimation, extended Kalman filter, covariance matrices adaptation

# 1. INTRODUCTION

Velocity estimation plays an important role in many applications such as autonomous vehicles, smart devices and positioning. In outdoor navigation, velocity sensors like air-data instruments, Doppler radar systems, or Doppler Velocity Logs offer velocity information, see Hua et al. (2014). However, in indoor navigation, these sensors lack accuracy. In Zmitri et al. (2022), a Bidirectional Long Short-Term Memory (BiLSTM) network was utilized for velocity estimation. The training data were obtained from low-cost inertial and magnetic sensors, including measurements of angular velocity, acceleration, and magnetic field. These velocity estimates are termed pseudo-measurements since they are not obtained from actual sensors. The extended Kalman filter (EKF) utilizes these velocity pseudo-measurements and data from inertial and magnetic sensors to estimate the velocity. The EKF comprises two steps: time update (prediction) and measurement update (correction). The time update step provides a prediction with a prediction error covariance matrix based on the dynamic model and the process noise covariance matrix, which expresses the model's uncertainty. The measurement update step yields the state estimation with the state estimation covariance matrix, relying on the measurements and the measurement noise covariance matrix, which expresses the measurement uncertainty. The process and measurement noise covariance matrices play a crucial role when using the EKF, and for the EKF to function correctly, knowledge of these matrices is essential. In real-world scenarios, these matrices are often unknown. The performance of

the Kalman filter may be seriously degraded under such circumstances. Additionally, these matrices may not be fixed and can vary with time.

One simple and naive approach is to manually adapt the covariance matrices in an offline mode, using available ground truth data. This process involves testing several matrices and selecting the one that produces the best results. A second simple approach involves obtaining the covariance matrices for process and measurement noises based on the statistical properties observed in extensive practical experiments. However, these two methods have drawbacks, including the considerable time required for tuning and the offline nature of the process, meaning that once the covariance matrices are chosen, they remain fixed during the estimation process. A third approach is the multiple model adaptive estimation (MMAE) approach, where a set of extended Kalman filters operates in parallel, each employing distinct covariance matrices. The best state estimation is then selected from these parallel filters, see Hanlon and Maybeck (2000). However, a drawback of this method lies in its computational complexity, which is linear with the number of parallel filters. A fourth approach is the robust extended Kalman filter (REKF), aiming to minimize the upper bound of the state estimation error for a potential set of process and measurement noise covariance matrices, see Kai et al. (2010). A fifth approach is the adaptive extended Kalman filter (AEKF), where the noise covariance matrices, along with the state vector, are simultaneously estimated, see Liu et al. (2016); Wang et al. (2017).
In Xiong et al. (2021), an algorithm called Q-learning extended Kalman filter (QLEKF) is designed, where the stability of QLEKF is investigated, and simulations showed that the QLEKF outperforms both AEKF and REKF. In this approach, process and measurement noise covariance matrices are adapted online. The adaptation process is based on the Q-learning method, a reinforcement learning technique wherein an agent iteratively learns through a trial-and-error process, utilizing rewards to evaluate state-action pairs. Q-learning also has several significant engineering applications, as demonstrated in Rizvi and Lin (2018); Maoudj and Christensen (2021). The QLEKF consists of three parallel filters. The first is a traditional EKF with nominal values for the covariance matrices. The second filter is the learning EKF, which explores various noise covariance matrices from a predefined set. The reward is calculated as the difference between the squared norm of the innovations of the first two filters. The best covariance matrices are selected which maximize the reward and are then applied to the third filter, which is the learned filter providing the final estimates. In Xiong et al. (2021), the validation of the QLEKF was conducted through a simple simulation example involving the estimation of the position and velocity of three bodies using relative distance measurements. Additionally, in Dai et al. (2022b), QLEKF was employed for attitude estimation using inertial and magnetic measurements and validated through simulations. To the best of the authors' knowledge, the adaptation of process and measurement noise covariance matrices for the EKF based on Q-learning has not been addressed within the context of velocity estimation using inertial and magnetic measurements. Furthermore, there is no literature validating the effectiveness of QLEKF using real data.

This paper aims to design a velocity estimation algorithm with noise covariance matrices adaptation based on the Q-learning method, using magnetic and inertial measurements along with velocity pseudo measurements generated by a BiLSTM network. The main contributions include:

* Successful adaptation of process and measurement noise covariance matrices with Q-learning for velocity estimation using inertial and magnetic measurement.

* Validation of the proposed algorithm using real data, demonstrating its effectiveness.

The paper is structured as follows: Section 2 provides some preliminaries and explains the dynamic model. Section 3 introduces the extended Kalman filter and the problem statement. Section 4 presents the solution as a block diagram. Section 5 explains the solution, starting by describing how BiLSTM generates the velocity pseudo-measurements, then provides preliminaries on Q-Learning, and finally, presents QLEKF algorithm. Section 6 presents the used real dataset, the implementation setup details, and the results. A conclusion in Section 7 ends the paper.

## 2. BACKGROUND AND PRELIMINARIES

### 2.1 Attitude dynamics

The attitude of a rigid body is the orientation between two frames: the body frame, attached to the moving body, and the navigation frame, fixed to the ground. The attitude can

be mathematically represented using the quaternions, and its dynamic can be described using the following kinematic equation, see Sabatini (2006):

$$ \frac{d}{dt} \mathbf{q} = \frac{1}{2} [\boldsymbol{\omega}_q \times] \mathbf{q} = \frac{1}{2} \begin{bmatrix} 0 & -\omega_x & -\omega_y & -\omega_z \\ \omega_x & 0 & \omega_z & -\omega_y \\ \omega_y & -\omega_z & 0 & \omega_x \\ \omega_z & \omega_y & -\omega_x & 0 \end{bmatrix} \mathbf{q}, \quad (1) $$

where $\boldsymbol{\omega}_q = [0 \ \omega_x \ \omega_y \ \omega_z]^T \in \mathbb{R}^4$ is the angular velocity vector written in quaternion form, and $[\boldsymbol{\omega}_q \times]$ is its skew-symmetric matrix.

，请 note that the equation is defined in the Euclidean space $\mathbb{R}^{3 \times 3}$, and the definition of the rotation matrix from the navigation frame to the body frame, denoted as $\mathbf{R}_n^b$, and vectors represented as $\mathbf{x}_n$ in the navigation frame and $\mathbf{x}_b$ in the body frame. The following relationships hold:

$$ \mathbf{x}_b = \mathbf{R}_n^b \mathbf{x}_n, \quad (2) $$

$$ \mathbf{x}_n = \mathbf{R}_b^n \mathbf{x}_b = (\mathbf{R}_n^b)^T \mathbf{x}_b. \quad (3) $$

### 2.2 Velocity and magnetic field dynamics

The velocity and magnetic field dynamics are described by the following equations, see Chesneau et al. (2016):

$$ \frac{d}{dt} \mathbf{v}_b = -[\boldsymbol{\omega} \times] \mathbf{v}_b + \mathbf{a}_b - \mathbf{R}_n^b \mathbf{g}, \quad (4) $$

$$ \frac{d}{dt} \mathbf{b}_b = -[\boldsymbol{\omega} \times] \mathbf{b}_b + \nabla \mathbf{b}_b \cdot \mathbf{v}_b, \quad (5) $$

where $\mathbf{v}_b$ and $\mathbf{b}_b$ are the body velocity and the magnetic field written in the body frame respectively, $\boldsymbol{\omega} = [\omega_x \ \omega_y \ \omega_z]^T \in \mathbb{R}^3$ is the angular velocity, and $\nabla \mathbf{b}_b$ is the magnetic field gradient written in the body frame. The total acceleration expressed in the body frame is denoted as $\mathbf{a}_b$, representing the sum of body acceleration resulting from movement and the Earth's gravity. The Earth's gravity is denoted as $\mathbf{g}$. The acceleration $\mathbf{a}_b$ is measured by a 3-axis accelerometer, the angular velocity $\boldsymbol{\omega}$ is measured by a 3-axis gyroscope, the magnetic field $\mathbf{b}_b$ is measured by a 3-axis magnetometer, and the gradient of the magnetic field $\nabla \mathbf{b}_b$ is computed using the measurements from several 3-axis magnetometers. All these sensors are attached to the body frame.

## 3. PROBLEM STATEMENT

After grouping the dynamics (1), (4) and (5), we get the state space dynamics:

$$ \left\{ \begin{aligned} \frac{d}{dt} \mathbf{q} &= \frac{1}{2} [\boldsymbol{\omega}_q \times] \mathbf{q}, \\ \frac{d}{dt} \mathbf{v}_b &= -[\boldsymbol{\omega} \times] \mathbf{v}_b + \mathbf{a}_b - \mathbf{R}_n^b \mathbf{g}, \\ \frac{d}{dt} \mathbf{b}_b &= -[\boldsymbol{\omega} \times] \mathbf{b}_b + \nabla \mathbf{b}_b \cdot \mathbf{v}_b, \end{aligned} \right. \quad (6) $$

where the system state is $\mathbf{x} = [\mathbf{q} \ \mathbf{v}_b \ \mathbf{b}_b]^T$, the input is $\mathbf{u} = [\boldsymbol{\omega} \ \mathbf{a}_b \ \nabla \mathbf{b}_b]$, and the output is $\mathbf{y} = [\mathbf{b}_b]$.

Discretizing model (6) and introducing process noise and measurement noises to address model and measurement uncertainties, enables us to express the state space in a nonlinear discrete form as follows:

$$ \left\{ \begin{aligned} \mathbf{x}_k &= f_d(\mathbf{x}_{k-1}, \mathbf{u}_{k-1}) + \mathbf{w}_{k-1}^x, \\ \mathbf{y}_k &= h_d(\mathbf{x}_k) + \mathbf{w}_k^y, \end{aligned} \right. \quad (7) $$

where, for any (discretized) time $k$, $\mathbf{x}_k$, $\mathbf{u}_k$ and $\mathbf{y}_k$ are the state, the input and the output, respectively. The discretized state transition and output functions are denoted
by $f_d$ and $h_d$, respectively. The process noise $w_k^x$ and the measurement noise $w_k^y$ are assumed to be uncorrelated zero-mean white random signals with positive definite covariance matrices $Q_k$ and $R_k$, respectively. The extended Kalman filter (EKF) described in Algorithm 1 (see Sorenson (1985)) is used for state estimation.

<caption>**Algorithm 1** Traditional Extended Kalman Filter</caption>

**Inputs:** $\hat{\mathbf{x}}_{k-1}$, $\mathbf{P}_{k-1}$, $\mathbf{y}_k$, $\mathcal{Q}_{k-1}$, $\mathcal{R}_k$, $\mathbf{u}_{k-1}$
1: $\mathbf{A}_k = \frac{\partial f_d}{\partial \mathbf{x}} |_{\mathbf{x}} \hat{\mathbf{x}}_{k-1}$
2: $\hat{\mathbf{x}}_{k|k-1} = f_d(\hat{\mathbf{x}}_{k-1}, \mathbf{u}_{k-1})$
3: $\mathbf{P}_{k|k-1} = \mathbf{A}_k \mathbf{P}_{k-1} \mathbf{A}_k^T + \mathcal{Q}_{k-1}$
4: $\mathbf{C}_k = \frac{\partial h_d}{\partial \mathbf{x}} |_{\mathbf{x}} \hat{\mathbf{x}}_{k|k-1}$
5: $\mathbf{K}_k = \mathbf{P}_{k|k-1} \mathbf{C}_k^T (\mathbf{C}_k \mathbf{P}_{k|k-1} \mathbf{C}_k^T + \mathcal{R}_k)^{-1}$
6: $\tilde{\mathbf{y}}_k = \mathbf{y}_k - \hat{h}_d(\hat{\mathbf{x}}_{k|k-1})$
7: $\hat{\mathbf{x}}_{k|k} = \hat{\mathbf{x}}_{k|k-1} + \mathbf{K}_k \tilde{\mathbf{y}}_k$
8: $\mathbf{P}_{k|k} = \mathbf{P}_{k|k-1} - \mathbf{K}_k \mathbf{C}_k \mathbf{P}_{k|k-1}$
9: return $\hat{\mathbf{x}}_k$, $\mathbf{P}_k$, $\tilde{\mathbf{y}}_k$

The input to EKF includes the estimated state in the previous time step $\hat{\mathbf{x}}_{k-1}$, along with its corresponding covariance matrix $\mathbf{P}_{k-1}$, the input $\mathbf{u}_k$ and the output $\mathbf{y}_k$. Linearization of the system dynamic and the output function are done in steps 1 and 4, respectively. Steps 2 and 3 involve the computation of the prediction $\mathbf{x}_{k|k-1}$ and its covariance matrix $\mathbf{P}_{k|k-1}$. Subsequently, in step 5, the Kalman gain $\mathbf{K}_k$ is calculated. Step 6 introduces the innovation, defined as the difference between measurements and predicted output. Step 7 provides the state estimate $\mathbf{x}_k$, and its corresponding covariance matrix $\mathbf{P}_k$ is provided in step 8. The output of the EKF consists of the estimated state $\mathbf{x}_k$ with its corresponding covariance matrix $\mathbf{P}_k$ and the innovation $\tilde{\mathbf{y}}_k$. The process covariance matrix $\mathcal{Q}_k$ is employed in step 3 to compute the prediction error covariance matrix, while the measurement noise covariance matrix $\mathcal{R}_k$ is employed in step 5 to compute the Kalman gain.

The selection of the process and measurement noise covariance matrices is one of the key considerations in designing the EKF. The performance of the EKF algorithm degrades when these matrices are not chosen appropriately. Therefore, the objective of this paper is to design an algorithm that adapts the covariance matrices $\mathcal{Q}_k$ and $\mathcal{R}_k$ online using the Q-learning method, with the aim of maximizing a reward based on minimizing the innovation $\tilde{\mathbf{y}}_k$.

## 4. SOLUTION DESIGN

Figure 1 illustrates the block diagram of QLEKF-BiLSTM, where three EKF-BiLSTM blocks run in parallel. All three perform the same function, utilizing velocity pseudo-measurements generated by the BiLTSM network, with differences in the used process and measurement noise covariance matrices. The traditional EKF-BiLSTM employs nominal values of the noise covariance matrices to establish a benchmark for learning evaluation. The learning EKF-BiLSTM explores different noise covariance matrices from a predefined set by the user $\{(\mathcal{Q}^{(i)}, \mathcal{R}^{(j)}) \ 1 \le i \le M, 1 \le j \le N\}$, where $M$ and $N$ represent the numbers of predefined process and measurement noise covariance matrices, respectively. Subsequently, a scalar reward is constructed

by comparing the innovation sequences of the traditional and the learning filters, denoted as $\tilde{\mathbf{y}}_k^t$ and $\tilde{\mathbf{y}}_k^l$, respectively. The Q-Learning method is then applied to select the best pair of noise covariance matrices, which are subsequently utilized in the learned EKF-BiLSTM, providing the final state estimation.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Solution block diagram of QLEKF-BiLSTM</figcaption></figure>

More details about the solution are provided in Section 5, where the generation of velocity pseudo-measurements by the BiLSTM network is explained in Subsection 5.1, some preliminaries on Q-Learning are discussed in Subsection 5.2, and the QLEKF is presented in Subsection 5.3.

## 5. QLEKF-BiLSTM

### 5.1 EKF-BiLSTM

EKF-BiLSTM was proposed in Zmitri et al. (2022), combining the extended Kalman filter (EKF) with an AI-based approach to enhance estimation performance. Bidirectional Long Short-Term Memory (BiLSTM) network was selected due to its suitability for time series data and its effectiveness with limited training. During training, the BiLSTM network is trained with both input and output data. The input data consists of measurements from a magnetometer, accelerometer, and gyroscope. The output data is the reference velocity data. Following the training process, the BiLSTM network produces a velocity estimate using only the input data, which includes measurements from the magnetometer, accelerometer, and gyroscope. This estimate serves as a pseudo-measurement added to the output vector to be utilized in the EKF. The EKF-BiLSTM estimation approach is illustrated in Figure 2.

### 5.2 Preliminaries Q-learning

Q-learning is a reinforcement learning technique, where an agent learns iteratively through a trial-and-error process and uses a reward to evaluate the s-state and action pairs. The iterative process involves the following steps:

1. The agent observes the s-state.

2. It selects an action $a$, and executes it.

3. The environment responds to the action, providing feedback in the form of the reward $\mathcal{R}$.
<figure><img src="image_3.png" /><figcaption>Fig. 2. EKF-BiLSTM estimation approach.</figcaption></figure>

(4) The s-state will change from s into the next s-state s'.

(5) At the next time-step, the agent chooses the next action a' based on prior experience gained through interaction with the environment.

The primary objective of Q-learning is to determine the optimal action for a given s-state, maximizing cumulative rewards.

In the Q-learning method, the cumulative reward is expressed as a Q-value, which is updated every time step. The one-step updating rule of the Q-value is described as follows:

$$Q(s, a) \leftarrow Q(s, a) + \alpha \left[ \mathcal{R} + \gamma \max_a Q(s', a) - Q(s, a) \right], \quad (8)$$

where

* s is a s-state and belongs to the S set.

* a is an action and belongs to A action set.

* Q(s, a) is the Q-value for performing action a in s-state s.

* α ∈ [0, 1] is the learning rate, determining the impact of new information on the Q-value. Commonly, a constant learning rate like α = 0.1 is used.

* γ ∈ [0, 1] is the discount factor, regulating the significance of future rewards. Often, a constant discount factor like γ = 0.9 is employed.

* R is the reward, defining the agent's short-term learning objectives.

## 5.3 QLEKF

In this subsection, we explain how QLEKF works. As shown in Subsection 5.2, to implement Q-learning, we need to define the s-state set S, the action set A and the reward R:

**s-State set S:** It is a grid-based representation of size M × N, where M denotes the number of design values for the process noise covariance matrices (Q')_{1≤i≤M}, and N denotes the number of design values for the measurement noise covariance matrices (R')_{1≤i≤N}. The nominal values of the traditional EKF correspond to the central cell (M and N are set to be odd integers), and the other cells are set according to a geometric progression with a predefined common ratio. For example, Q'' = r^{i-\frac{M+1}{2}} Q''_{nom}, where r

is the predefined ratio, and the Q''nom is the nominal value of the process covariance matrix, as illustrated in Dai et al. (2022b); Xiong et al. (2021).

**Action set A:** This set describes the s-state transition, assuming that the agent can either transfer to its adjacent s-state or remain at the current s-state.

**The reward R:** The reward is constructed based on the difference in the squared norm of the innovation ŷ_k between a traditional EKF and the learning one.

Figure 3 illustrates the s-state and action sets, where S_{ij} represents the s-state in the i-row and j-column, corresponding to the matrices Q''^{(i)} and R''^{(j)}. The arrows indicate s-state transitions (actions). It is assumed that the agent can transfer to its adjacent s-state or stay in the current s-state. The steps of the QLEKF are listed

<figure><img src="image_4.png" /><figcaption>Fig. 3. s-State set and corresponding action set.</figcaption></figure>

in Algorithm 2, where ˆx_k^{tr}, P_k^{tr}, and ˜y_k^{tr} correspond to the traditional EKF. ˆx_k^{le}, P_k^{le}, and ˜y_k^{le} correspond to the learning EKF. ˆx_k, P_k, and ˜y_k correspond to the learned EKF. The first step is parameter initialization, and the second step is a loop on the periods, where the total duration is divided into periods of length T, and for each period, some covariance matrices are selected. Step 3 is the action selection, where a random variable from a uniform distribution U(0, 1) is compared with a predefined threshold ϵ ∈ [0, 1]. If it is smaller, the action is chosen randomly from the action set so that the Q-learning method can explore more actions; otherwise, the action is chosen based on the maximum Q-value from past experiences. In step 4, the action is executed, and we obtain the s-state s', which corresponds to the cell (i, j). Step 6 involves a loop on time steps in the current period. The traditional, learning, and learned EKF are executed in steps 8, 9, and 11, respectively. In the learning and learned EKF, the covariance matrices Q''^{(i)} and R''^{(j)} are utilized, while the nominal ones Q''nom and R''nom are employed in the traditional EKF. The reward is cumulated in step 10. Step 13 calculates the Q-value as in (8), step 14 is reset for the next period, and finally, the state estimation with the corresponding covariance matrix is returned in step 16.

## 6. EXPERIMENTS AND RESULTS

In this section, we explain the used dataset, implementation details, and the results of velocity estimation.
**Algorithm 2 QLEKF**

1. Initialization

$$ \hat{\mathbf{x}}_{0}^{\mathrm{tr}}=\hat{\mathbf{x}}_{0}^{\mathrm{le}}=\hat{\mathbf{x}}_{0}, \mathbf{P}_{0}^{\mathrm{tr}}=\mathbf{P}_{0}^{\mathrm{le}}=\mathbf{P}_{0}, k \leftarrow 0, \mathfrak{R} \leftarrow 0 \\ \text { For all } s \in S, a \in A, \mathfrak{Q}(s, a) \leftarrow 0, s \leftarrow \text { first cell in the grid. } $$

2. for each period do

3. Action a selection:

if rand(.) < ϵ then:

choose a from A randomly

else:

$$ a=\arg \max _{a} \mathfrak{Q}(s, a) $$

4. execute a and observe reached state s'.

5. (i, j) ← s'

6. for t = 1, 2, 3, .., T

7. k ← k + 1

8. $[\hat{\mathbf{x}}_{k}^{\mathrm{tr}}, \mathbf{P}_{k}^{\mathrm{tr}}, \tilde{\mathbf{y}}_{k}^{\mathrm{tr}}]=\mathrm{EKF}(\hat{\mathbf{x}}_{k-1}^{\mathrm{tr}}, \mathbf{P}_{k-1}^{\mathrm{tr}}, \mathbf{y}_{k}, \mathbf{Q}^{\mathrm{nom}}, \mathbf{R}^{\mathrm{nom}}, \mathbf{u}_{k})$

9. $[\hat{\mathbf{x}}_{k}^{\mathrm{le}}, \mathbf{P}_{k}^{\mathrm{le}}, \tilde{\mathbf{y}}_{k}^{\mathrm{le}}]=\mathrm{EKF}(\hat{\mathbf{x}}_{k-1}^{\mathrm{le}}, \mathbf{P}_{k-1}^{\mathrm{le}}, \mathbf{y}_{k}, \mathbf{Q}^{(i)}, \mathbf{R}^{(j)}, \mathbf{u}_{k})$

10. $\mathfrak{R} \leftarrow \mathfrak{R}+\left(\left\|\tilde{\mathbf{y}}_{k}^{\mathrm{tr}}\right\|^{2}-\left\|\tilde{\mathbf{y}}_{k}^{\mathrm{le}}\right\|^{2}\right)$

11. $[\hat{\mathbf{x}}_{k}, \mathbf{P}_{k}, \tilde{\mathbf{y}}_{k}]=\mathrm{EKF}(\hat{\mathbf{x}}_{k-1}, \mathbf{P}_{k-1}, \mathbf{y}_{k}, \mathbf{Q}^{(i)}, \mathbf{R}^{(j)}, \mathbf{u}_{k})$

12. end for

13. $\mathfrak{Q}(s, a) \leftarrow \mathfrak{Q}(s, a)+\alpha[\mathfrak{R}+\gamma \max _{a} \mathfrak{Q}(s', a)-\mathfrak{Q}(s, a)]$

14. $s \leftarrow s', \mathfrak{R} \leftarrow 0, \hat{\mathbf{x}}_{k}^{\mathrm{le}} \leftarrow \hat{\mathbf{x}}_{k}^{\mathrm{tr}}, \mathbf{P}_{k}^{\mathrm{le}} \leftarrow \mathbf{P}_{k}^{\mathrm{tr}}$

15. end for

16. return $\hat{\mathbf{x}}_{k}, \mathbf{P}_{k}$

## 6.1 The used dataset

The dataset employed in this study comprises real-experimental data collected in Zmitri et al. (2022). The data was acquired using a Magneto-Inertial Tachymeter (MIT) provided by SYSNAV, specifically designed for pedestrian navigation, particularly in indoor scenarios.

These datasets encompass both inertial and magnetic measurements obtained from a sensor model attached to the body frame, along with ground truth data acquired from a motion capture system.

For a more comprehensive understanding of the datasets and their acquisition methodology, readers are encouraged to refer to Zmitri et al. (2022).

## 6.2 Implementation details

**Implementation details of BiLSTM:** The BiLSTM network, implemented in MATLAB for the purpose of generating velocity pseudo-measurements, follows a structured architecture inspired by the methodology outlined in Zmitri et al. (2022). The network configuration is detailed as follows: The input layer consists of 9 features, representing measurements from three sensors, namely the 3-axis gyroscope, 3-axis accelerometer, and 3-axis magnetometer. Subsequently, two BiLSTM layers, each containing 100 hidden units. To mitigate overfitting, dropout layers with a probability of 0.25 are inserted after each BiLSTM layer. A fully connected layer connects the learned features to the output layer, responsible for predicting the velocity. The model's predictive accuracy is assessed through a regression layer, computing the Mean Squared Error (MSE) by comparing the predicted velocity with the reference velocity acquired from the motion capture system. The optimization process is done by the Adam optimizer, characterized by an initial learning rate of 0.0015. The model undergoes training for a total of 500 epochs.

**Implementation details of QLEKF:** The grid dimensions are defined as $M = 5$ and $N = 5$. The nominal values of the process and measurement noise covariance matrices consist of the identity matrix scaled by a constant factor. Specifically, this constant is set to 1 for the process noise covariance and 10 for the measurement noise covariance. The remaining grid elements are set according to the methodology outlined in Section 5.3, utilizing a geometric progression with a common ratio of $r = 10$.

Key parameters governing the Q-learning process include a learning rate $\alpha$ of 0.1, a discount factor $\gamma$ set to 0.9, and a threshold for action selection probability $\epsilon$ established at 0.1.

## 6.3 Velocity estimation results

In this subsection, we present the results of velocity estimation for two trajectories over 80 seconds: a pedestrian walking in a triangular path and a circular path, with Figures 4 and 5 showing the velocity estimation results along the three axes for these trajectories, respectively. In each figure, the continuous red line represents the ground truth velocity, the green dotted line corresponds to the QLEKF-BiLSTM, and the blue dashed line corresponds to the EKF-BiLSTM. These figures illustrate that QLEKF-BiLSTM is closer to the ground truth compared to EKF-BiLSTM. Additionally, we calculate the Root Mean Square Error (RMSE) for each algorithm, as shown in Table 1. The results indicate an improvement in estimation performance when using Q-learning. These first results prove that QLEKF is capable of selecting appropriate noise covariance matrices and then enhance velocity estimation. However, it's important to note that in Q-learning, the search for the best covariance matrices is limited to a predefined set. This limitation could potentially be addressed by implementing a Dynamic Grid-based Q-learning approach, as proposed in Dai et al. (2022a).

<table><caption>Table 1. Velocity RMSE for EKF-BiLSTM, and QLEKF-BiLSTM</caption><thead><tr><th></th><th>EKF-BiLSTM</th><th>QLEKF-BiLSTM</th></tr></thead><tbody><tr><td>Velocity RMSE (m/s)</td><td>0.1025</td><td>0.0833</td></tr><tr><td>Triangular trajectory</td><td></td><td></td></tr><tr><td>Velocity RMSE (m/s)</td><td>0.0830</td><td>0.0653</td></tr><tr><td>Circular trajectory</td><td></td><td></td></tr></tbody></table>

## 7. CONCLUSION

This paper introduces a solution for velocity estimation using measurements from inertial and magnetic sensors. The approach combines two methods: EKF-BiLSTM and the Q-learning approach. The validation of this solution is conducted using real experimental datasets. The results demonstrate that the proposed QLEKF-BiLSTM solution outperforms EKF-BiLSTM in velocity estimation performance, highlighting the capability of QLEKF to select appropriate noise covariance matrices. Future work aims to enhance velocity estimation by adding gyroscope and accelerometer biases into the dynamic model, and exploring the utilization of a dynamic grid-based Q-learning approach.
<figure><img src="image_5.png" /><figcaption>Fig. 4. Velocity estimation along the three axes for pedestrian walking in a triangular trajectory.</figcaption></figure>

## REFERENCES

Chesneau, C.I., Hillion, M., and Prieur, C. (2016). Motion estimation of a rigid body with an EKF using magnetoinertial measurements. In *2016 IEEE Int. Conf. Indoor Position. Indoor Navig. (IPIN)*, 1–6. Alcala de Henares, Spain.

Dai, X., Fourati, H., and Prieur, C. (2022a). A dynamic grid-based Q-learning for noise covariance adaptation in EKF and its application in navigation. In *61st IEEE Conf. Decis. Control (CDC)*, 4984–4989. Cancun, Mexico.

Dai, X., Nateghi, V., Fourati, H., and Prieur, C. (2022b). Q-learning-based noise covariance adaptation in Kalman filter for MARG sensors attitude estimation. In *2022 IEEE Int. Symp. Inertial Sensors Syst.*, 1–6. Avignon, France.

Hanlon, P. and Maybeck, P. (2000). Multiple-model adaptive estimation using a residual correlation Kalman filter bank. *IEEE Trans. Aerosp. Electron. Syst.*, 36(2), 393–406.

Hua, M.D., Martin, P., and Hamel, T. (2014). Velocity-aided attitude estimation for accelerated rigid bodies. In *53rd IEEE Conf. Decis. Control (CDC)*, 328–333. Los Angeles, CA, USA.

Kai, X., Wei, C., and Liu, L. (2010). Robust extended Kalman filtering for nonlinear systems with stochastic uncertainties. *IEEE Trans. Syst. Man Cybern. A Syst.*

<figure><img src="image_6.png" /><figcaption>Fig. 5. Velocity estimation along the three axes for pedestrian walking in a circular trajectory.</figcaption></figure>

Hum., 40(2), 399–405.

Liu, M., Lai, J., Li, Z., and Liu, J. (2016). An adaptive cubature Kalman filter algorithm for inertial and land-based navigation system. *Aerosp. Sci. Technol.*, 51, 52–60.

Maoudj, A. and Christensen, A.L. (2021). Q-learning-based navigation for mobile robots in continuous and dynamic environments. In *17th IEEE Int. Conf. Autom. Sci. Eng. (CASE)*, 1338–1345. Lyon, France.

Rizvi, S.A.A. and Lin, Z. (2018). Output feedback Q-learning for discrete-time linear zero-sum games with application to the H-infinity control. *Automatica*, 95, 213–221.

Sabatini, A.M. (2006). Quaternion-based extended Kalman filter for determining orientation by inertial and magnetic sensing. *IEEE Trans. Biomed. Eng.*, 53(7), 11.

Sorenson, H.W. (1985). *Kalman filtering: theory and application*. IEEE Press, New York.

Wang, H., Deng, Z., Feng, B., Ma, H., and Xia, Y. (2017). An adaptive Kalman filter estimating process noise covariance. *Neurocomputing*, 223, 12–17.

Xiong, K., Wei, C., and Zhang, H. (2021). Q-learning for noise covariance adaptation in extended Kalman filter. *Asian J. Control*, 23(4).

Zmitri, M., Fourati, H., and Prieur, C. (2022). BiLSTM network-based extended Kalman filter for magnetic field gradient aided indoor navigation. *IEEE Sens. J.*, 22(6), 4781–4789.