# LSTM Network-Assisted Binocular Visual-Inertial Person Localization Method under a Moving Base
## 2 Materials and Methods
### Basic Description of Visual-Inertial
#### 2.1.2 IMU Pre-Integration and Residual Measurement Model  
In the continuous-time integration form, the position \(p_{wb_{k}}\), velocity \(v_{k}^{w}\), and attitude \(q_{wb_{k}}\) of the IMU at moment \(k\) are known, and the position \(p_{wb_{k+1}}\), velocity \(v_{k+1}^{w}\), and attitude \(q_{wb_{k+1}}\) at moment \(k+1\) can be recursively derived by measuring the acceleration \(a\) and angular velocity \(w\) of the IMU, as shown in Equation (7), where \(g^{w}=\begin{bmatrix}0&0&g\end{bmatrix}^{T}\).  
\[\begin{split} p_{wb_{k+1}}=p_{wb_{k}}+v_{k}^{w}\Delta t+\iint \
olimits_{t\in[k,k+1]}(R_{wb_{t}}a^{b_{t}}-g^{w})\delta t^{2}\\ v_{k+1}^{w}=v_{k}^{w}+\int\
olimits_{t\in[k,k+1]}(R_{wb_{t}}a^{b_{t}}-g^{ w})\delta t\\ q_{wb_{k+1}}=\int\
olimits_{t\in[k,k+1]}q_{wb_{t}}\otimes\left[\begin{array} []{cc}0\\ \frac{1}{2}a^{b_{t}}\end{array}\right]\delta t\end{split} \tag{7}\]  
Typically, the frame rate of the IMU is higher than the frame rate of the cameras. The pre-integration expresses the relative observations of the IMU during the time between the two camera keyframes and can be used as a constraint on the keyframes for tightly coupled optimization. The pre-integration can decouple the IMU computation from the world coordinate system, avoiding the problem of changing Equation (7) for repeated integration of the IMU state quantities during the optimization process and reducing the computational effort. The integral model can be converted to a pre-integrated model by the transformation of Equation (8) as follows:  
\[R_{wb_{t}}=R_{wb_{t}}\otimes R_{b_{t}b_{t}} \tag{8}\]  
Figure 3: Point feature reprojection residuals diagram.  
Substituting Equation (8) into Equation (7), the integral term in Equation (7) becomes the attitude concerning the moment of K instead of the attitude with respect to the world coordinate system:  
\[\begin{split} p_{\omega b_{k+1}}&=p_{\omega b_{k}}+ \tau_{k}^{\omega}\Delta t-\tfrac{1}{2}g^{\omega}\Delta t^{2}+R_{\omega b_{k}} \int\!\int_{t\in[k,k+1]}(R_{b_{k}b_{k}}a^{b_{k}})\delta t^{2}\\ &\tau_{k+1}^{\omega}=\tau_{k}^{\omega}-g^{\omega}\Delta t+R_{ \omega b_{k}}\!\int_{t\in[k,k+1]}\!\left(R_{b_{k}b_{k}}a^{b_{k}}\right)\delta t \\ &\quad R_{\omega b_{k+1}}=R_{\omega b_{k}}\!\int_{t\in[k,k+1]}R_{ b_{k}b_{k}}\otimes\left[\begin{array}{c}0\\ \frac{1}{2}\omega b_{t}\end{array}\right]\delta t\end{split} \tag{9}\]  
The pre-integrated quantity is related only to the IMU measurement, which is obtained by integrating the IMU data directly over a period of time:  
\[\begin{split}\alpha_{b_{k}b_{k+1}}&=\int\!\int_{t\in[k,k+1]} (R_{b_{k}b_{k+1}}a^{b_{t}})\delta t^{2}\\ \beta_{b_{k}b_{k+1}}&=\int_{t\in[k,k+1]}(R_{b_{k}b_{k+1}}a^{b_{k}}) \delta t\\ R_{b_{k}b_{k+1}}&=\int_{t\in[k,k+1]}R_{b_{k}b_{k}}\otimes\left[ \begin{array}{c}0\\ \frac{1}{2}\omega b_{t}\end{array}\right]\delta t\end{split} \tag{10}\]  
Further sorting of Equation (9) leads to the IMU pre-integrations \(\alpha_{b_{k}b_{k+1}}\), \(\beta_{b_{k}b_{k+1}}\), and \(q_{b_{k}b_{k+1}}\) without considering the zero bias, and the first-order Taylor expansion approximations for the accelerometer zero bias and gyro zero bias \(b_{a}\), \(b_{g}\) and expressed as \(\hat{\alpha}_{b_{k}b_{k+1}}\), \(\hat{q}_{b_{k}b_{k+1}}\), and \(\hat{\beta}_{b_{k}b_{k+1}}\). Finally, the residual term of the pre-integration in this paper is shown in Equation (11):  
\[r_{b}(z_{b_{k}b_{k+1,x}})=\begin{bmatrix}r_{p}\\ r_{\theta}\\ r_{v}\\ r_{ba}\\ r_{bg}\end{bmatrix}=\begin{bmatrix}R_{b_{k}a}(p_{\omega b_{k+1}}-p_{\omega b _{k}}-\tau_{k}^{\omega}\Delta t+\frac{1}{2}g^{\omega}\Delta t^{2})-\hat{ \alpha}_{b_{k}b_{k+1}}\\ 2\left[\hat{q}_{b_{k}b_{k+1}}\otimes(q_{b_{k}a\omega b_{k+1}})\right]_{xyz}\\ R_{b_{k}a}(\tau_{k+1}^{\omega}-\tau_{k}^{\omega}+g^{\omega}\Delta t)-\hat{ \beta}_{b_{k}b_{k+1}}\\ b_{a}^{b_{k+1}}-b_{a}^{b_{k}}\\ b_{g}^{b_{k+1}}-b_{g}^{b_{k}}\end{bmatrix}_{\ref{eq:r_b_b_b_b_b_b_b_b_b_b_b_b_b_b \(b\)