# LSTM Network-Assisted Binocular Visual-Inertial Person Localization Method under a Moving Base
## 2 Materials and Methods
### Neural Network-Assisted Visual-Inertial Person Localization Method under a Moving Base
#### 2.3.2 Sliding Window Nonlinear Optimization Mechanism for Estimating Poses  
For the correlation of visual-inertial data, all visual-inertial state quantities in the sliding window can be optimized using a tightly coupled approach based on a graph optimization framework, and Figure 7 using a factor graph approach shows the constrained relationship between the camera, the waypoint, and the IMU.  
The state quantities to be estimated in the sliding window are shown in Equation (17), where \(x_{k}\) in Equation (18) is the position, attitude, velocity, acceleration zero bias, and gyroscope zero bias state quantities of the \(k\)th data; and \(\lambda\) is the inverse depth of all point features seen by the camera in the sliding window.  
\[\chi=[\mathbf{x}_{0},\mathbf{x}_{1},\cdots,\lambda_{0},\lambda_{1},\cdots,\lambda_{m}] ^{\text{T}} \tag{17}\]  
\[\mathbf{x}_{k}=\Big{[}\mathbf{p}_{vh_{2}},\mathbf{q}_{ub_{1}},\mathbf{v}_{k}^{v},\mathbf{b}_{a^{k}} ^{b_{k}},\mathbf{b}_{g}^{b_{k}}\Big{]}^{\text{T}},k\in[0,n] \tag{18}\]  
Based on the constructed visual constraint information between the two keyframes and the IMU pre-integration constraint information, it is possible to obtain the optimization objective function of Equation (19).  
\[\begin{split}\underset{\chi}{\text{min}}\rho\Big{(}\parallel \mathbf{r}_{p}-\mathbf{J}_{p}\mathcal{X}\parallel_{\widetilde{\mathbf{x}}_{p}}^{2}\Big{)} +\underset{i\in B}{\sum}\rho\Big{(}\parallel\mathbf{r}_{b}\Big{(}\mathbf{z}_{b_{i}b_{ k+1}},\mathcal{X}\Big{)}\parallel_{\widetilde{\mathbf{x}}_{p_{i}b_{k+1}}^{2}}\Big{)}\\ +\underset{(k,j)\in F}{\sum}\rho\Big{(}\parallel\mathbf{r}_{p} \Big{(}\mathbf{z}_{p_{j}^{\text{c}}}^{\alpha},\mathcal{X}\Big{)}\parallel_{ \widetilde{\mathbf{x}}_{p_{j}^{\text{c}}}^{\alpha}}^{2}\Big{)}\end{split} \tag{19}\]  
where \(\mathbf{r}_{p}\big{(}\mathbf{z}_{p_{j}^{\text{c}}}^{\alpha_{k}},\mathcal{X}\big{)},\bm {r}_{b}\big{(}\mathbf{z}_{b_{i}b_{k+1}},\mathcal{X}\big{)}\) are the point feature reprojection residuals and IMU residuals represented by Equations (6) and (11), respectively; \(\rho()\) is the Huber robust kernel function; \(B\) is the set of all pre-integrated constraint terms within the sliding window. \(F\) is the feature of all observed points of the camera within the sliding window, \(\sum\) and is the respective covariance term. The first term \(r_{prior}-J_{prior}\chi\) is the a priori information term left after marginalizing keyframes in the sliding window, i.e., the constraint term left behind when the oldest frame is removed when a new keyframe is inserted. As mentioned above, the objective function and the state quantity to be optimized are known, and the Gauss-Newton method is used to optimize this nonlinear least squares problem, solving for the increment \(\Delta\chi\) of the state quantity \(\chi\). The optimal state quantity is solved by continuous iterative optimization, as shown in Equation (20):  
\[\big{(}\mathbf{H}_{prior}+\mathbf{H}_{b}+\mathbf{H}_{p}\big{)}\Delta\chi=\big{(}\mathbf{b}_{ prior}+\mathbf{b}_{b}+\mathbf{b}_{p}\big{)} \tag{20}\]  
Figure 7: Factor diagram for visual-inertial positioning system.  
where the relationship between Hessian matrix \(H\) and Jacobi matrix \(J\) is \(H=J^{T}\sum^{-1}J\), and the right-hand term of the equation is \(b=-J^{T}\sum^{-1}r\).