# LSTM Network-Assisted Binocular Visual-Inertial Person Localization Method under a Moving Base
## 2 Materials and Methods
### Basic Description of Visual-Inertial
#### 2.1.1 Point Feature Representation  
According to the pinhole camera model [29], a point \(p_{w}=[x_{w},y_{w},z_{w}]^{T}\) in the camera world coordinate system can be mapped to a point \(p_{I}=[u,v,1]^{T}\) in the pixel plane. From Equation (1), the external reference matrix \(T_{cw}\) (containing rotation matrix \(R_{cw}\), translation vector \(t_{cw}\)) converts \(p_{w}\) to \(p_{c}\) under the camera coordinate system; From Equation (2), the internal reference matrix \(K\) (containing parameter \(f_{x},f_{y},c_{y}\)) converts \(p_{c}\) to \(p_{I}\). Using the chi-square coordinates \(s=1/z_{c}\) is introduced.  
\[p_{c}=T_{cw}p_{w}=\begin{bmatrix}R_{cw}&t_{cw}\\ 0&1\end{bmatrix}\begin{bmatrix}x_{w}\\ y_{w}\\ z_{w}\end{bmatrix} \tag{1}\]  
\[p_{I}=\begin{bmatrix}u\\ v\\ 1\end{bmatrix}=sKp_{c}=\frac{1}{z_{c}}\begin{bmatrix}f_{x}&0&c_{x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{bmatrix}\begin{bmatrix}x_{c}\\ y_{c}\\ z_{c}\end{bmatrix} \tag{2}\]  
As shown in Figure 3, the \(k\)th spatial point \(p_{k}\) is measured in the two camera pixel planes \(z_{p_{k}}^{c_{i}}=\begin{bmatrix}u_{p_{k}}^{c_{i}}&v_{p_{k}}^{c_{i}}&1\end{bmatrix} ^{T}\), \(z_{p_{k}}^{c_{i}}=\begin{bmatrix}u_{p_{k}}^{c_{i}}&v_{p_{k}}^{c_{i}}&1\end{bmatrix} ^{T}\), respectively, and the points in the pixel planes are transformed to the camera normalization planes \(\overline{p}_{p_{k}}^{c_{i}}=\begin{bmatrix}x_{p_{k}}^{c_{i}}&y_{p_{k}}^{c_{i}} &1\end{bmatrix}^{T}\), \(\overline{p}_{p_{k}}^{c_{i}}=\begin{bmatrix}x_{p_{k}}^{c_{i}}&y_{p_{k}}^{c_{i} }&1\end{bmatrix}^{T}\) by the \(\pi^{-1}\)(0) function, see Equations (3) and (4).  
\[\overline{p}_{p_{k}}^{c_{i}}=\pi^{-1}\big{(}z_{p_{k}}^{c_{i}}\big{)} \tag{3}\]  
Figure 2: Neural network-assisted visual-inertial person localization framework diagram. The framework diagram is divided into three parts: data pre-processing, initialization, sliding window nonlinear optimization Mechanism to estimate the positional pose.  
Converting \(\overline{p}_{p_{k}}^{c_{j}}\) to \(p_{p_{k}}^{c_{j}}\), see Equation (5). In this paper, the reprojection residuals are expressed in the camera-normalized plane as shown in Equation (6):  
\[p_{k}^{c_{j}}=R_{bc}^{T}(R_{wb_{j}}^{T}(R_{wb_{i}}((R_{bc}\frac{1}{\lambda_{k}}[ \overline{p}_{p_{k}}^{c_{j}}]+p_{bc})+p_{wb_{j}})-p_{wb_{j}})-p_{bc}) \tag{5}\]  
where \(p_{k}^{c_{j}}\) is the reprojection coordinate of the \(k\) point feature under the frame \(j\), \(\overline{p}_{p_{k}}^{c_{j}}\) is the projection of the \(k\) feature point in the unit camera plane of the frame \(i\). \((R_{wb_{i}},p_{wb_{i}})\), \((R_{wb_{i}}^{T},p_{wb_{j}})\) is the position and pose of frames \(i,j\), \((R_{bc},p_{bc})\) is the external reference between IMU and camera. \(\lambda_{k}\) is the inverse depth of the \(k\) feature point.  
\[r_{p}(z_{p_{k}}^{c_{j}})=\left[\frac{p_{k}^{c_{j}}}{\|p_{k}^{c_{j}}\|}-\overline {p}_{k}^{c_{j}}\right]=\left[\begin{matrix}\frac{x_{p_{k}}^{c_{j}}}{z_{p_{k}}^ {c_{j}}}-\overline{x}_{p_{k}}^{c_{j}}\\ \frac{y_{p_{k}}^{c_{j}}}{z_{p_{k}}^{c_{j}}}-\overline{y}_{p_{k}}^{c_{j}}\\ \frac{z_{p_{k}}^{c_{j}}}{z_{p_{k}}^{c_{j}}}-\overline{y}_{p_{k}}^{c_{j}}\\ \end{matrix}\right] \tag{6}\]