## 4 Methodology
### Pyramid Motion Tracking Model Based on Image Semantic Information  
The image pyramid is composed of images with different resolutions, obtained by scaling the original image. Taking the original image as the bottom of the pyramid, the image of the upper layer is obtained by scaling the image of the lower layer at a certain rate. When solving the camera pose by minimizing the photometric error, the optimization is started from the top layer of the pyramid, and then the optimization result of the previous layer is used as the initial value of the optimization of the next layer. Due to the upper layer image being relatively coarse, this process is called the coarse-to-fine strategy. Direct visual odometry requires the camera motion between two adjacent frames to be as small as possible, which avoids encountering a local minimum during optimization. The advantage of using the image pyramid is that when the motion between two adjacent frames is large, the pixel motion is still in a small range in the top image of pyramid, which is shown in Figure 4.  
Figure 4: Assuming that the original image resolution is 6 \(\times\) 8 (row \(\times\) column), the pixels projected from \(P\) to the two frames \(F_{1}\) and \(F_{2}\) are \(p_{1}\) and \(p_{2}\), and the distance between \(p_{1}\) and \(p_{2}\) is 4 pixels. The original image is scaled with a 0.5 scaling factor, and the scaled image size is 3 \(\times\) 4, the distance between the two projection points \(p_{1}\)\({}^{\prime}\) and \(p_{2}\)\({}^{\prime}\) is reduced to 2 pixels. The size of an actual image is much larger, and the image in the top layer of the pyramid can limit the pixel motion to a small range. According to Section 4.1, when the candidate points are extracted from the keyframe, points in dynamic region are removed. However, when the candidate points in the keyframe are projected to the subsequent frame for photometric error calculation, due to the changing relative pose in the optimization process and occlusion caused by dynamic objects, candidate points in static region of keyframe may still be projected to the dynamic region of the subsequent frame, thus affecting the optimization process. Our basic idea is to remove the photometric error calculated by the projection points in a dynamic region. As shown in Figure 5. These residuals will not participate in the solution of the optimization equation. For the keyframe, the image pyramid is constructed with the scale factor 0.5. According to Section 4.1, candidate points are extracted from the image of each pyramid layer, as shown in Figure 6a. For the subsequent frame, the same pyramid model is constructed (Figure 6b), and the semantic image of each layer containing the prior information of dynamic objects is obtained, as shown in Figure 6c. Taking the _k-th_ layer of the pyramid model as example, for a candidate point \(p\) in keyframe \(I_{i}^{k}\), the photometric error \(E_{pj}^{k}\) generated by projecting \(p\) onto subsequent frame \(I_{j}^{k}\) can be described follows:  
\[E_{pj}^{k}=\sum_{p\in N_{p}}w_{p}\|(I_{j}^{k}[p^{\prime}]-b_{j})-\frac{t_{j}e^ {d_{j}}}{t_{j}e^{d_{i}}}(I_{i}^{k}[p]-b_{i})\|_{\gamma} \tag{6}\]  
where \(I_{i}^{k}\) and \(I_{j}^{k}\) are images of \(I_{i}\) and \(I_{j}\) at the _k-th_ pyramid layer, \(p^{\prime}\) is the projection of point \(p\) on \(I_{j}^{k}\), \(t_{i}\) and \(t_{j}\) are the exposure time for \(I_{i}\) and \(I_{j}\), \(\|.\|_{\gamma}\), represents the Huber norm, \(a_{i}a_{j}b_{i}\), \(b_{j}\) are brightness  
Figure 5: \(F_{1}\) is a keyframe. \(p_{1}\), \(p_{2}\) and \(p_{3}\) are candidate points extracted according to Section 4.1. \(F_{2}\) is the subsequent frame of \(F_{1}\). Three points are projected to \(F_{2}\) to form three residuals. If the projection point falls in the dynamic region of \(F_{2}\), the corresponding residual will be removed. Figure 6: (a,b) show the image pyramids constructed by the keyframe and subsequent frame. (c) shows the semantic image of each layer in subsequent frame pyramid.