## 5 Semantic-Aware Global Orientation Estimation
### Global Orientation Optimization  
When a new image appears, first extract the marker line collection and determine the local dominant direction according to the method described in the subsection before. As shown in Figure 4, the orientation of the current frame equals that angle between the global orientation and the dominant direction of the current frame; thus, it can be computed as  
\[\theta_{z}=\arccos(\frac{I_{g}\cdot l_{c}}{|I_{g}||I_{c}|}), \tag{5}\]  
Figure 4: The three subfigures in the upper row illustrate the procedure of structure information extraction: (**a**) denotes the raw input synthesized image, (**b**) the extracted raw lines, and (**c**) the segmentation results, in which green lines denote the preserved marking lines and red lines denote the noisy lines. (**d**–**f**) denote the relationship between the local dominant direction and global orientation, in which the orange arrow denotes the local dominant direction and the blue arrow denotes the global dominant direction for the current frame, respectively, while the green lines denote the marking lines. where \(I_{g}\) denotes the global orientation and \(I_{c}\) denotes the dominant direction of the current frame. A collection of marker lines may contains some noise lines. However, these noise lines also satisfy the geometric rules of the Manhattan world assumption, so they have little effect on the orientation estimates. This way of estimating orientation is independent of the orientation estimation of adjacent frames, so there is no cumulative error. When the orientation is known, only the translation term remains to be estimated. In this way, the originally nonlinear pose estimation problem is transformed into a linear least-squares problem. Due to the existence of occlusion, visual blurring, and other factors, the consistency of line segment detection is poor, and a line segment is often split into two independent line segments, or the length of the line segment will change. This results in the line feature not binding enough on the amount of translation in the extension direction and may even introduce incorrect constraints. Although it is also possible to extract the features of points separately to constrain the amount of translation, this will cost additional computational resources. As shown in Figure 4, the endpoints of many segments are also corner points. Therefore, in order to solve the above problem, this paper takes the endpoints of the line segment as point features and uses bidirectional optical flow to trace these endpoints to establish data associations between endpoints, rather than between line features. More specifically, mapping the feature points from the reference frame to the current frame, denoted as \(P_{r}\), re-mapping the feature points from the current frame to the reference frame, denoted as \(P_{c}\), only if enough point pairs are obtained, we can compute the translation term as  
\[{}^{W}\mathbf{t}_{B_{i}}={}^{W}\mathbf{R}_{B_{j}}\cdot\overline{\mathbf{P}}_{ B_{j}}+{}^{W}\mathbf{t}_{B_{j}}-{}^{W}\mathbf{R}_{B_{i}}\cdot\overline{ \mathbf{P}}_{B_{i}}, \tag{6}\]  
where \(\bar{p}\) denotes the mean position of the point sets. Then, estimate the translation of the current frame via the reprojection errors. If the current frame is more than 10 frames away from the previous keyframe and the required number of features is met, the current frame is selected as the keyframe and added to the local map. Then, optimize the local map using an objective function that minimizes the global orientation residuals and reprojection errors as  
\[\min_{{}^{W}\mathbf{T}_{B_{i}}}w\sum_{i}\left(1-\frac{I_{i}\cdot\left({}^{B_{ i}}\mathbf{R}_{W}I_{g}\right)}{|I_{i}||^{B_{i}}\mathbf{R}_{W}I_{g}}\right)+\sum_{i} \left\|\mathbf{p}_{i}-{}^{W}\mathbf{T}_{B_{i}}\pi_{\mathrm{s}}^{-1}(\mathbf{u }_{i})\right\|_{2}, \tag{7}\]  
where the former term means the global orientation residual errors since the angle between \(I_{i}\) and \({}^{B_{i}}\mathbf{R}_{W}I_{g}\) should be zero.