# Pose Estimation Utilizing a Gated Recurrent Unit Network for Visual Localization
## 2 System Overview
### Framework of Pose Estimation Utilizing GRU Network
#### 2.3.2 Framework with Classical VO  
The proposed method is combined with a classical VO. The framework combining the classical VO and the proposed GRU network is shown in Figure 7. An image sequence is input to the classical VO and feature points are detected when the classical VO is a feature-based VO method. After detecting the feature points, feature matching (i.e., tracking) is conducted. As a result, the rotation matrix \(R\) and the translation vector \(t\) are obtained from the classical VO. In this process, a memory buffer containing the yaw angle for the last 5 timesteps is kept. If the camera enters the cornering section, the GRU  
Figure 6: Architecture of Gated Recurrent Unit (GRU) network for predicting yaw angle and GRU cell. (**a**) GRU network. (**b**) A GRU cell where \(\times\) is element-wise multiplication. network is activated and uses the previous 5 historical yaw angles obtained from the classical VO to predict the yaw angle for correcting the pose obtained from the classical VO. To detect when the camera enters a cornering section, a process is defined as follows  
\[\left|\psi_{VO}\right|_{t\ =\ k-5,...,k-1}\geq\gamma \tag{13}\]  
where \(\gamma\) is the threshold to check whether the camera is in the cornering section. The camera is determined to be in a cornering section when the yaw angles of the camera are continuously larger than the threshold angle \(\gamma\). In this work, the threshold angle \(\gamma\) is set to 0.85 deg through trial and error based on whether the cornering section could be well distinguished. It is recommended to set the threshold angle \(\gamma\) slightly higher than the angle where the cornering section is well detected as will be discussed in the Section 3.1. The condition to identify when the estimated yaw angle from the classical VO deviates significantly from the previous rotation trend is defined as follows  
\[\left|\frac{\Delta\psi_{VO}}{\Delta t}\right|_{t\ =\ k}\geq\alpha max(\left|\frac{ \Delta\psi_{VO}}{\Delta t}\right|_{t\ =\ k-5,...,k-1}) \tag{14}\]  
where \(\alpha\) is scale factor determining the lower limit of the trend. The scale factor \(\alpha\) for simulations is set to 1.5, considering the range of change allowed for variation of estimation by the classical VO in preliminary simulations. The condition to identify when the yaw angle estimated by the classical VO deviates from the yaw angle obtained from the GRU network is defined as follows  
\[\left|\frac{\Delta\psi_{VO}}{\Delta t}\right|_{t\ =\ k}\geq\left|\frac{ \Delta\psi_{GRU}}{\Delta t}\right|_{t\ =\ k}. \tag{15}\]  
If the change of rotation is fairly large so that Equations (14) and (15) are satisfied, the yaw angle is corrected. Correction of yaw angle is executed by fusing the yaw angle obtained from the GRU network and the yaw angle obtained from the classical VO. The fusion is executed using the Normalized  
Figure 7: Proposed pose estimation framework with classical VO. Cross-Correlation (NCC) as the weighting factor. The NCC represents the similarity between images and takes values between 0 and 1 [40]. The value of NCC rapidly decreases at the cornering section as shown in Figure 8. In addition, estimation by the classical VO becomes inaccurate at the cornering section as described in Section 2.2. The value of NCC can be considered as a reliability factor of the yaw angle calculated by the classical VO. In some sections with large NCC values, the proposed framework can trust less the estimated value by the GRU, thereby preventing possible problems of increased error when the GRU is fully trusted. Reconstruction of the rotation matrix is computed as follows  
\[