# Real-Time Stereo Visual Odometry Based on an Improved KLT Method
## 3 Detailed Description of the Method
### Adaptive KLT Tracker and Simplified KLT Matcher  
In successive stereo frame \(k-1\), the AKT tracks the selected features to acquire their pixel coordinates in frame \(k\). First, the optical flow corresponding to the feature point is solved in two consecutive left images \(I_{k-1}^{L}\) and \(I_{k}^{L}\). In the notation, the superscripts \(L\) and  
Figure 2: Stereo observation model. \(R\) index the \"left\" and \"right\" images, respectively, and the subscript \(k\) indexes the frame. The AKT minimizes Equation (1) using the Newton-Raphson method:  
\[\mathbf{d}=\underset{\mathbf{d}}{\operatorname{argmin}}\iint_{W_{k-1}}\left[I_{ k}^{L}(\mathbf{x}+\mathbf{d})-I_{k-1}^{L}(\mathbf{x})\right]^{2}\omega(\mathbf{x})d \mathbf{x} \tag{1}\]  
where \(\mathbf{x}=(u,v)\) is a feature point; \(\mathbf{d}=(\Delta u,\Delta v)\) is the translation of the feature window's center; \(W_{k-1}\) is the adaptive window; and \(\omega(\mathbf{x})\) is a Gaussian weighting function. The correct optical flow computation of feature correspondences in frame \(k-1\) generates the tracked features in the left image in frame \(k\). Because the standard KLT tracker assumes that the feature patch undergoes only translation motion, a fixed-window KLT is susceptible to scale distortion and affine transformation [26]. A small feature window is sensitive to noise, whereas a large feature window may not exhibit a clear or sharp response. In the following paragraphs, we indicate that it would be better to determine an adaptive window size for the KLT using the average disparity, the translation velocity, and the yaw angle. From the FRVO discussion in [6], a larger window size should be used for features with large motion, while a smaller window size should be used for a feature with small motion. FRVO uses disparity information to represent the motion experienced by the features, as shown in Figure 3; thus, an adaptive window size for the KLT is employed based on the disparity field. On a large scale, this must be true. However, for a specific feature, a large disparity does not mean that a large window is necessarily suitable. Especially for a feature near the road, a large motion is often accompanied by a large affine transformation. The experimental results demonstrate that a small window allows for a better tracking accuracy. Therefore, the proposed method does not pursue the optimal tracking accuracy for every feature, but for better overall performance. Instead of a dense disparity map, the average disparity is used as an indicator for the adaptive window size, which also reduces the computational complexity. On the other hand, disparity alone is not sufficient to characterize the motion of features. For example, when the translation velocity of the camera is slow, the local optical flow vector of each feature is small. Alternatively, for a turning maneuver, Figure 4 shows that the feature with a small disparity still has a larger optical flow vector. At this point, the tracking error tends to increase rapidly if the AKT relies on disparity information. Therefore, in addition to the average disparity in the current frame, the translation velocity of the camera and the yaw angle in the previous frame should be used to guide the adaptive window strategy. This makes it possible to use a small window for the AKT even when the disparity of features is large. Based on the discussion above, a joint adaptive function (JAF) is built as follows:  
\[W_{k}=c_{d}(d_{k}-d_{0})+c_{v}(v_{k-1}-v_{0})+c_{\alpha}(\alpha_{k-1}-\alpha_{ 0})+b \tag{2}\]  
where \(W_{k}\) is the adaptive window size for the AKT in frame \(k\); \(d_{k}\), \(v_{k-1}\), and \(\alpha_{k-1}\) are the average disparity in frame \(k\), the translation velocity of the camera, and the yaw angle in frame \(k-1\), respectively; \(d_{0}\), \(v_{0}\), and \(\alpha_{0}\) are the constant offsets; \(c_{d}\), \(d_{v}\), and \(c_{\alpha}\) are the disparity, the velocity, and the yaw angle weighting coefficients, respectively; and \(b\) is the base window size.