# StereoVO: Learning Stereo Visual Odometry Approach Based on Optical Flow and Depth Information
## 3 Approach
### The Structure of StereoVO  
The proposed StereoVO comprises three main components: the Flow-Net, the Depth-Net, and the VO-Net. For the Flow-Net, FlowNet2 [38] is employed to estimate optical flow due to its superior generalization capability in realistic scenes and fast processing speed. In the Flow-Net, the optical flow of each pair of consecutive stereo frames are generated and then down-sampled to 1/8 of the original resolution using Average Pooling. In the Depth-Net, we utilized SSIMCV to acquire depth information. Different from previous methods, such as [7; 8; 15; 16], we use down-sampled features to calculate depth information without up-sampling to obtain a full resolution depth map, which allows for a simpler structure of the network. The resulting optical flow and extracted depth information were used as input to the VO-Net. The architecture of the proposed StereoVO method is illustrated in Figure 1.  
In the Flow-Net, the FlowNet2 network takes two continuous stereo frames and produces raw optical flow. These optical flows and frame at time \(t+1\) are used to warp the frame at time \(t\). The structural similarity index (SSIM index) [39] between the warped frame at \(t\) and the original frame at \(t\) is utilized as the confidence of the optical flow. The SSIM index for two frames, x and y, is calculated as follows:  
\[ssim(x,y)=\frac{\left(2\mu_{x}\mu_{y}+C_{1}\right)\left(2\sigma_{xy}+C_{2} \right)}{\left(\mu_{x}{}^{2}+\mu_{y}{}^{2}+C_{1}\right)\left(\sigma_{x}{}^{2} +\sigma_{y}{}^{2}+C_{2}\right)}, \tag{1}\]where \(\mu\) and \(\sigma\) are the mean intensity and standard deviation of frames, respectively, \(C_{1}\) and \(C_{2}\) are constant while we set \(C_{1}=9\times 10^{-4}\), and \(C_{2}=10^{-4}\) follows [39]. Then, the SSIM index is concatenated with optical flow and fed into Conv Block1, whose architecture is shown in Figure 2. Additionally, depth information is obtained from the left and right frame using the SSIMCV block, as described in Section 3.2. The architecture of the Conv and FC blocks is shown in Figure 3, and the parameters for all convolution and pooling layers can be found in Table 1.  
Figure 1: Overview of the StereoVO network. Trainable blocks are represented in blue and non-trainable blocks are shown in green. The blocks with the same name are weight shared. The blue arrow corresponds to the t frame, and the orange arrow corresponds to the \(t+1\) frame, while the intermediate variables are shown in gray. Flow is shown in RGB format transformed from raw optical flow, while we use raw optical flow during training and testing.  
Figure 2: Architecture of the Conv Block, which contains three sequential Basic Blocks. Each Conv contains a sequential structure of a convolution layer, a BatchNorm layer, and a LeakReLU layer.  
In the FlowNet, the left-side and right-side optical flows, generated by FlowNet2, are concatenated with \(ssim\) right and ssim left, respectively. The SSIM index (Equation (1)) is computed between the destination image \(\mathbf{I_{d}}\) and the warped destination image \(\mathbf{W_{d}}\),  
\[ssim_{s}=SSIM(\mathbf{W_{d}},\mathbf{I_{d}}), \tag{2}\]  
and we warp the source image \(\mathbf{I_{s}}(x,y)\) via flow from source image to destination image \(\mathbf{flow_{sd}}(u,v)\):  
\[\mathbf{W_{d}}=\mathbf{I_{s}}\big{(}x+u_{x},y+v_{y}\big{)}. \tag{3}\]