# Unsupervised Deep Learning-Based RGB-D Visual Odometry
## 3 The Proposed Approach
### System Architecture  
In this paper, we further adopt the method presented in [34], which combines the advantages of CNN and RNN into a Recurrent Convolutional Neural Network (RCNN). Since VO is the process of estimating a camera's pose through changes between consecutive frames, the input of the CNN is fused by sequences of adjacent image frames. In addition, in order to inject absolute scale information, we also feed the depth images into the network. The RGB images and depth images have the same CNN network structure. Figure 2 shows the network architecture of the front-end dual-stream CNN, which extracts image features from the RGB stream and the depth stream, respectively. The additional depth channel is highlighted in green. Each time, two consecutive frames of RGB-D image pairs are used for training. Regarding RGB images, we adjust it to \(416\times 128\times 3\) along its color channel and then stack consecutive frames together to \(416\times 128\times 6\). In terms of the depth image, we first convert it from a 3D point cloud to a single-channel 2D depth image, in which each pixel represents the distance from the viewpoint to the scene object. We then duplicate its single channel to a 3-channel depth image and stack it with the ones of the adjacent frame as \(416\times 128\times 6\). The RGB stream and the depth stream form a dual-stream CNN structure. The gray cuboids and the green cuboids in Figure 2 represent the RGB stream and the depth stream, respectively. The design of the CNN is based on the network developed by Oxford's Visual Geometry Group (VGG) [35]. We modify the VGG model to fit the VO system proposed in this paper. Both the RGB stream and the depth stream of the CNN begin with a \(7\times 7\) convolutional layer, followed by a \(5\times 5\) convolutional  
Figure 2: Architecture of the Convolutional Neural Network with input images. layer, both of which are used to capture basic components in RGB-D images. The size of the rest 5 convolutional layers is then reduced to \(3\times 3\) to capture fine detailed local features. Since the size of input images is relatively small, there is no need to use a big stride to fast reduce the image dimension. In the proposed network, the stride we use is 2. Zero-padding is also deployed to prevent the reduction in image size going too fast. The padding size drops along with kernel size. A Rectified Linear Unit (ReLU) activation function follows each convolutional layer to introduce non-linearity to the network. The conventional batch normalization is not applied in our system since it results in slow and unstable loss convergence during our experiments. Table 1 shows the filter size, stride, padding, and channel number of each convolutional layer in detail. After the dual-stream CNN outputs \(4\times 1\times 512\) RGB features and \(4\times 1\times 512\) depth features, they are finally reshaped and tiled onto two 2048-dimensional feature vectors. In this way, the absolute scale information is captured by the neural network through the additional depth channel. The rear-end RNN, as shown in Figure 3, can be viewed as a pose estimator across the whole image sequence. Firstly, a concatenation layer is used to join the RGB and depth feature vectors together. A fully connected layer is followed to balance the weights of these two features. The feature vectors are then ready for the RNN. Since the current inference is highly dependent on the previously estimated position and orientation, we apply RNN to learn patterns of the entire input image sequence. In this case, the use of standard RNN will have long-term dependence problems. Therefore, we use Long Short-Term Memory (LSTM) to replace the standard RNN in our system.