Subset based deep learning for RGB-D object recognition

RGB-D camera can easily record both color and depth images and previous works have proved that combining them together could dramatically improve the RGB-D based object recognition accuracy. In this paper, a new method based on a subset approach was introduced to learn higher level features from the raw data. The raw RGB and depth images were divided into several subsets according to their shapes and colors, guaranteeing that any two different objects in each subset are nearly not similar. Then a RGB-Subset-Sparse auto-encoder was trained to extract features from RGB images and a Depth-Subset-Sparse auto-encoder was trained to extract features from depth images for each subset. Then the learned features were transmitted to recursive neural networks (RNNs) to reduce the dimensionality of the features and learn robust hierarchical feature representations. The feature representations learned from RGB images and depth images were concatenated as the final features and then sent to a softmax classifier for classification. The proposed method is evaluated on three benchmark RGB-D datasets, RGB-D dataset of Lai et al., 2D3D dataset of Browatzki et al and Aharon dataset of Aharon et al. Compared with other methods, ours achieves state-of-the-art performance on the first two datasets. Furthermore, to validate the generalization of our subset approach, we also do some extra experiments of applying the subsets approach to several previous works, these accuracies improved significantly. Keywords: RGB-D Object Recognition, Subset Based Feature Extracting, Sparse Auto-Encoder, Recursive Neural Networks, Deep Learning. Object recognition is not only a crucial but also a hard task for household robot to understand and interact with the indoor environments. It has been widely studied and a variety of methods, and standardized benchmarks problems have been proposed by the researchers. The performance on these benchmarks (e.g. CIFAR [1] and NORB [2]) has been gradually improved over the years. One key point to recognize objects accurately is to learn robust representations because the objects in the real-world often appear in cluttered scenes increasing the difficulty of the object recognition problems. However, most of the previous works on object recognition either use RGB images or gray images separately. There are several reasons making many algorithms perform well on previous object recognition. Those images are relatively clear, and the whole object is included in the image. Besides, the resolutions of the images are not high. The birth of cheap RGB-D cameras such as the Microsoft Kinect has changed the situation. With this new sensing technology, we can easily obtain high quality synchronized videos of both color and depth. Depth information extracted from depth image is invariant to lighting or color * Corresponding Author variations, providing geometrical cues. Combining RGB and depth information in the object recognition can dramatically improve the recognition accuracy. Nevertheless, it is unclear how to exploit the depth information. Utilizing traditional methods directly on these color images combining the depth information may not achieve good results. Although the technology represents an opportunity to dramatically improve the performance of object recognition, it also raises the problem of developing expressive features for the color and depth channels of the new sensors. Various unsupervised learning algorithms have been proposed for learning robust features from RGB and depth images. Lai et al. [3] proposed Instance Distance Learning (IDL) algorithm which defined a view-to-object distance where a novel view is compared simultaneously to all views of a previous object. Blum et al. [4] proposed a convolutional k-means descriptor (CKM) which can learn features in the neighborhood of detected interest points automatically and combine all available information into a meaningful representation. Bo et al. [5] introduced hierarchical matching pursuit (HMP) method for RGB-D data. They use unsupervised feature learning based on sparse coding to learn dictionaries from 8 different channels including grayscale intensity, RGB, depth scalars, and surface normals. Features are then used in hierarchical matching pursuit which consists of two layers. Each layer has three modules: batch orthogonal matching pursuit, spatial pyramid max pooling (SP), and contrast normalization. Socher et al. [6] introduced a new model combining convolutional and recursive neural networks (CNN-RNN) for learning features and classifying RGB-D images. Chen et al. [22] proposed a semi-supervised learning (SSL) framework to improve RGB-D object recognition with only 20% labeled data. The method can compete against other state-of-the-art methods proposed before theirs. These methods mainly extract random patches from all the images and then train the feature learning network with these patches. With the trained network, they extract features from all the images. However, from their results we found that the classifiers perform poor on the objects similar to each other in both color and shape. The reason is that all the images share the same feature extracting network, so the similar objects are easily to be extracted similar features which are difficult for classifier to distinguish. Fig. 1 shows some pairs of objects which are easily to be confused with each other. Fig. 1 Some pairs of objects that are always confusable with each other. In this paper, a novel method is proposed to learn robust representations of objects. Before extracting features from the raw images, we divide these objects into some subsets by their colors and shapes. Firstly, the objects with similar shapes and colors are gathered into the same subset. Secondly, only one object is randomly selected from several subsets to constitute a new subset, n subsets are obtained where n is the number of objects in the first step. Then for each new subset, an RGB-Subset-Sparse auto-encoder (RGB-Subset-SAE) is trained for RGB images and a Depth-Subset-Sparse auto-encoder (Depth-Subset-SAE) is trained for depth images. Finally, SAE is combined with the recursive neural networks to extract features from the RGB-D images in each subset. In this way, discriminative features can be extracted from similar objects such that the classifiers can perform better. The rest of this paper is organized as follows. Section 2 describes the related works of this paper. Section 3 describes the method of learning higher level features from subsets of the raw data. Experiments and interesting results are shown in section 4. Finally, we make a conclusion and point out our future work in section 5. In this section, the related work, mainly containing sparse auto-encoder and recursive neural networks, to the proposed novel deep learning method are reviewed. Auto-encoder [7] learning algorithm is an unsupervised approach to automatically learn features from unlabeled data. Fig. 2 is an example of an auto-encoder with structure of 4-2-4 for input-hidden-output layers. Suppose that the raw unlabeled training example set is , where and is the number of the examples. By applying the back propagation algorithm, an auto-encoder learning algorithm is expected to get optimal parameters which can be used to transform the input to the hidden layer and then map the to the desired output which is almost equal to the corresponding inputs, i.e., . We can say that is another representation of the input . Fig. 2: A simple 4-2-4 auto-encoder sample denotes the weight associated with the connection between the neuron in the layer and the neuron in the ! " layer. # denotes the biased term corresponding to the neuron in the layer. Let $ # represent all the parameters in the auto-encoder network. For a given training set , the overall cost function of an auto-encoder can be defined as follows: % &'()* " + ,". / . 0 ! 1 2 + + + " 3456 2 34 2 47 82 The first term in the definition of 9:; <=>? is an mean squared error term. The second term is a regularization term (also called a weight decay term) that tends to penalize the weights, and thus prevent overfitting, where @ represents the number of network layer, and A denotes the number of units in layer . The weight decay parameter 1 controls the relative importance of the two terms. Sparse auto-encoder learning algorithm [8] is a modified auto-encoder algorithm. By imposing a sparsity constraint to control the number of active hidden neurons, the sparse auto-encoder learning algorithm will discover interesting structure in the data. Informally, A neuron is "active" if its output value of sigmoid function (referring to formula (2)) is close to 1, or is "inactive" if its output value of sigmoid function (referring to formula (2)) is close to 0. It is better to constrain the neurons to be inactive most of the time. So, a penalty term is added to the overall cost function to achieve the sparsity constraint. B C " " ! D7E And the overall cost function of Sparse Auto-Encoder is as formula (3): %3FG*3& % &'()* ! H + IJKLMML N O 2 P Where is a weight to control the sparsity penalty term and IJKLMML N L QRS L L ! " / L QRS " / L " / L T Where KL is Kull-back Leibler divergence [9] to control the sparsity of hidden neuron, and L is an activated expectation to keep the most of hidden neurons being inactive and L is the activation of the UV hidden neuron which can be calculated by: L " W + XYZ ? [ ?2 ] Where denotes the activation of the UV hidden neuron for a given input . The goal is to minimize %3FG*3& as a function of and #. An optimization algorithm such as batch gradient descent is a good choice. One iteration of gradient descent updates the parameters and # as follows: Ẑ_ Ẑ_ / ` a b c Ẑ_ %3FG*3& ^ d e f d_ d_ / ` a b cd_ %3FG*3& ^ d e g where ` is the learning rate. The key step is computing the partial derivatives. And the back propagation algorithm is an efficient way to compute these partial d