## Materials and Methods
### Our Approach
#### 2.2.1 Template-Based 3D Scene Modeling
In this work, the convolutional-based deep viewpoint classification neural network (DVCNN) is trained in the 3D scene modeling phase. As shown in Figure 4, the auto-encoder neural networks, which are variants of the semantic segmentation neural network (SegNet) [47], can be used for RGB-D deep feature prelearning. The layers of the encoder and decoder network in Figure 4 are topologically identical to the layers of auto-encoder and decoder in SegNet except that we use ADAM as the optimizer instead of SGD since the ADAM achieves better training performance. In this work, we also discard the last softmax layer of the decoder and use a \(1\times 1\) convolution layer with ReLU instead. Figure 4: In deep feature learning neural networks (DFLNN) which consists of an encoder, a decoder, and a \(1\times 1\) convolutional layer: (**a,b**) are the neural networks that transform the original RGB and depth images into the quantized versions, respectively; (**c**) is the depth recovering neural network (DRNN). Figure 3: An example of the proposed binary tree representation algorithm for 3D reconstruction: (**a,b**) are the original RGB and depth images, respectively; (**c**) is the 3D point cloud generated from (**a,b**); (**d,e**) are the super-pixel representation of (**a,b**), respectively; (**f**) is the super-point representation of (**c**). Given a training RGB-D video V of \(n\) image frames for 3D scene modeling, for each RGB image \(I\) and depth image \(D\) in V, the proposed Algorithm 1 generates the corresponding quantized RGB image \(ql\) and quantized depth image \(qD\), and then produces the training data sets: RGB-dataset and depth-dataset which consist of collections of RGB image tuples \(\left(I_{j},ql_{j}\right)_{j=1}^{n}\) and depth image tuples \(\left(D_{j},qD_{j}\right)_{j=1}^{n}\), respectively. Using a generic back-propagation gradient descent optimization algorithm, the RGB-dataset (depth-dataset) can be applied to train the RGB-deep feature learning neural network (depth-deep feature learning neural network) using the loss function defined as follows:  
\[loss=\frac{1}{n}{\sum_{j=1}^{n}}{\left\|{q\hat{I}_{j}-\hat{I}_{j}}\right\|}_{2} \tag{13}\]  
where \(\hat{I}_{j}\) is the image generated by the feature learning neural network if the \(j\)-th input image \(I_{j}\) is applied for training. As shown in Figure 4c, a deep feature learning neural network (DFLNN) can also be used to recover the depth information from the input RGB image by applying the training dataset which consists of a collection of image tuples \(\left(I_{j},D_{j}\right)_{j=1}^{n}\) to train the resulting depth recovering neural network (DRNN). The last layer of both DFLNN and DRNN is a \(1\times 1\) convolutional layer, which reduces the dimensionality of the output of the RGB-decoder (or depth-decoder), i.e., the prelearned deep features, to the shape of the original input. The performance of the above neural networks for deep feature pretraining is obviously limited by the size of training datasets. This is the so-called overfitting problem of deep neural networks. Instead of using the auto-encoders to compute the deep features of the training RGB-D images, in our approach, we directly cascade the visual features of all super-pixels into a feature vector to characterize the content of an input RGB image. Next, we perform the well-known \(k\)-means clustering on the collection of these visual feature vectors to group the training RGB-D images into clusters. With each cluster representing a viewpoint, the clustering results automatically generate the viewpoint (\(vp\)) labels of the training RGB-D images. This forms a collection of viewpoint training tuples \(\left(I_{j},vp_{j}\right)_{j=1}^{n}\) for training the proposed viewpoint classifier which is a variant of the VGG convolutional neural network (CNN) [43]. As shown in Figure 5, the trained deep viewpoint classification neural network (DVCNN) is accurate in annotating the viewpoint of an input RGB image because it is transferred from the effective VGG neural network.