## Materials and Methods
### Our Approach
#### 2.2.1 Template-Based 3D Scene Modeling  
The core modules of the 3D scene modeling algorithm include the proposed 3D point cloud modeling using the MPPA and the training of the deep neural networks for describing the content of the scene model. In this section, we analyze these two algorithms in detail. * 3D Point Cloud Modeling Using _MPPA_  
In this work, we propose a deep learning approach to train a viewpoint classifier that learns the feature representation of each training RGB-D frame in a multiview video which captures the image and depth information in different views of a 3D scene. The algorithm starts by performing the 3D reconstruction algorithm, shown in Figure 2, to represent the scene as a 3D point cloud using the training depth maps and the given camera pose parameters. The 3D scene reconstruction algorithm is outlined in Algorithm 1. Figure 1: The workflow of the proposed 3D scene modeling for model updating using a single RGB image: (**a**) the 3D scene modeling; (**b**) the model updating approach. The CNN object detector is an optional operator which is used only when the target scene to be modeled is an object. Figure 2: Example of 3D Scene Reconstruction using the dataset ‘freiburg1-plant’ [44]: (**a**) Multiview RGB-D video; (**b**) the unprojecting result of each frame in (**a**) using the proposed multiple principal plane analysis on the corresponding point cloud; (**c**) the unprojected image of the whole scene point cloud which is generated by the 3D point cloud registration algorithm [34]. The point cloud generated from Step 1.1 is first partitioned by the binary tree representation to define the set of super-points (SPs) which will be used as the basic units for the RGB-D image quantization and point cloud registration. We have two types of nodes in a binary tree: nonleave and leave nodes. A nonleave node will be decomposed into two child nodes in the binary tree. A leave node stores a small number of points that are basically located on a common 3D plane, i.e., the principal plane. We perform the depth-first-traversal process to flatten the tree as a list of leave nodes, i.e., the so-called super-points. Thus, each super-point is a leave node of the binary tree, consisting of a subset of points that describes a principal plane of the point cloud. The shape of the point cloud is finally represented as multiple principal planes (MPPs). Let a nonleave node N contain a collection X of \(m\) points:  
\[\text{X}=\left[ \begin{array}{c}x_{1},y_{1},z_{1}\\ x_{2},y_{2},z_{2}\\ \ldots\\ x_{m},y_{m},z_{m}\end{array} \right] \tag{8}\]  
where \((x_{i},y_{i},z_{i})\), \(i=1\) to \(m\), are the 3D points of a cloud. To perform singular value decomposition (SVD) on X, we obtain three eigenvalues \(\lambda_{1}\), \(\lambda_{2}\) and \(\lambda_{3}\) (\(\lambda_{1}\geq\lambda_{2}\geq\lambda_{3}\)) and the corresponding eigenvectors \(\
u_{1}\), \(\
u_{2}\) and \(\
u_{3}\). In this work, we define the following metric to estimate the score of flatness (\(SF\)) of the local surface in N:  
\[SF(\text{N})=\frac{\lambda_{3}}{\sum_{i=1,\ldots 3}\lambda_{i}} \tag{9}\]  
Using (9), the following rule (\(R1\)) is applied to define the type of the node N:  
\[R1(\text{N}):\left\{ \begin{array}{ll}\textit{Lave},&\text{if }SF(\text{N})< \gamma\\ \textit{Non-leave},&\text{otherwise}\end{array} \right. \tag{10}\]  
where \(\gamma\) is a predefined threshold which controls the number of leave nodes of the resulting binary tree.