## Materials and Methods
### Our Approach
#### 2.2.1 Template-Based 3D Scene Modeling
Furthermore, we define the plane with the eigenvector \(v_{3}=(v_{3,x}\), \(v_{3,y}\), \(v_{3,z})\), corresponding the smallest eigenvalue of SVD(X), as the normal vector to be the principal plane (PP) of N, which can be used to compute the distance from a point p = \((x,y,z)\) to PP:  
\[d(\text{p})=v_{3,x}(x-\overline{x}_{\text{N}})+v_{3,y}(y-\overline{y}_{\text {N}})+v_{3,z}(z-\overline{z}_{\text{N}}) \tag{11}\]where \(\overline{\mathrm{p}}=\left(\overline{\mathrm{x}}_{\mathrm{N}},\overline{\mathrm{y}}_{ \mathrm{N}},\overline{\mathrm{z}}_{\mathrm{N}}\right)=\)\(\sum_{i=1,...,m}\mathrm{p}_{i}/m\) be the center point of a node N which contains \(m\) points. Next, based on (11), rule R2 decomposing a nonleave node N into two child nodes N\({}_{1}\) and N\({}_{2}\), is defined as:  
\[R2(\mathrm{N}):\left\{ \begin{array}{ll}\mathrm{p}\in\mathrm{N}_{1},&\text{ if }d(\mathrm{p})\geq 0\\ \mathrm{p}\in\mathrm{N}_{2},&\text{ otherwise}\end{array} \right. \tag{12}\]  
If N is a leave node, N is a super-point in which the \(v_{3}\) is stored as the normal vector of the node. The proposed _MPPA_ keeps a queue of nonleave nodes and removes each node in the queue one-by-one for further decomposition. Thus, the input point cloud is finally represented as a list of leave nodes (super-points) in which the shape of each super-point is modeled as its principal plane. Let \(\mathrm{P}_{i}=\left\{(x_{j},y_{j},z_{j},R_{j},G_{j},B_{j},D_{j})\right\}_{j=1}^ {m_{i}}\) be the point cloud of the \(i\)-th RGB-D image frame of the training video V. Initially, we store \(\mathrm{P}_{i}\) into the root note of the binary tree and set the root node to be a nonleave one. Next, based on keeping a list \(\mathrm{L}_{i}\) of nonleave nodes, we build up the resulting binary tree \(\mathrm{T}_{i}\) with the following summarized iterative algorithm. ```
Input: A point cloud \(\mathrm{P}_{i}=\left\{(x_{j},y_{j},z_{j},R_{j},G_{j},B_{j},D_{j})\right\}_{j=1}^ {m_{i}}\). Output: A binary tree of \(\mathrm{P}_{i}\) and the list of super-points \(\mathrm{\Gamma}_{i}=\left\{SP_{j}\right\}_{j=1}^{\|\overline{\mathrm{P}}_{i} \|}\). Method:
1. Initially, add \(\mathrm{P}_{i}\) to the nonleave node list \(\mathrm{L}_{i}\). 2. Remove a nonleave node N from \(\mathrm{L}_{i}\). 3. Check the type of N using (10) and perform SVD to compute the principal plane of N.
4. if N is a leave node, store the eigenvector \(\mathrm{v}_{3}\) into N, which is defined as the normal vector of the principal plane in N.
5. if N is a nonleave node do: apply rule R2 to decompose N into two child nodes N\({}_{1}\) and N\({}_{2}\), which are added to \(\mathrm{L}_{i}\). 6. if \(\mathrm{L}_{i}\) is not empty, go to Step 2. 7. Perform the depth-first tree traversal on the binary tree \(\mathrm{T}_{i}\) to order the leave nodes as a list of super-points \(\mathrm{\Gamma}_{i}=\left\{SP_{j}\right\}_{j=1}^{\|\overline{\mathrm{P}}_{i} \|}\). ```  
**Algorithm 2** Binary tree Representation of Point Cloud with the _MPPA_. As mentioned in Algorithm 1, the \(j\)-th super-point \(SP_{j}\) in \(\mathrm{P}_{i}\) is represented as the tuple (\(\overline{\mathrm{p}}_{j}\), \(\overline{\mathrm{c}}_{j}\), \(\overline{\mathrm{D}}_{j}\), \(\overline{\mathrm{n}}_{j}\)) in which \(\overline{\mathrm{n}}_{j}\) is the normal vector of the principal plane that approximates the shape of the points in \(SP_{j}\). For each point in \(SP_{j}\), we set the color and depth values of the corresponding pixel in 2D image to be \(\overline{\mathrm{c}}_{j}\) and \(\overline{D}_{j}\), respectively. This process generates the super-pixels of the input RGB and depth images. Figure 3 shows an example of representing an RGB-D image as a point cloud, which is processed to generate the super-point-based point cloud and super-pixel-based RGB-D images using the proposed MPPA. methodology of self-taught learning, which is recognized to be an effective method for training a supervised-learning based neural network without using a tedious human annotated dataset.