## 1 Introduction
### Our Contributions  
In this research, we reconstruct the image-based 3D model of a scene (or an object) from a set of training RGB-D images in which templates corresponding to individual viewpoints of the scene are carefully selected. For the potential application of scene surveillance in a 3D space, the moving camera in a drone needs to be calibrated in advance for computing its intrinsic parameters. In the phase of scene modeling, a structure-from-motion approach [11] is then used to compute the pose parameters and depth maps of frames in the captured RGB video. This augments the input video with an additional depth channel and transforms it into a video of four channels. To put these parameters together, we can reconstruct a scene model by transforming individual templates into view-specific 3D point clouds which are further registered into a single model point cloud [37]. Based on the input RGB-D video that captures pictures at all the viewpoints of the target scene, the proposed 3D scene modeling makes contributions in four aspects. Firstly, we propose an approach based on the multiple principal plane analysis (MPPA) to partition a point cloud generated from a training RGB-D image into multiple super-points. For each super-point, we have a corresponding super-pixel in the associated training RGB image. To represent a training RGB image as a list of visual features of individual super-pixels, our approach then clusters the training feature vectors into a viewpoint codebook using the well-known k-means clustering algorithm [38]. A training image is labeled as the \(i\)-th viewpoint if its feature vector belongs to the \(i\)-th cluster of the viewpoint codebook. Secondly, a convolution neural network for viewpoint classification is trained, which predicts the viewpoint of a single RGB image. The deep neural network annotates the viewpoint classes of individual training RGB images in terms of membership probability. In a viewpoint class, among all member images, the training image with the highest probability is selected as the template of the viewpoint. Thirdly, for estimating the pose parameters of an input RGB image, the RGB image and the corresponding viewpoint template are aligned in terms of feature vectors of key-points (super-pixels). The pose estimation process is an iterative loop which finely tunes the initial camera parameters and depth map by matching the input frame against the most similar template in terms of visual features of super-pixels. Finally, taking the different levels of 3D noise into account, we propose an iterative closest point (ICP) framework which embeds the generalized Procrustes analysis [39] to decrease the interference of high noise on the accurate calculation of a projection matrix describing how a 3D model projects onto a set of input frames. Through template-to-frame registration, the up-to-date scene model can be reconstructed and loaded in an augmented reality (AR) environment to facilitate displaying, interaction, and rendering of an image-based AR application. To compare with conventional pose estimation algorithms which use sparse features for pose estimation, our approach enhances the quality of reconstructing the 3D scene point cloud using a RGB-D video. Lastly, we evaluate our system on open RGB-D benchmarks provided by the Technical University of Munich [40]. Parts of our work are implemented based on the open source project Open3D [37]. It provides some visualization functions for the 3D point cloud process and achieves the basic function of the ICP algorithm.