## Materials and Methods
### Our Approach
#### 2.2.2 The Pose Estimation Algorithm  
The usage of a highly mobile drone facilitates the modeling task of a 3D outdoor scene. The selected templates to model the target 3D scene also play an important role in the application of visual inspection because the associated GPS information of templates is useful to define the checkpoints of the scene. In the inspection phase, we navigate the drone to the GPS locations of the templates and capture individual RGB images which can be used to detect abnormal events that occur in the real-world scene. Furthermore, the scene under surveillance is often changing slightly along the timeline. It is quite important to keep the model up-to-date when we load the model into a 3D scene surveillance application. Basically, the 3D model records the background information which corresponds to the normal situation of the scene under surveillance. On the contrary, an abnormal event can be detected in a specific checkpoint if the matching error between the current input RGB image and the corresponding RGB template is large. However, the detection accuracy is often limited by this simple scheme since the captured RGB image in a checkpoint is often not the same as the corresponding template RGB image as the location of the navigated drone contains an error using the prestored GPS information of templates. This leads to the requirement of the pose estimation algorithm that recovers the depth image and external camera parameters using an alignment algorithm between the input RGB image and the corresponding template RGB image. In the model updating phase, given an RGB image \(I\) captured from an unknown location of the scene, the proposed pose estimation algorithm is an iterative procedure consisting of four major steps: (1) search for a suitable template that samples the possible appearance of the model in the input image \(I\) using the proposed DVCNN; (2) compute the point cloud of \(I\) for generating the super-points and the super-pixels; (3) compute the transformation matrix that aligns the input image \(I\) with the template in terms of the visual features of super-pixels; (4) fine tune the pose parameters and depth map of \(I\) by an iterative closest point (ICP) algorithm [48; 49]. Given an input RGB image, the pose estimation algorithm uses the external camera parameters and depth map of the corresponding template as the initial coarse parameters to compute the first 3D point cloud. Incorrect pose parameters would generate a larger error when we register the current point cloud with the model point cloud in a 3D space. The iterative pose estimation stops when the registration error is minimized. With object inspection as an example, Figure 6 shows the proposed scheme to estimate the external camera parameters of the input images, which are often unknown when we use a moving camera to conduct 3D object inspection. To deal with this issue, the proposed image-based scene (object) modeling can be used to accurately estimate the depth map as well as the external camera parameters of the input image. The first step of the proposed drone-based surveillance scheme is to detect the target objects in the input RGB \(I_{j}\) using the well-proved CNN object recognition system [43]. Next, with the detected object \(O\) as the input data, the proposed MPAA returns a set of discriminative visual features. Let \(i\) be the viewpoint class of the input image \(I_{j}\) which is annotated by the proposed DVCNN. This implies that the \(i\)-th RGB-D template \(\Phi_{i}\) would be retrieved from the image-based 3D model. Initially, the depth map of the \(i\)-th template \(\Phi_{i}\) is set to be the depth image \(D_{j}\) of \(I_{j}\). We also set the external camera parameters of \(\Phi_{i}\) as the initial parameters for the test image \(I_{j}\). Obviously, the external camera parameters should be modified because it is not possible for the positions of the cameras to capture the test image and the template to be the same.