# A Novel Real-Time Virtual 3D Object Composition Method for 360\({}^{\circ}\) Video
## 2 Background Theory and Related Studies  
### 3D Composition  
For a realistic 3D composition, it is mandatory that the RGB camera in the real space and the virtual camera in the virtual space have the same viewpoint. In the traditional method, the internal and external parameters can be estimated by searching the feature points from bright spots and dark spots and analyzing the feature point correspondence between each frame. Typical examples of this include simultaneous localization and mapping (SLAM) [6, 7, 8] and structure-from-motion (SfM) [9, 10]. The external parameters extracted by these algorithms can be linked with virtual cameras in various 3D programs such as 3D Max and Maya, as applied in video production, and the Unity 3D and Unreal engines for game production. Figure 1 shows a traditional 3D composition method.  
In general, the 3D composition method tends to depend on the camera-tracking result. Therefore, if the camera-tracking process fails the video must be reshot, which wastes time and money. In previous studies, we reported that various factors may lead to the failure of camera tracking, including an occlusion by a person or object, and motion blur caused by fast camera movement [11]. However, this is more likely to occur in a 2D video shot with relatively numerous camera movements. For a 360\({}^{\circ}\) video clip there is a low possibility of camera tracking failures from such factors because stable camera  
Figure 1: The traditional process using camera-tracking software (Boujou, After Effects) for creating a 3D composition by extracting feature points and estimating camera trajectory from video frames. The blue box shows the recording step (production) and the black boxes show the post-recording steps (post-production).  
movements are applied to prevent user motion sickness when wearing an HMD. Nevertheless, there is a factor that has not yet been mentioned, caused by a difference in the production processes between 2D and 360\({}^{\circ}\) video. In 360\({}^{\circ}\) video more than two cameras are used for capturing each different camera view, and after recording in real-time a 360\({}^{\circ}\) panoramic view is created through a matching process called \"stitching\", which overlaps parts from each video clip [12]. During this stitching process errors can occur as a result of inaccurate matching due to lens distortion. These errors interfere with the tracking of the feature points in a 360\({}^{\circ}\) video clip containing camera movement. As a result, accurate 3D composition is hindered, and human resources are wasted. Figure 2 shows such stitching errors.  
There have been various studies undertaken with the aim of solving this problem. Most of them use a method of applying camera tracking to perspective views of a 360\({}^{\circ}\) video clip before the stitching process. One such method proposed by [PERSON] et al. uses a perspective view from one of the 360\({}^{\circ}\) camera rigs to obtain an undistorted image for eliminating the stitching errors [13]. In addition, [PERSON] et al. proposed a method for obtaining stable tracking results, which uses an image correction by overlapping the point where the distortion occurs with the position difference between frames [14]. Furthermore, tracking algorithms for spherical images such as spherical scale invariant feature transform (SSIFT) [15] and spherical oriented fast and rotated brief (SPHORB) have been developed [16]. These methods can reduce the stitching errors caused by a misplaced feature point, but basically, it is progressed from the recorded video. In addition, most 360\({}^{\circ}\) video clips have a high resolution of more than 4K, which means a significant amount of time is consumed in camera tracking.