# A Novel Real-Time Virtual 3D Object Composition Method for 360\({}^{\circ}\) Video
## 3 Proposed System and Experiment
### Real-Time 3D Composition Using Stereo Vision  
In our proposed system, we use a 360\({}^{\circ}\) camera \"Z1\", developed by Ricoh-theta [27]. Z1 can record in 4K (3840 \(\times\) 2160). It can also use real-time video streaming with stitching to 3D programs such as Unity and Unreal. This 360\({}^{\circ}\) camera system and the ZED mounted on a rig are connected to a PC through a USB 3.0 port. In addition, the ZED is configured such that it faces the same direction as the front of the 360\({}^{\circ}\) camera. The 360\({}^{\circ}\) camera is used to record the images of the real scene, and at the same time, the ZED extracts the external parameter by generating a depth map in real-time. The ZED generates the initial value of position data (0,0,0) when the program starts, so the difference in the  
Figure 3: The proposed system workflow using stereo vision for extracting the external parameters of the 360\({}^{\circ}\) cameras, which were mounted together. The blue boxes show the recording step (production) and the black boxes show the post-recording steps (post-production).  
physical distance between the ZED and Z1 is not considered. Figure 4 shows the rig-mounted 360\({}^{\circ}\) camera and the ZED.  
The extraction and saving of external stereo-vision parameters are applied within Unity 3D, which is used for simultaneous processing with a pre-visualization system to confirm the composition result. For our method, we propose a stabilization process for external parameters in order to obtain better performance from the noise that generally contains stereo vision. The external parameters extracted from the ZED are saved as new data through a linear Kalman filter in real-time.  
The Kalman filter is an algorithm that was developed by [PERSON] during the early 1960s [28; 29]. It is used to track the optimal value by removing the noise included in the measured value using the prior and prediction states. It consists of a prediction step and an update step. In the prediction step, an expected value is calculated when the input value is received, according to the prior estimated value. In the update step, an accurate value is calculated based on the prior predicted value and the actual measured value. In other words, a correct value is derived by repeatedly applying the prediction and update steps. It is suitable for real-time processing because it makes predictions based on the immediately preceding data, rather than all previous data [30; 31; 32].  
The trajectory data stabilized through the Kalman filter can be saved in various formats for application to 3D programs during post-production. In this paper, we saved the data using the 3 ds Max file scripting language (.ms) to create a virtual camera in 3 ds Max. Figure 5a shows the 3 ds Max script file and Figure 5b shows the 3D composition in the 3 ds Max program.  
Figure 4: 360\({}^{\circ}\) camera and stereo-vision ZED camera.  
Figure 5: 3D composition process in 3 ds Max: (**a**) Max script and (**b**) 3 ds Max scene.  
To measure the accuracy of the camera trajectory with a Kalman filter, the traditional tracking method using an RGB camera was set to the ground truth, in order to compare the applied Kalman filter and raw data of the camera trajectory. The use of the traditional tracking method as a ground truth--even if it is not the best--allows us to show that the proposed method has the same camera trajectory accuracy as the traditional method.