# MonoMR: Synthesizing Pseudo-2.5D Mixed Reality Content from Monocular Videos
## 7 Discussion  
We present the MonoMR system that generates the pseudo-2.5D MR content from monocular camera videos. The system can render MR content from a large number of previously captured videos of various types. Our system does not require special imaging equipment, such as multiple monocular cameras or depth cameras, similar to most conventional systems and complicated settings, such as the synchronization among the cameras. In addition, because the proposed system can generate MR content using a single monocular camera with minimal user interaction, this has higher usability than any previously proposed systems. Therefore, users without expert knowledge can easily create MR content. Figure 15: Application examples of the proposed system. **(a)** Sports broadcasting and **(b)** entertainment content provide improved stereoscopic effect and an immersive feeling to users than original monocular videos. **(c)** Surveillance systems based on MonoMR allow users to easily recognize situations and spatial information of multiple cameras. Our system consists of not only a DNN but also uses typical image processing algorithms. Based on the experimental results, we confirmed that our proposed method has reasonable performance for depth information estimation and texture extraction required for producing MR content from monocular videos. In addition, the proposed method is processed relatively high speed, except for the DNN process. Therefore, if a high-performance GPU is used and parallel optimization for image processing is applied, the proposed system can reach real-time performance. We conducted the small user study, and the results show the feasibility of converting existing monocular videos into more exciting and immersive content with the proposed system. Although the MonoMR system has good performance and usability, we describe some technical challenges and limitations of the system based on the conducted experiment and system implementation. **Limited camera posture. ** MonoMR does not use a global motion compensation algorithm and the camera posture estimation method using specific landmarks [7; 47] because of high computation and low generalization capacity. Therefore, the input video's viewpoint, which is converted into MR content, should be fixed. **Pseudo-3D position. ** The system estimates the person's \(x\) and \(z\) positions based on the correlation between the ankle position of the human and the ground. However, if the subject moves on the \(y\)-axis, such as jumping and tumbling, it's difficult to estimate the correct 3D position. To address this problem, we will attempt to apply the global depth estimation DNN [23; 24] to our system. **Texture quality. ** MonoMR extracts textures using a background subtraction algorithm to increase the entire processing speed. As can be observed from the previous experiments, MonoMR extracts person textures with acceptable quality. However, if the texture quality reduces due to a detection failure of the human detector or drastic illumination changes in the capturing environment (Figure 16), then some artifacts could be observed. Mask R-CNN, the current state-of-the-art algorithm, shows excellent quality; however, as mentioned in Section 4.2, it has drastically reduced the system's entire processing speed. Therefore, this method has not been applied in this study. If a segmentation network with good accuracy and performance is proposed, then we will consider applying the network to our system. **2D texture. ** The system displays 2D textures instead of 3D mesh models. Hence, it's difficult for the human vision to recognize that the generated model is planar if the user views the content at a certain distance [48]. However, if the user views the content at a short-range, then they feel the unnaturalness. Therefore, we are considering applying methods to recover the information not captured by the camera, such as generative adversarial nets [49] and 3D mesh recovery network [50], as future work. **Large content size.