# MonoMR: Synthesizing Pseudo-2.5D Mixed Reality Content from Monocular Videos
## 2 Related Work
### Free-Viewpoint Video System  
Since the method of capturing a real-world object using multiple cameras and converting the object into a 3D object produces a high-quality result, systems utilizing this method are mainstream in free-viewpoint video systems. Initially, [PERSON] et al. proposed a virtualized reality system [PERSON] et al. (2013), in which 51 monocular cameras are arranged in a dome-shaped structure. In this system, a target object can be converted into free-viewpoint content, and since the system has scalability, this system was used for early free-viewpoint sports broadcasting ([PERSON] et al., 2013). This multi-view method is applied to various free-viewpoint systems ([PERSON] et al., 2013; [PERSON] et al., 2013; [PERSON] et al., 2013; [PERSON] et al., 2013), and since these systems consist of RGB cameras, they are less restrictive to the environment. In addition, free-viewpoint sports broadcasting systems, such as Intel True View ([PERSON] et al., 2016) and Canon's free-viewpoint video system ([PERSON] et al., 2016), based on this method are actively commercialized by various companies. However, the systems based on this method require many cameras, synchronization devices, and a large amount of computation. Consequently, utilizing these systems by individuals or small groups is still challenging. Studies on generating highly detailed free-viewpoint video using depth cameras have been conducted. The introduction of commercial depth cameras, such as RealSense ([[https://software.intel.com/en-us/realsense](https://software.intel.com/en-us/realsense)]([https://software.intel.com/en-us/realsense](https://software.intel.com/en-us/realsense)), accessed on 24 August 2021) and Kinect ([[https://developer.microsoft.com/en-us/windows/kinect](https://developer.microsoft.com/en-us/windows/kinect)]([https://developer.microsoft.com/en-us/windows/kinect](https://developer.microsoft.com/en-us/windows/kinect)), accessed on 24 August 2021), facilitates these studies. Accordingly, the systems ([PERSON] et al., 2013; [PERSON] et al., 2013; [PERSON] et al., 2013) that generate high-quality free-viewpoint video through depth cameras have been proposed. [PERSON] et al. proposed a system ([PERSON] et al., 2016) that generates free-view video with 106 RGB and depth cameras and compresses the video for real-time free-viewpoint video streaming. Based on this research, [PERSON] et al. presented the Holoportation (2016) system that enables real-time telepresence on MR HMDs. However, operating this method outdoors is difficult because most depth cameras are not suited for natural light; thus, the capturing environment is restricted to indoor environments. Furthermore, since the systems still require many cameras and synchronization devices, the end user's accessibility is still limited. Various methods for generating a free-viewpoint video from a monocular video have been proposed. However, it's still challenging because estimating the depth from a single camera is not easy, and the visual information captured by the monocular camera is limited. One of the early proposed systems, Tour into the picture (a system proposed by [PERSON] et al. (2016)), transforms artwork into a 3D scene using a perspective transformation based on user interaction. Recently, DNNs ([PERSON] et al., 2013; [PERSON] et al., 2013) have been proposed to estimate a monocular image's depth information, which is an essential basis in generating stereoscopic content from monocular videos. As one of the state-of-the-art technologies, [PERSON] et al. proposed a free-viewpoint soccer video system ([PERSON] et al., 2016) by restoring the player's position and 3D mesh from a single soccer broadcasting video using multiple DNNs. Research on how to view the generated free-viewpoint videos has been conducted. Before the development of the HMD, the user controls the viewpoint of the generated free-viewpoint videos through the primary input interface, such as a keyboard, mouse, and joystick.