# Real-Time 3D Reconstruction Pipeline for Room-Scale, Immersive, Medical Teleconsultation
## 2 Related Work  
Within the research domain of immersive telepresence, we review relevant literature for real-time 3D reconstruction from point clouds, as shown in Figure 2. Over several decades, researchers explored ways to capture a physical space and transmit it to a distant location [1] to enable immersive, mixed reality telepresence [6; 7; 14; 18; 19]. Most previous works depend on some form of visual depth sensing. Multiple view stereo algorithms can be used with calibrated camera pairs [20] to extract surface information by determining the disparity between two rectified images. Such methods require sufficiently textured surfaces for the stereo matching and are computationally demanding for real-time processing. Alternatively, specialized depth sensors with projected infrared patterns [21] or a time-of-flight camera can directly acquire 2.5D surface point clouds [2; 5; 8; 22]. Structured-light capture systems project static infrared textures onto surfaces and, therefore, enable the accurate acquisition of surfaces also from textureless objects and environments. However, such systems can be quite expensive, can have a limited acquisition frame rate, and require extensive calibration. Sensors that measure the time of flight (ToF) are typically small and work well in indoor scenarios. Sufficiently high resolution and frame rates can be captured with modern ToF sensors such as the Microsoft Azure Kinect camera at a relatively low price.  
Two distinct concepts have emerged for capturing the local environment: a dynamic camera for static scenes [2; 3; 23] and multiple stationary cameras for dynamic scenes [5; 8; 15; 24; 25]. A large body of work exists for capturing static scenes with moving cameras. Surfaces can be extracted with high quality from moving color images with known camera poses using the structure-from-motion method [26]; however, the process is typically offline. Alternatively, simultaneous localization and mapping [27] combines camera pose estimation and map creation to enable real-time operation. If the camera  
Figure 1: Overview. Processing steps of the proposed 3D reconstruction pipeline from left to right: (**a**) fused point cloud of all depth cameras, (**b**) voxelization with reconstructed surface normals, (**c**) simplification based on vertex clustering of flat surfaces, (**d**) simplified mesh extracted with dual contouring for point clouds, and (**e**) textured mesh with the smallest viewing angle difference to the color camera.  
sensor also captures depth images, surface reconstruction can be performed by fusing the acquired point clouds using the estimated camera poses [28].  
If multiple static, calibrated depth cameras capture a scene, the acquired images can be easily projected into a common reference frame to form an unstructured surface point cloud. A vast amount of work [29] for the reconstruction of implicit surfaces from point clouds is available. One popular formulation is radial basis functions [30; 31], which are continuous interpolation functions representing a single smooth and manifold object. Different approaches, such as Poisson reconstruction [32; 33], Fourier-based reconstruction [34; 35], smooth signed distance surface reconstruction [12], or multilevel partition of unity [36], present different behaviors in terms of robustness, smoothness, and run-time complexity.  
Our work focuses on setups consisting of multiple stationary cameras for dynamic environments to capture a room from multiple perspectives as input for surface reconstruction algorithms that work in real time. We further aim to simplify the resulting surface mesh to lower the bandwidth requirements for the communication network while keeping sufficient surface detail to enable medical teleconsultation.