Error Correction for Time-of-Flight Images Using Validity Classification

In this paper, we devise a method that reduces distance errors in time-of-flight (ToF) images. Errors are exhibited at boundaries and surfaces that are not capable of reflecting the infrared ray. For the proposed method, at least two ToF cameras are required in the camera setup. ToF distance error region is estimated by comparing the captured ToF image with warped ToF image from the neighboring ToF camera. The distance values in the error region are replaced. A number of methods are examined to select the optimum replacement value. After distance error reduction, this method is inserted into the aforementioned depth map generation framework. The performance is analyzed by evaluating a synthetic image which is generated by the depth map result. Introduction In general, 3D video adds depth perception to 2D video, providing a realistic feel. The market for 3D video has grown extensively since the successes of numerous 3D commercial films in the late 2000s. Now with more advanced technologies, content providers attract customers with marketable 3D contents and 3D displays such as stereoscopic or auto-stereoscopic displays reconstruct satisfactory 3D video. Fig. 1 shows 3D services using a variety of displays. Generally, the process of left and right eyes seeing slightly different scenes achieves 3D experience [1]. In other words, 3D perception is derived from two separate views. 3D displays such as 3DTV on the market provide stereoscopic images to users. In the near future, users will most likely be able to experience 3D perceptional depth freely. The 3D video system transmits compressed N views of color and depth video data. At the receiver end, M views are generated based on the N decoded views and synthesized views. Hence, the number of output views is always greater than that of input views [8]. For view synthesis, decoded color and depth video data are used as input in depth image based rendering. Thus, the quality of decoded color and depth data directly affects the quality of rendered images. Depth images present the distance between the camera and the object. Generally, depth images are produced by depth cameras or estimated by stereo matching. Depth cameras allow fast data acquisition but cost can be an issue. In addition, interference must be checked which is caused by frequency overlaps. Stereo matching does not have limitations of depth cameras [9], [10]; however, it can be time-consuming, thus, not suitable for practical applications. Figure 1. Framework of a 3D video system including 3D content production and depth image based rendering Depth Map Generation Using ToF Images ToF cameras are useful for obtaining object distances in a scene [11, 12]. Yet, they can produce low resolution images only. In order to match the resolution of depth images and color images, ToF-to-color view 3D warping is applied using ToF and color camera parameters. Intrinsic and extrinsic camera parameters are necessary for this process [13]. Intrinsic camera parameters include focal length, principal point, and skew coefficients. In addition, extrinsic camera parameters represent rotation and translation characteristics of the camera. In the ToF image, each pixel is projected to a 3D point, i.e., world coordinate, then this 3D point is projected to the destination image. For ToF-to-color 3D warping, the ToF camera and the color camera are source and destination, respectively. The 2D image coordinate at the destination image is acquired by the 2D image point at the source image and its intrinsic and extrinsic camera parameters. From depth image warping, the depth image corresponding to the color camera view is acquired. These images contain empty pixels due to the resolution difference between ToF and color cameras. Joint bilateral filter (JBF) is used to fill such areas [15]. The filter is applied to the object region only, assuming the approximate depth value range of the object is known. JBF is an extension of the bilateral filter [14], which is widely used for edge preservation. 3D Content Generation Depth Image Based Rendering Stereo camera Multi-view camera Depth camera D M-view 3D display N-view color + depth Encoding Decoding Decoded N-view color + depth IS&T International Symposium on Electronic Imaging 2018 Image Processing: Algorithms and Systems XVI 464-1 https://doi.org/10.2352/ISSN.2470-1173.2018. 13.IPAS-464 © 2018, Society for Imaging Science and Technology