Three-dimensional Video Capturing for Realistic Broadcasting Services

In this paper, we present a new three-dimensional (3-D) video capturing system to provide realistic broadcasting services by integrating multiple high-definition (HD) camera arrays and one standard-definition (SD) depth camera. In the proposed hybrid camera system, we first create the initial disparity for each HD cameras by applying 3-D warping operation on the depth map acquired by the depth camera. Then, the final disparity for each HD camera is obtained by a stereo matching algorithm with the initial disparity. Experimental results show that the 3-D video generated by the hybrid camera system provides reliable depth information for 3-D realistic broadcasting services. Besides, the proposed system minimizes the inherent problems of conventional depth cameras, such as limitation of measuring distance for depth information and generation of low-resolution depth maps. INTRODUCTION As the three-dimensional (3-D) video becomes attractive in a variety of multimedia applications, it is essential to get accurate depth information for future 3-D applications. Recently, ISO/IEC JTC1/SC29/WG11 Moving Picture Experts Group (MPEG) has been also recognized the importance of multiview video and depth information, which are often referred as a multiview video-plus-depth, for free-viewpoint TV (FTV) or 3DTV [1] [2]. However, traditional depth estimation methods, such as stereo matching, are limited to estimate accurate depth map due to the failure of correspondence point matching on the textureless and occluded regions. In general, depth estimation methods can be categorized into two major classes: active range sensors [3] and passive range sensors [4]. Passive depth sensors estimate depth information indirectly from 2-D images captured by two or more cameras. On the other hand, active depth sensors obtain depth information from the natural scenes directly using physical sensors. However, these kinds of TOF depth cameras have some built-in problems: low resolution, noisy, and poorly calibrated. Recently, fusion methods that combines multiview camera and a TOF depth camera have been introduced [5]. These fusion camera systems generate enhanced depth maps by applying a stereo matching algorithm to multiview image with depth information captured by the TOF depth camera. However, the previous hybrid camera systems have produced low-resolution depth maps and focused on generating depth maps for static scenes. Since most 3-D applications are expected to use high-resolution videos, it is necessary to create a high-quality multiview video-plus-depth for dynamic scenes. To generate multiview depth map, we propose a new hybrid camera system constructed by one depth camera and multiple video cameras. The proposed system provides multiview high-definition (HD) depth map using depth information acquired from the standard-definition (SD) depth camera as a supplement. The main contribution of this work is that we provide a practical solution to create a high-resolution multiview video-plus-depth using multiple video cameras and a depth camera. In addition, the proposed system reduces the inherent problems of generating depth maps from the currently available depth camera system. HYBRID CAMERA SYSTEM The proposed hybrid camera system consists of six cameras; one depth camera, Z−Cam, and five HD cameras. There is one clock generator sending a synchronization signal constantly. This signal is distributed to all personal computers. Figure 1 shows main components of our hybrid camera system. We obtain test sequences with 1-D parallel camera arrangement from the hybrid camera system. Input videos consist of seven synchronized images; five HD images from the multiview camera, one SD color image and its corresponding depth map from the depth camera. Since the measureable depth range of Z−Cam is up to seven meters, depth accuracy is not guaranteed in the practical environments. The depth range becomes bigger as the quality of depth map becomes lower. To obtain a more accurate depth map from the depth camera, we reduce the depth range by capturing foreground and background, respectively. We capture a color image and its depth map for background in advance. Before obtaining synchronized multiview image from the hybrid camera system, we calibrate each camera using checkerboard patterns independently. There are two coordinate systems: depth camera coordinate system and HD camera coordinate system. Therefore, we register two camera coordinate systems using their relative camera information. To use the initial disparity information for each HD camera, we perform a 3-D warping operation using camera parameters and depth information acquired from the depth camera. Pixel intensities of the warped data are then used as the initial disparity for each view. In multiview camera, all HD images are rectified and color-segmented. The initial disparities generated from the depth camera are assigned into the corresponding segments. We separate each HD image into three different regions to detect occluded and disoccluded regions: background, foreground, and unknown regions. The disparity of each segment is independently estimated by a color segmentation-based stereo matching algorithm. Figure 1. The proposed hybrid camera system GENERATION OF MULTIVIEW DEPTH MAP In the hybrid camera system, the intrinsic and extrinsic camera parameter of each camera are different, since we merge two different types of cameras. Therefore, it is essential to find out relative camera information about the camera set using a camera calibration algorithm. Basically, the camera calibration is executed with pattern images acquired from the hybrid camera. we employ the well-known camera calibration algorithm provided by Caltech to estimate intrinsic and extrinsic parameters of each camera [11]. Since the hybrid camera is composed of six cameras, we carry out the camera calibration as many as the number of cameras independently. Image rectification is a process that makes epipolar lines of two images captured at different position parallel each other. Vertical coordinates of all image points of two images become identical and there remain the horizontal disparities only. For multiview image rectification, the horizontal axis of each camera is parallel to the baseline and the principal axis of each camera is perpendicular to the baseline For rectifing multiview image at the same time, we calculate the common baseline considering all camera positions and apply the rectifying transformation defined by camera rotations and camera intrinsic parameters. Then, rectified images have uniform horizontal disparities and no vertical mismatches between adjacent views [12]. We employ both down-sampling and linear interpolation operations to reduce optical noises in the depth map. After we apply mean filtering on the depth map, the depth data enhancement algorithm executes down-sampling on the mean-filtered depth map. We then perform bilateral filtering on the downsampled depth map. Finally, we recover the depth map using a linear interpolation method. We regard depth information acquired from the depth camera as initial disparity information for the multiview camera. For matching the depth information with its corresponding color value in the multiview image, we perform camera calibration for the multiview camera and the depth camera independently. To calculate the relative positions between the depth camera and the multiview camera, we move the depth camera to the origin of the world coordinate by R R R I (1) t t t 0 (2) Then, we determine the new multiview camera position based on that of the depth camera by Eq. (3) and Eq. (4). We multiply the rotation matrix Rhn of the multiview camera by the inverse matrix R s −1 of the rotation matrix Rs of the depth camera. t′hn is the translational difference between thn and ts. R R R (3) t t t . (4) The 3-D warping matrix to move pixels from the SD depth camera to the HD multiview camera is given by p P P p (5) where phn is the image coordinate in the multiview image corresponding to the ps, and the depth information D(phnx, phny) of phn is followed by D p , p t t D p , p (6) The 3-D warped depth information is used to the initial depth information of the multiview image. To generate the initial disparity map, we apply the 3-D warping in Eq. (7) on the color and depth map from the depth camera. In the 3-D warping operation, we project the color and depth data to the world Depth Camera HD Multi-view Camera SD/HD Sync. Generator