Removing Highlight Spots in Visual Hull Rendering

Highlight spots could often cause artifacts in image-based visual hull (IBVH) rendering. In this paper, we propose a method that can remove highlight spots from reference images by utilizing the features of visual hull. First, we find the counterparts of a highlight sub-image in other images, with the aid of the correspondence between reference images in visual hull rendering. Then the illumination and color detail of the pixels in the subimage could be retrieved from their corresponding pixels by color blending. Thus, highlight spots are removed seamlessly without affecting other regions of the scene. Using this improved rendering method, we can obtain more realistic and precise result, and relighting of the visual hull also becomes possible. Introduction Visual hull is an efficient technique for image-based rendering. Its main idea comes from 3D reconstruction methods of “Shape-from-Silhouette”. It takes a group of photos as input, and produces a convex hull of the target object. Same as other imagebased methods, visual hull also suffers from the highlight spots on the images, because they could often cause undesired artifacts in rendering, especially when the illumination in the virtual environment is changed. In literatures, some image editing methods have been proposed to remove highlight spots from photos. Some of them need more than one photo at a single position to complete this task. For instance, the method of Agrawal et al [1] requires a pair of flash and ambient images, taken at the same position, and reduces highlights by comparing their image gradients. This kind of method is not practical in visual hull rendering, because it will largely increase the difficulty of acquiring and storing source images. Some other methods could work on a single image, such as [2] and [3]. The former introduces illumination-based constrains into image inpainting, and the latter changes local illumination by solving Poisson equations. However, they only report good results when highlight spots lie in areas with simple or uniformly textured background. On another aspect, single-image-based methods cannot guarantee the consistency of corresponding areas in different source images, which is very important in visual hull rendering. In fact, visual hull method itself provides much convenience for highlight removal. The source images, also called reference images, of a visual hull often have much overlaps. These overlaps will offer sufficient information to remove highlights. That is because the counterpart of a highlighted area in another image is often out of highlight, due to the relative movement of the object and the light. Utilizing the calibration information of the cameras, the correspondence of pixels in different images could be found during the constructing of the visual hull. Thus, highlighted areas could be resampled from other reference images, and a new reference image without highlight spots could be generated by certain pixel blending strategy. With these new images, we could obtain more realistic and precise rendering result, and re-lighting of the visual hull is also possible. In this paper, we employ an Image-based Visual Hull (IBVH) rendering method [4], which is described in the second section. The approach to interactively removing highlight spots in IBVH rendering is detailed in the third section, and the experiment results follow in the last. Visual Hull Rendering Methods The research on visual hull reconstruction dates back to 1970s [5]. Its main idea comes from “Shape-from-Silhouette”. A reference image is separated into foreground and background. The foreground mask, i.e. silhouette, along with the calibration information of the camera, defines a back-projected cone in 3D space that contains the target object. Thus, the intersection of all silhouette cones forms a convex hull, which is the visual hull of the object. There are mainly two sorts of visual hull construction methods: voxel-based and boundary-based methods. [6] Voxel-based methods [7, 8] usually start from a working volume, which contains the target object and is quantized into voxels. The voxels are one by one put through tests: those lie inside all silhouette cones are preserved; the others are cleared. Therefore, the remaining voxels form the visual hull of the object. These methods are able to reconstruct very complex objects, such as trees. However, they usually suffer from quantization artifacts and are very expensive in both space and time costs. In boundary-based methods [4, 9], silhouette cones are represented as boundary elements, such as surfaces or lines. The visual hull is constructed by computing the intersection of these elements, and the result could be composed by a group of surfaces patches, line segments, or points. Such methods usually consume little memory and run faster than voxel-based methods, and quantization artifacts are avoided. If necessary, boundaryrepresented visual hull could also be triangulated into meshes to produce an explicit 3D model. That makes it more useful in applications. However, these methods cannot reconstruct concave or very complex objects. Image-based Visual Hull Rendering Method In this paper, we adopt a method called Image-based Visual Hull (IBVH). It is a boundary-based method, originally proposed by Matusik et al [4]. In this algorithm, silhouettes are represented as a bench of 3D rays emitted from a desired view, and the resulting visual hull is made up of a group of line segments. It is remarkable that the computing is limited to the image space of the reference images, and the result is view-dependent, which makes the method quite efficient. As illustrated in Fig. 1, the kernel algorithm of IBVH method includes three main steps: First, for every pixel of the desired view, a viewing ray is calculated, which is emitted from the virtual camera center and passes through current pixel. By using the calibration information of cameras and epipolar geometry theory, the projection of this viewing ray on a reference image, i.e. the epipolar line could be computed. Second, the epipolar line intersects the 2D silhouette on the reference image, and results in a group of 2D intervals. To reduce the computation cost and increase speed, the original algorithm sorts silhouette edges in so-called bin structures by their slopes. In our implementation, we adopt a different strategy in building bins: instead of slopes, the edges are sorted by their direction angles, so that all edges could be properly sorted, even when the epipole (the intersection point of all epipolar lines) falls in the object area. Third, the 2D intervals on the reference image are projected back to 3D space and get corresponding 3D segments on the viewing ray. Then the intersection of all the 3D segments from all the reference images indicates the visual hull boundary at current pixel of desired view. Projecting the nearest endpoint of final 3D segments onto the reference images, the color of current pixel could be synthesized. Removing Highlight Spots on Visual Hull The IBVH rendering method provides great convenience in finding pixel correspondence between reference images. Utilizing this correspondence, we could rapidly remove highlight spots by resampling the target image, using the information from its counterparts on other reference images. To reduce computing cost and minimize the error introduced by pixel mapping and resampling, we first select sub-images that contain highlight spots interactively. The resampling performance would be applied only to these selected highlight sub-images, and the rest part of the image would remain unchanged. Pixel Correspondence between Reference Images The approach to finding pixel correspondence between different reference images is similar to that of rendering a desired new view in IBVH, as described in the last section. Their fundamental differences lie in: 1) When rendering a desired view, the target image is a new, virtual and synthesized one, but here it is exactly one of the reference images; 2) The calculations are no longer performed over the whole image, but limited to the selected highlight sub-images. Fig. 2 gives an illustration of how pixel correspondences are found. Given a target image I0, for every pixel p0 in a highlight sub-image, we could also compute a 3D viewing ray r. For another reference image Ik, the epipolar line of p0 (denoted as le) is calculated by using the fundamental matrix between I0 and Ik. If line le intersects the silhouette of Ik at point pa k and pb , then there are two 3D rays, ra k and rb k that are emitted from camera Ck and pass through pa k and pb , respectively. Note that r, ra k and rb k are all in the same plane. Therefore, they would intersect and result in a 3D segment (va , vb ). Applying the same calculation to other reference images, we get a group of 3D segments {(va , vb ) | k=1.n} (n is the number of images in use). The intersection of all these segments, denoted as (va, vb), is considered as the intersection of r and the visual hull of the object, and the nearer endpoint va is the corresponding 3D point of pixel p0. Utilizing the calibration information of camera Ck, va can be projected onto image Ik, and get pixel pk. So far, we have found the corresponding pixel of p0 on every reference images. Highlight Sub-image Resampling When the counterparts of a highlight sub-image on other reference images are found, we must re-calculate the color of its pixels to reduce highlight effect. This is completed by blending the appearance colors of pixel p0 and its corresponding pixels {pk | k=1.n}. The fact that we could take advantage of is: usually, due to the relative movement of the object and the light, most part of the Figure 1. The three steps of IBVH computing [4]: 1) A desired viewing ray is projected to a reference image. 2) The epipolar line intersects the silhouette. 3) The resulting 2D intervals are projected back to 3D and get