A Refocus-Interface for Diminished Reality Work Area Visualization

In this paper, we present a refocus interface to set the parameters used for diminished reality (DR)-based work area visualization and a multiview camera-based rendering scheme. The refocus interface allows the user to determine two planes — one for setting a virtual window, through which the user can observe the background occluded by an object, and the other for a background plane, which is used for the subsequent background rendering. The background is rendered considering the geometric and appearance relationships of the multiview cameras observing the scene. Our preliminary results demonstrate that our DR system can visualize the hidden background. Introduction Various handheld tools are used for efficient manual operations in fields like woodworking, cooking, and surgery. However, this benefit comes with the drawback of visual occlusion of the working area, as the devices tend to be large due to their driving parts, joints, batteries, and so on. For example, readers may have experienced situations where a gripped electric drill shielded the work target, thereby making direct observation of the work target from the front impossible. One classical solution to this problem is observing the working area from another viewpoint through mirrors or cameras. However, this method requires cumbersome mental viewpoint conversions from the different view to the eye view; in the case of mirrors, mirror image conversion is also necessary. To overcome this problem, computer-aided viewpoint conversions may be substituted for mental viewpoints. Augmented reality (AR) allows the information of the hidden background to be directly overlaid onto the real environment [1, 2], but the information is described in indirect forms, such as digital annotations. Contrary to this concept of augmentation, visualization technology for making things less noticeable or invisible is called diminished reality (DR) [3, 4, 5]. While our recent DR method achieves high-quality background recovery compared with existing methods, the rendering results have jitters induced by the RGB-D camera [8]. Thus, in this paper, we present a DR system that does not use an RGB-D camera as an alternative for DR-based work area visualization methods. To achieve a method for DR-based background visualization without an RGB-D camera, here, we address the two following issues: 1) how to determine the region to be removed and recovered (i.e., the region of interest (ROI) and background surface detection); and 2) how to render the background based on the retrieved region information (i.e., background recovery). These are described below. ROI and Background Surface Detection: For the ROI and background surface detection, many researchers have used automated [3, 6] or semi-automated methods [7]. Compared with such approaches, in our case (i.e., work area visualization), we suppose that the removal region can be placed in a fixed position, just like a fixed loupe on a workers desk. Thus, we place an AR Magic Lens [9] as a virtual loupe through which we can see the background. The position is determined when the user sweeps the lens manually to show the refocus images, using multiview images as guidance (i.e., the refocused image gives a fine focus image when the focal plane is placed at a certain object surface). We also use this approach for estimating the background surface. Background Recovery: For background recovery, we modify an existing rendering scheme known as unstructured lumigraph rendering (ULR) [10] for our purpose. This image-based rendering method provides an arbitrary viewpoint image by calculating weights for each camera, a camera blending field (CBF), based on the geometric relationships between the calibrated cameras and a geometric proxy, such as a polygon mesh for the scene surfaces. We added an appearance-based term to this formulation to determine whether the camera is observing the background or the occluding object in the working space. Our contributions regarding these issues in DR can be summarized as follows: • A Magic Lensinspired refocus interface to set DR-relevant parameters; and • The geometricand appearance-based blending field calculation of ULR. Refocus-based Visualization Overview Our system requires the user to set up multiview cameras and parameters related to image-based rendering. Figure 1 shows the system workflow. First, the user must arrange the multiview cameras to capture the work area. Then, the system automatically calibrates the cameras using a well-known computer vision technique. After the procedure, the system is ready for rendering and starts to render the scene. Second, the user changes his/her virtual focal plane, which is rendered in the calibrated multiview camera environment, to find a fine focused depth to set a Magic Lens window, where he/she can see through the hidden space. Third, the user again changes his/her virtual focal plane to set the background plane for visualization via the multiview image-based refocusing. Given the depth information induced by the focal plane set by the user, the system renders the scene without occluding objects in the Magic Lens window. Setup and Calibration The system first calibrates multiple cameras to obtain the intrinsic and extrinsic parameters of each camera. First, the system performs feature point extraction and matching between the imIS&T International Symposium on Electronic Imaging 2018 Stereoscopic Displays and Applications XXIX 112-1 https://doi.org/10.2352/ISSN.2470-1173.2018. 04.SDA-112 © 2018, Society for Imaging Science and Technology