Depth Painting for Image-based Rendering Applications

We describe a system that functions as a painter’s tool for adding a third dimension in single images. In contrast with Horry et al.’s “Tour into the picture” system [5], which adds the third dimension to the single image through polygons and planar regions, our system provides a means for more expressive depth variations that result in more realistic rendering. The key to intuitive “depth painting” is the use of four auxiliary side views in the interface that allows the user to observe direct changes to depth. In addition to a host of local and region-based operations, “depth painting” also involves the option to pad regions within the image so as to remove holes or gaps when viewed at novel camera poses. Finally, the resulting image with depth lends itself to multi-layered imagebased rendering, where new environments can be created from different representations of 3-D scenes and rendered at novel viewpoints in a geometrically-correct manner. Introduction In the video and film industry, the animation process can take any number of various forms: the more conventional cel animation, the use of 3-D models and effects, and image-based rendering. The first two categories of animation techniques are typically laborious and time-consuming, but nevertheless, they create visually-pleasing results. Image-based rendering (IBR) techniques are based on using images as substrates for visualizing in 3-D as well as editing and manipulating 3-D scenes. Techniques that interpolate images in a geometrically consistent fashion are examples of IBR techniques. IBR techniques can obviously also be used to generate special effects that may not be geometrically consistent. It is not unusual for animation techniques to use 2-D layers as independent elements, with more sophisticated techniques employing warping operations to provide illusions of 3-D. However, it is more desirable to assign some reasonable depth distribution to each layer in order to generate more satisfactory 3-D effects. Being able to do this for any single image has the advantage of reusing or repurposing existing images, such as paintings or photos, in the animation process. Work done by Horry et al. [5] helps significantly in this process, though the output of their system is limited to polyhedral and planar approximations only. The proposed idea of depth painting allows users to add depth to single images in an analogous way that painters paint. The “brush,” (whose thickness can be controlled) rather than adding paint, adds or subtracts depth instead. The interface for depth painting has the main view plus four auxiliary views corresponding to left, right, up, and down side views. This facilitates visualization of the user’s progress, and any change is quickly propagated across these views. In addition to a host of local and region-based operations, “depth painting” also involves the option to pad regions within the image so as to remove holes or gaps when viewed at novel camera poses. Depth painting enables single existing images (paintings, photos) to be more easily reused for animation, video editing, or multimedia authoring purposes. Current approaches for repurposing existing images are limited to simple representations, such as planar, affine warps, or polyhedral. Techniques that could assign arbitrary shapes to single images will be very useful in creating more realistic 3-D effects. Our system provides such a solution. Prior approaches Many of the work that dealt with layered images provide suggestions on rendering them efficiently (e.g., [2, 8]. Others dealt with recovering layers from multiple images (e.g., [10]), or recovering 3-D models from multiple images (e.g., [9]). Recently, an approach that allows the user to generate a 3-D effect for single images has been described [5]. This technique, however, allows the creation of an open polyhedral 3-D model only. Arbitrary shapes are more difficult to create using this approach. The depth painting system Figure 1 shows the overall functionality of our “depth painting” system. The system allows interactive specification of regions within the image as well as interactive change of depth. The regions are specified by tracing the edge of what the user perceives as a separate body within the image. Mistakes in region specification can be easily corrected by “brushing” over the pixels in question to remove from or add onto the current region. Obviously, more sophisticated techniques for interactive boundary extraction, such as the “Intelligent Scissors,” [7], can be used. Critical to the system is the display of four auxiliary side views corresponding to left, right, up, and down side views (right of Figure 1). These views help add the third dimension to the input image. They facilitate visualization of the user’s progress; any change in depth can be quickly propagated across these views.