High Dynamic Range Imaging: Spatially Varying Pixel Exposures

While real scenes produce a wide range of brightness variations, vision systems use low dynamic range image detectors that typically provide 8 bits of brightness data at each pixel. The resulting low quality images greatly limit what vision can accomplish today. This paper proposes a very simple method for significantly enhancing the dynamic range of virtually any imaging system. The basic principle is to simultaneously sample the spatial and exposure dimensions of image irradiance. One of several ways to achieve this is by placing an optical mask adjacent to a conventional image detector array. The mask has a pattern with spatially varying transmittance, thereby giving adjacent pixels on the detector different exposures to the scene. The captured image is mapped to a high dynamic range image using an efficient image reconstruction algorithm. The end result is an imaging system that can measure a very wide range of scene radiances and produce a substantially larger number of brightness levels, with a slight reduction in spatial resolution. We conclude with several examples of high dynamic range images computed using spatially varying pixel exposures. 1 High Dynamic Range Imaging Any real-world scene has a significant amount of brightness variation within it. The human eye has a remarkable dynamic range that enables it to detect subtle contrast variations and interpret scenes under a large variety of illumination conditions [Blackwell, 1946]. In contrast, a typical video camera, or a digital still camera, provides only about 8 bits (256 levels) of brightness information at each pixel. As a result, virtually any image captured by a conventional imaging system ends up being too dark in some areas and possibly saturated in others. In computational vision, it is such low quality images that we are left with the task of interpreting. Clearly, the low dynamic range of existing image detectors poses a severe limitation on what computational vision can accomplish. This paper presents a very simple modification that can be made to any conventional imaging system to dramatically increases its dynamic range. The availability of extra bits of data at each image pixel is expected to enhance the robustness of vision algorithms. This work was supported in part by an ONR/DARPA MURI grant under ONR contract No. N00014-97-1-0553 and in part by a David and Lucile Packard Fellowship. Tomoo Mitsunaga is supported by the Sony Corporation. 2 Existing Approaches First, we begin with a brief summary of existing techniques for capturing a high dynamic range image with a low dynamic range image detector. 2.1 Sequential Exposure Change The most obvious approach is to sequentially capture multiple images of the same scene using different exposures. The exposure for each image is controlled by either varying the F-number of the imaging optics or the exposure time of the image detector. Clearly, a high exposure image will be saturated in the bright scene areas but capture the dark regions well. In contrast, a low exposure image will have less saturation in bright regions but end up being too dark and noisy in the dark areas. The complementary nature of these images allows one to combine them into a single high dynamic range image. Such an approach has been employed in [Azuma and Morimura, 1996], [Saito, 1995], [Konishi et al., 1995], [Morimura, 1993], [Ikeda, 1998], [Takahashi et al., 1997], [Burt and Kolczynski, 1993], [Madden, 1993]  In [Mann and Picard, 1995], [Debevec and Malik, 1997] and [Mitsunaga and Nayar, 1999] this approach has been taken one step further by using the acquired images to compute the radiometric response function of the imaging system. The above methods are of course suited only to static scenes; the imaging system, the scene objects and their radiances must remain constant during the sequential capture of images under different exposures.  Beam splitters are used to generate multiple copies of the optical image of the scene. Each copy is detected by an image detector whose exposure is preset by using an optical attenuator or by changing the exposure time of the detector. This approach has the advantage of producing high dynamic range images in real time. Hence, the scene objects and the imaging system are free to move during the capture process. The disadvantage of course is that this approach is expensive as it requires multiple image detectors, precision optics for the alignment of all the acquired images and additional hardware for the capture and processing of multiple images. 1063-6919/00 $10.0