Real-time embedded algorithms for local tone mapping of high dynamic range images (2007)

Digital imaging is a widespread and powerful technology, and yet the quality of digital images suffer from a fundamental mismatch in the physical devices used in imaging systems: new technology camera sensors can capture scenes of much wider dynamic range than standard display devices can display. The result of this mismatch is that printed and displayed images are commonly lacking in clear detail. Cameras are able to capture the detail in, for example, both shaded and sunlit areas of a natural scene, but when the image is displayed either the sunlit area is saturated to become uniformly bright, or the shaded area is saturated to become uniformly dark. The problem is even more pronounced for synthetic images produced by fusing multiple views of a scene. Image processing is required to compress the dynamic range of the image before it is displayed, and to control its brightness and contrast. Ideally, this processing would be done in real-time, and embedded within the camera or display. 
Published algorithms for processing images of wide dynamic range have been implemented mostly on workstations and applied to stored images, without hardware and timing constraints. The algorithms have been developed without taking into account the difficulties inherent in hardware implementation, and are not suitable for implementation in embedded systems that work in real time. This limits their application. This work proposes to develop algorithms for processing images of high dynamic range that are suitable for implementation on embedded systems, and for real-time applications. 
One difficulty in developing real-time algorithms for image enhancement is the interdisciplinary nature of the work. Both image processing and digital hardware implementation aspects must be carefully considered, and these two aspects are intertwined so that they must be considered simultaneously. It is not sufficient to simply translate algorithms that perform well in floating-point mathematics on workstations into hardware; the hardware in a real-time application so constrains what can be done that the algorithm and the hardware must be developed hand-in-hand. The quality of the system as a whole depends heavily on both signal processing and hardware performance aspects. 
The intellectual merit of the work lies in the following key concepts. Algorithms for local tone mapping of high dynamic range images that are uniquely appropriate for implementation on embedded systems and can do the processing in real-time are developed. Both hardware performance and signal processing aspects of the design are simultaneously considered so that system-level trade-offs can be made. The work also addresses an additional and crucial manifestation of computing implementations: the effect of fixed-point quantization and computational structure on the quality with which a digital signal processing algorithm is implemented. 
The final outcome of our research is a local tone mapping operator, implemented on an FPGA that can process a 3×32-bit high dynamic range color image of 1024×768 pixels. It has a maximum operating frequency of 65.86 MHz, which is compatible with a video rate of 60 frames per second. The quality of the images processed by this operator was compared to a floating-point state-of-the-art local tone mapper considered to be a gold standard. Images processed by this operator had a peak signal-to-noise ratio of about 35dB on average with respect to the gold standard. The operator requires relatively simple hardware, and is well suited for embedded real-time application.