Mixing tone mapping operators on the GPU by differential zone mapping based on psychophysical experiments

In this paper, we present a new technique for displaying High Dynamic Range (HDR) images on Low Dynamic Range (LDR) displays in an efficient way on the GPU. The described process has three stages. First, the input image is segmented into luminance zones. Second, the tone mapping operator (TMO) that performs better in each zone is automatically selected. Finally, the resulting tone mapping (TM) outputs for each zone are merged, generating the final LDR output image. To establish the TMO that performs better in each luminance zone we conducted a preliminary psychophysical experiment using a set of HDR images and six different TMOs. We validated our composite technique on several (new) HDR images and conducted a further psychophysical experiment, using an HDR display as the reference that establishes the advantages of our hybrid three-stage approach over a traditional individual TMO. Finally, we present a GPU version, which is perceptually equal to the standard version but with much improved computational performance.

Related papers:

title: Dynamic range compression by differential zone mapping based on psychophysical experiments
abstract: In this paper we present a new technique for the display of High Dynamic Range (HDR) images on Low Dynamic Range (LDR) displays. The described process has three stages. First, the input image is segmented into luminance zones. Second, the tone mapping operator (TMO) that performs better in each zone is automatically selected. Finally, the resulting tone mapping (TM) outputs for each zone are merged, generating the final LDR output image. To establish the TMO that performs better in each luminance zone we conducted a preliminary psychophysical experiment using a set of HDR images and six different TMOs. We validated our composite technique on several (new) HDR images and conducted a further psychophysical experiment, using an HDR display as reference, that establishes the advantages of our hybrid three-stage approach over a traditional individual TMO.

title: Objective Quality Assessment of Tone-Mapped Images
abstract: Tone-mapping operators (TMOs) that convert high dynamic range (HDR) to low dynamic range (LDR) images provide practically useful tools for the visualization of HDR images on standard LDR displays. Different TMOs create different tone-mapped images, and a natural question is which one has the best quality. Without an appropriate quality measure, different TMOs cannot be compared, and further improvement is directionless. Subjective rating may be a reliable evaluation method, but it is expensive and time consuming, and more importantly, is difficult to be embedded into optimization frameworks. Here we propose an objective quality assessment algorithm for tone-mapped images by combining: 1) a multiscale signal fidelity measure on the basis of a modified structural similarity index and 2) a naturalness measure on the basis of intensity statistics of natural images. Validations using independent subject-rated image databases show good correlations between subjective ranking score and the proposed tone-mapped image quality index (TMQI). Furthermore, we demonstrate the extended applications of TMQI using two examples - parameter tuning for TMOs and adaptive fusion of multiple tone-mapped images.

title: Tone reproduction for realistic images
abstract: Radiosity and other global illumination methods for image synthesis calculate the real world radiance values of a scene instead of the display radiance values that will represent them. Though radiosity and ray tracing methods can compute extremely accurate and wide-ranging scene radiances, modern display devices emit light only in a tiny fixed range. The radiances must be converted, but ad-hoc conversions cause serious errors and give little assurance that the evoked visual sensations are truly equivalent. Sensation-preserving conversions for display, already known in photography, printing, and television as tone reproduction methods, are discussed. Computer graphics workers can apply the existing photographic methods, but may also extend them to include more complex and subtle effects of human vision using the published findings of vision researchers. Ways of constructing a sensation-preserving display converter, or tone reproduction operator, for monochrome images are demonstrated.<<ETX>>

title: Advanced High Dynamic Range Imaging: Theory and Practice
abstract: Imaging techniques seek to simulate the array of light that reaches our eyes to provide the illusion of sensing scenes directly. Both photography and computer graphics deal with the generation of images. Both disciplines have to cope with the high dynamic range in the energy of visible light that human eyes can sense. Traditionally photography and computer graphics took different approaches to the high dynamic range problem. Work over the last ten years though has unified these disciplines and created powerful new tools for the creation of complex, compelling and realistic images. This book provides a practical introduction to the emerging new discipline of high dynamic range imaging that combines photography and computer graphics. By providing detailed equations and code, the book gives the reader the tools needed to experiment with new techniques for creating compelling images. A supplemental website contains downloads and additional information.

title: Multidimensional image retargeting
abstract: Retargeting refers to the process by which an image or video is adapted from the display device for which it was meant (target display) to another one (retarget display). The retarget display has different features from the target one such as dynamic range, discretization levels, color gamut, multi-view, and refresh rate spatial resolution. This is a very relevant topic in graphics, given the increasing number of display devices from large, high-contrast screens to small cell phones with limited dynamic range; a lot of techniques are being published in different venues, and it's hard to keep up. For most cases retargeting can be an ill-posed problem, for example in the process of displaying Low Dynamic Range (LDR) or 8-bit content on High Dynamic Range (HDR) displays. Such a problem requires the retargeting algorithm to generate new content which is missing in the input image/frame. In this course, we will present the latest solutions and techniques for retargeting images along various dimensions such as dynamic range, colors, temporal and spatial resolutions, and for the first time offer a much-needed holistic view of the field. Moreover, we are going to show how to measure and analyze the changes applied to an image or video in terms of quality using both psychophysical experiments (subjective) and computational metrics (objective). The course should be of interest to anyone involved in graphics in a broader sense, given the almost unavoidable need to retarget results to different devices -from developers interested in implementing retargeting techniques, to users that just need an overall perspective. For researchers fully engaged in developing multi-dimensional retargeting techniques, this course will serve as a solid background for future algorithms.

title: A visibility matching tone reproduction operator for high dynamic range scenes
abstract: The authors present a tone reproduction operator that preserves visibility in high dynamic range scenes. The method introduces a new histogram adjustment technique, based on the population of local adaptation luminances in a scene. To match subjective viewing experience, the method incorporates models for human contrast sensitivity, glare, spatial acuity and color sensitivity. They compare the results to previous work and present examples the techniques applied to lighting simulation and electronic photography.

title: Segmentation and adaptive assimilation for detail-preserving display of high-dynamic range images
abstract: Realistic display of high-dynamic range images is a difficult problem. Previous methods for high-dynamic range image display suffer from halo artifacts or are computationally expensive. We present a novel method for computing local adaptation luminance that can be used with several different visual adaptation-based tone-reproduction operators for displaying visually accurate high-dynamic range images. The method uses fast image segmentation, grouping, and graph operations to generate local adaptation luminance. Results on several images show excellent dynamic range compression, while preserving detail without the presence of halo artifacts. With adaptive assimilation, the method can be configured to bring out a high-dynamic range appearance in the display image. The method is efficient in terms of processor and memory use.

title: Lightness Perception in Tone Reproduction for High Dynamic Range Images
abstract: An anchoring theory of lightness perception comprehensively explains many characteristics of human visual system such as lightness constancy and its spectacular failures which are important in the perception of images. We present a novel approach to tone mapping of high dynamic range (HDR) images which is inspired by the anchoring theory. The key concept of this method is the decomposition of an HDR image into areas (frameworks) of consistent luminance and the local calculation of the lightness values. The net lightness of an image is calculated via the merging of the frameworks proportionally to their strength. We stress out the importance of relating the luminance to a known brightness value (anchoring) and investigate the advantages of anchoring to the luminance value perceived as white. We validate the accuracy of the lightness reproduction in the presented algorithm by simulating a well known perception experiment. Our approach does not affect the local contrast and preserves the natural colors of an HDR image due to the linear handling of luminance.

title: An anchoring theory of lightness perception.
abstract: A review of the field of lightness perception from Helmholtz to the present shows the most adequate theories of lightness perception to be the intrinsic image models. Nevertheless, these models fail on 2 important counts: They contain no anchoring rule, and they fail to account for the pattern of errors in surface lightness. Recent work on both the anchoring problem and the problem of errors has produced a new model of lightness perception, one that is qualitatively different from the intrinsic image models. The new model, which is based on a combination of local and global anchoring of lightness values, appears to provide an unprecedented account of a wide range of empirical results, both classical and recent, especially the pattern of errors. It provides a unified account of both illumination-dependent failures of constancy and background-dependent failures of constancy, resolving a number of long-standing puzzles.

title: Interactive local adjustment of tonal values
abstract: This paper presents a new interactive tool for making local adjustments of tonal values and other visual parameters in an image. Rather than carefully selecting regions or hand-painting layer masks, the user quickly indicates regions of interest by drawing a few simple brush strokes and then uses sliders to adjust the brightness, contrast, and other parameters in these regions. The effects of the user's sparse set of constraints are interpolated to the entire image using an edge-preserving energy minimization method designed to prevent the propagation of tonal adjustments to regions of significantly different luminance. The resulting system is suitable for adjusting ordinary and high dynamic range images, and provides the user with much more creative control than existing tone mapping algorithms. Our tool is also able to produce a tone mapping automatically, which may serve as a basis for further local adjustments, if so desired. The constraint propagation approach developed in this paper is a general one, and may also be used to interactively control a variety of other adjustments commonly performed in the digital darkroom.

title: Exposure Fusion
abstract: We propose a technique for fusing a bracketed exposure sequence into a high quality image, without converting to HDR first. Skipping the physically-based HDR assembly step simplifies the acquisition pipeline. This avoids camera response curve calibration and is computationally efficient. It also allows for including flash images in the sequence. Our technique blends multiple exposures, guided by simple quality measures like saturation and contrast. This is done in a multiresolution fashion to account for the brightness variation in the sequence. The resulting image quality is comparable to existing tone mapping operators.

title: Perception Motivated Hybrid Approach to Tone Mapping
abstract: We propose a general hybrid approach to the issue of reproduction of high dynamic range images on devices with limited dynamic range. Our approach is based on combination of arbitrary global and local tone mapping operators. Recent perceptual studies concerning the reproduction of HDR images have shown high importance of preservation of overall image attributes. Motivated by these findings, we apply the global method first to reproduce overall image attributes correctly. At the same time, an enhancement map is constructed to guide a local operator to the critical areas that deserve enhancement. Based on the choice of involved methods and on the manner of construction of an enhancement map, we show that our approach is general and can be easily tailored to miscellaneous goals of tone mapping. An implementation of proposed hybrid tone mapping produces good results, it is easy to implement, fast to compute and it is comfortably scalable, if desired. These qualities nominate our approach for utilization in time-critical HDR applications like interactive visualizations, modern computer games, HDR image viewers, HDR mobile devices applications, etc.

title: A selective approach for tone mapping high dynamic range content
abstract: Large resolution High Dynamic Range (HDR) textures/images are becoming common in computer graphics applications. However, it is still a prohibitive task to perform on them a filtering step such as tone mapping in real-time. We introduce a novel technique called Selective Tone Mapper (STM) for accelerating the tone mapping step on large resolution HDR images. This technique presents an efficient memory usage, when compared with traditional methods, due to the reduced information required to localize the strong contrast areas.

title: Selective local tone mapping
abstract: When preparing high dynamic range images (HDR) for display on standard monitors, it is often necessary to make a choice between global and local tone mapping. While the former is simple and efficient, it may fail to reproduce details in high contrast image regions. Although, the latter can better reproduce details in such regions, it often comes at the cost of increased complexity and computational time. In this paper, we present an algorithm that combines the best of both approaches. We perform local tone mapping only in high frequency image regions where the visibility of details can be an issue. In low frequency regions, we employ global tone mapping to save computational resources without degrading quality. Our algorithm is most suitable for tone mapping operators (TMOs) that utilize the concept of local adaptation luminances.

title: Temporal coherency for video tone mapping
abstract: Tone Mapping Operators (TMOs) aim at converting real world high dynamic range (HDR) images captured with HDR cameras, into low dynamic range (LDR) images that can be displayed on LDR displays. Several TMOs have been proposed over the last decade, from the simple global mapping to the more complex one simulating the human vision system. While these solutions work generally well for still pictures, they are usually less e_cient for video sequences as they are source of visual artifacts. Only few of them can be adapted to cope with a sequence of images. In this paper we present a major problem that a static TMO usually encounters while dealing with video sequences, namely the temporal coherency. Indeed, as each tone mapper deals with each frame separately, no temporal coherency is taken into account and hence the results can be quite disturbing for high varying dynamics in a video. We propose a temporal coherency algorithm that is designed to analyze a video as a whole, and from its characteristics adapts each tone mapped frame of a sequence in order to preserve the temporal coherency. This temporal coherency algorithm has been tested on a set of real as well as Computer Graphics Image (CGI) content and put in competition with several algorithms that are designed to be time-dependent. Results show that temporal coherency preserves the overall contrast in a sequence of images. Furthermore, this technique is applicable to any TMO as it is a post-processing that only depends on the used TMO.

title: Photographic tone reproduction for digital images
abstract: A classic photographic task is the mapping of the potentially high dynamic range of real world luminances to the low dynamic range of the photographic print. This tone reproduction problem is also faced by computer graphics practitioners who map digital images to a low dynamic range print or screen. The work presented in this paper leverages the time-tested techniques of photographic practice to develop a new tone reproduction operator. In particular, we use and extend the techniques developed by Ansel Adams to deal with digital images. The resulting algorithm is simple and produces good results for a wide variety of images.

title: Delivering Interactivity to Complex Tone Mapping Operators
abstract: The accurate display of high dynamic range images requires the application of complex tone mapping operators. These operators are computationally costly, which prevents their usage in interactive applications. We propose a general framework that delivers interactive performance to an important subclass of tone mapping operators, namely global tone mapping operators. The proposed framework consists of four steps: sampling the input image, applying the tone mapping operator, fitting the point-sampled tone mapping curve, and reconstructing the tone mapping curve for all pixels of the input image. We show how to make use of recent graphics hardware while keeping the advantage of generality by performing tone mapping in software. We demonstrate the capabilities of our method by accelerating several common global tone mapping operators and integrating the operators in a real-time rendering application.

title: Interactive time-dependent tone mapping using programmable graphics hardware
abstract: Modern graphics architectures have replaced stages of the graphics pipeline with fully programmable modules. Therefore, it is now possible to perform fairly general computation on each vertex or fragment in a scene. In addition, the nature of the graphics pipeline makes substantial computational power available if the programs have a suitable structure. In this paper, we show that it is possible to cleanly map a state-of-the-art tone mapping algorithm to the pixel processor. This allows an interactive application to achieve higher levels of realism by rendering with physically based, unclamped lighting values and high dynamic range texture maps. We also show that the tone mapping operator can easily be extended to include a time-dependent model, which is crucial for interactive behavior. Finally, we describe the ways in which the graphics hardware limits our ability to compress dynamic range efficiently, and discuss modifications to the algorithm that could alleviate these problems.

title: Real-Time Photographic Local Tone Reproduction Using Summed-Area Tables
abstract: High dynamic range (HDR) rendering is becoming an increasingly popular technique in computer graphics. Its challenge consists in mapping the resulting images’ large range of intensities to the much narrower ones of the display devices in a way that preserves contrastive details. Local tone mapping operators effectively perform the required compression by adapting the luminance level of each pixel with respect to its local neighborhood. While they generate significantly better results when compared to global operators, their computational costs are considerably higher, which has prevented their use in real-time applications. This paper presents a real-time technique for photographic local tone reproduction that runs entirely on the GPU and is significantly faster than existing implementations that produce similar results. Our approach is based on the use of summed-area tables for accelerating the convolution of the local neighborhoods with a box filter and provides an attractive solution for HDR rendering applications that require high performance without compromising image quality.

title: High dynamic range imaging pipeline on the GPU
abstract: Use of high dynamic range (HDR) images and video in image processing and computer graphics applications is rapidly gaining popularity. However, creating and displaying high resolution HDR content on CPUs is a time consuming task. Although some previous work focused on real-time tone mapping, implementation of a full HDR imaging (HDRI) pipeline on the GPU has not been detailed. In this article we aim to fill this gap by providing a detailed description of how the HDRI pipeline, from HDR image assembly to tone mapping, can be implemented exclusively on the GPU. We also explain the trade-offs that need to be made for improving efficiency and show timing comparisons for CPU versus GPU implementations of the HDRI pipeline.

title: SLIC Superpixels Compared to State-of-the-Art Superpixel Methods
abstract: Computer vision applications have come to rely increasingly on superpixels in recent years, but it is not always clear what constitutes a good superpixel algorithm. In an effort to understand the benefits and drawbacks of existing methods, we empirically compare five state-of-the-art superpixel algorithms for their ability to adhere to image boundaries, speed, memory efficiency, and their impact on segmentation performance. We then introduce a new superpixel algorithm, simple linear iterative clustering (SLIC), which adapts a k-means clustering approach to efficiently generate superpixels. Despite its simplicity, SLIC adheres to boundaries as well as or better than previous methods. At the same time, it is faster and more memory efficient, improves segmentation performance, and is straightforward to extend to supervoxel generation.

title: Evaluation of tone mapping operators using a High Dynamic Range display
abstract: Tone mapping operators are designed to reproduce visibility and the overall impression of brightness, contrast and color of the real world onto limited dynamic range displays and printers. Although many tone mapping operators have been published in recent years, no thorough psychophysical experiments have yet been undertaken to compare such operators against the real scenes they are purporting to depict. In this paper, we present the results of a series of psychophysical experiments to validate six frequently used tone mapping operators against linearly mapped High Dynamic Range (HDR) scenes displayed on a novel HDR device. Individual operators address the tone mapping issue using a variety of approaches and the goals of these techniques are often quite different from one another. Therefore, the purpose of this investigation was not simply to determine which is the "best" algorithm, but more generally to propose an experimental methodology to validate such operators and to determine the participants' impressions of the images produced compared to what is visible on a high contrast ratio display.

title: Two methods for display of high contrast images
abstract: High contrast images are common in night scenes and other scenes that include dark shadows and bright light sources. These scenes are difficult to display because their contrasts greatly exceed the range of most display devices for images. As a result, the image constrasts are compressed or truncated, obscuring subtle textures and details. Humans view and understand high contrast scenes easily, “adapting” their visual response to avoid compression or truncation with no apparent loss of detail. By imitating some of these visual adaptation processes, we developed methods for the improved display of high-contrast images. The first builds a display image from several layers of lighting and surface properties. Only the lighting layers are compressed, drastically reducing contrast while preserving much of the image detail. This method is practical only for synthetic images where the layers can be retained from the rendering process. The second method interactively adjusts the displayed image to preserve local contrasts in a small “foveal” neighborhood. Unlike the first method, this technique is usable on any image and includes a new tone reproduction operator. Both methods use a sigmoid function for contrast compression. This function has no effect when applied to small signals but compresses large signals to fit within an asymptotic limit. We demonstrate the effectiveness of these approaches by comparing processed and unprocessed images.

title: Fast bilateral filtering for the display of high-dynamic-range images
abstract: We present a new technique for the display of high-dynamic-range images, which reduces the contrast while preserving detail. It is based on a two-scale decomposition of the image into a base layer,...

title: Adaptive Logarithmic Mapping For Displaying High Contrast Scenes
abstract: We propose a fast, high quality tone mapping technique to display high contrast images on devices with limited dynamicrange of luminance values. The method is based on logarithmic compression of luminance values, imitatingthe human response to light. A bias power function is introduced to adaptively vary logarithmic bases, resultingin good preservation of details and contrast. To improve contrast in dark areas, changes to the gamma correctionprocedure are proposed. Our adaptive logarithmic mapping technique is capable of producing perceptually tunedimages with high dynamic content and works at interactive speed. We demonstrate a successful application of ourtone mapping technique with a high dynamic range video player enabling to adjust optimal viewing conditions forany kind of display while taking into account user preference concerning brightness, contrast compression, anddetail reproduction.

title: Meet iCAM: A Next-Generation Color Appearance Model
abstract: For over 20 years, color appearance models have evolved to the point of international standardization. These models are capable of predicting the appearance of spatially-simple color stimuli under a wide variety viewing conditions and have been applied to images by treating each pixel as an independent stimulus. It has been more recently recognized that revolutionary advances in color appearance modeling would require more rigorous treatment of spatial (and perhaps temporal) appearance phenomena. In addition, color appearance models are often more complex than warranted by the available visual data and limitations in the accuracy and precision of practical viewing conditions. Lastly, issues of color difference measurement are typically treated separate from color appearance. Thus, the stage has been set for a new generation of color appearance models. This paper presents one such model called iCAM, for image color appearance model. The objectives in formulating iCAM were to simultaneously provide traditional color appearance capabilities, spatial vision attributes, and color difference metrics, in a model simple enough for practical applications. The framework and initial implementation of the model are presented along with examples that illustrate its performance for chromatic adaptation, appearance scales, color difference, crispening, spreading, high-dynamic-range tone mapping, and image quality measurement. It is expected that the implementation of this model framework will be refined in the coming years as new data become available.

title: Do HDR displays support LDR content?: a psychophysical evaluation
abstract: The development of high dynamic range (HDR) imagery has brought us to the verge of arguably the largest change in image display technologies since the transition from black-and-white to color television. Novel capture and display hardware will soon enable consumers to enjoy the HDR experience in their own homes. The question remains, however, of what to do with existing images and movies, which are intrinsically low dynamic range (LDR). Can this enormous volume of legacy content also be displayed effectively on HDR displays? We have carried out a series of rigorous psychophysical investigations to determine how LDR images are best displayed on a state-of-the-art HDR monitor, and to identify which stages of the HDR imaging pipeline are perceptually most critical. Our main findings are: (1) As expected, HDR displays outperform LDR ones. (2) Surprisingly, HDR images that are tone-mapped for display on standard monitors are often no better than the best single LDR exposure from a bracketed sequence. (3) Most importantly of all, LDR data does not necessarily require sophisticated treatment to produce a compelling HDR experience. Simply boosting the range of an LDR image linearly to fit the HDR display can equal or even surpass the appearance of a true HDR image. Thus the potentially tricky process of inverse tone mapping can be largely circumvented.

title: Perceptual evaluation of tone mapping operators
abstract: High dynamic range (HDR) photography and physically based rendering produce images with full range of luminance data. The problem of mapping real world luminance into the limited luminance range of display devices has been addressed in the last decade through the development of many different tone mapping algorithms (see [Devlin et al. 2002] for a recent survey). Clearly, a sound methodology for comparing existing algorithms is needed to understand their strengths and weaknesses; this work takes a first exploratory step towards achieving this goal. We performed a series of psychophysical experiments in which human subjects assessed their perceptions associated with a set of 24 images, constructed by submitting four different scenes (both synthetic and photographic) to six popular tone mapping operators: Photographic Tone Reproduction (Reinhard et al. 2002), Uniform Scaling Quantization (Schlick 1994), Retinex (Frankle et al. 1983), Visual Adjustment (Ferwerda 1996), Revised Tumblin-Rushmeier (1993), and Histogram Adjustment (Ward et al. 1997; the human contrast sensitivity option was switched off). In this initial attempt, we chose an exploratory rather than confirmatory approach in which subjects first made global judgments regarding how perceptually similar or dissimilar the images were, without specifying the ways in which they might differ from one another. Global dissimilarity judgments were made for all pairwise comparisons of the six tone mapping operators separately for each of the four scenes by each of 11 subjects, and these data were submitted to INdividual DifferencesSCALing (INDSCAL) analysis. The primaryINDSCAL result of interest is a derivedStimulus Spacein which each stimulus is assigned coordinates on the dimensions describing the perceptual differences between the stimuli (see [Borg and Groenen 1997] for background on this analysis method). The spatial configuration of the six tone mappers on the two most salient dimensions is shown in

title: Evaluating HDR rendering algorithms
abstract: A series of three experiments has been performed to test both the preference and accuracy of high dynamic-range (HDR) rendering algorithms in digital photography application. The goal was to develop a methodology for testing a wide variety of previously published tone-mapping algorithms for overall preference and rendering accuracy. A number of algorithms were chosen and evaluated first in a paired-comparison experiment for overall image preference. A rating-scale experiment was then designed for further investigation of individual image attributes that make up overall image preference. This was designed to identify the correlations between image attributes and the overall preference results obtained from the first experiments. In a third experiment, three real-world scenes with a diversity of dynamic range and spatial configuration were designed and captured to evaluate seven HDR rendering algorithms for both of their preference and accuracy performance by comparing the appearance of the physical scenes and the corresponding tone-mapped images directly. In this series of experiments, a modified Durand and Dorsey's bilateral filter technique consistently performed well for both preference and accuracy, suggesting that it is a good candidate for a common algorithm that could be included in future HDR algorithm testing evaluations. The results of these experiments provide insight for understanding of perceptual HDR image rendering and should aid in design strategies for spatial processing and tone mapping. The results indicate ways to improve and design more robust rendering algorithms for general HDR scenes in the future. Moreover, the purpose of this research was not simply to find out the “best” algorithms, but rather to find a more general psychophysical experiment based methodology to evaluate HDR image-rendering algorithms. This paper provides an overview of the many issues involved in an experimental framework that can be used for these evaluations.

title: Perceptual evaluation of tone mapping operators with real-world scenes
abstract: A number of successful tone mapping operators for contrast compression have been proposed due to the need to visualize high dynamic range (HDR) images on low dynamic range devices. They were inspired by fields as diverse as image processing, photographic practice, and modeling of the human visual systems (HVS). The variety of approaches calls for a systematic perceptual evaluation of their performance. We conduct a psychophysical experiment based on a direct comparison between the appearance of real-world scenes and HDR images of these scenes displayed on a low dynamic range monitor. In our experiment, HDR images are tone mapped by seven existing tone mapping operators. The primary interest of this psychophysical experiment is to assess the differences in how tone mapped images are perceived by human observers and to find out which attributes of image appearance account for these differences when tone mapped images are compared directly with their corresponding real-world scenes rather than with each other. The human subjects rate image naturalness, overall contrast, overall brightness, and detail reproduction in dark and bright image regions with respect to the corresponding real-world scene. The results indicate substantial differences in perception of images produced by individual tone mapping operators. We observe a clear distinction between global and local operators in favor of the latter, and we classify the tone mapping operators according to naturalness and appearance attributes.

title: Image Attributes and Quality for Evaluation of Tone Mapping Operators
abstract: The problem of reproducing high dynamic range images on devices with restricted dynamic range has gained a lot of interest in the computer graphics community. There exist various approaches to this issue, which span several research areas including computer graphics, image processing, color science, physiology, neurology, psychology, etc. These approaches assume a thorough knowledge of both the objective and subjective attributes of an image. However, no comprehensive overview and analysis of such attributes has been published so far. In this paper, we present an overview of image quality attributes of different tone mapping methods. Furthermore, we propose a scheme of relationships between these attributes, leading to the definition of an overall image quality measure. We present results of subjective psychophysical tests that we have performed to prove the proposed relationship scheme. We also present the evaluation of existing tone mapping methods with regard to these attributes. Our effort is not just useful to get into the tone mapping field or when implementing a tone mapping operator, but it also sets the stage for well-founded quality comparisons between tone mapping operators. By providing good definitions of the different attributes, user-driven or fully automatic comparisons are made possible at all.

title: Evaluating HDR rendering algorithms
abstract: A series of three experiments has been performed to test both the preference and accuracy of high dynamic-range (HDR) rendering algorithms in digital photography application. The goal was to develop a methodology for testing a wide variety of previously published tone-mapping algorithms for overall preference and rendering accuracy. A number of algorithms were chosen and evaluated first in a paired-comparison experiment for overall image preference. A rating-scale experiment was then designed for further investigation of individual image attributes that make up overall image preference. This was designed to identify the correlations between image attributes and the overall preference results obtained from the first experiments. In a third experiment, three real-world scenes with a diversity of dynamic range and spatial configuration were designed and captured to evaluate seven HDR rendering algorithms for both of their preference and accuracy performance by comparing the appearance of the physical scenes and the corresponding tone-mapped images directly. In this series of experiments, a modified Durand and Dorsey's bilateral filter technique consistently performed well for both preference and accuracy, suggesting that it is a good candidate for a common algorithm that could be included in future HDR algorithm testing evaluations. The results of these experiments provide insight for understanding of perceptual HDR image rendering and should aid in design strategies for spatial processing and tone mapping. The results indicate ways to improve and design more robust rendering algorithms for general HDR scenes in the future. Moreover, the purpose of this research was not simply to find out the “best” algorithms, but rather to find a more general psychophysical experiment based methodology to evaluate HDR image-rendering algorithms. This paper provides an overview of the many issues involved in an experimental framework that can be used for these evaluations.

title: Soft Color Segmentation and Its Applications
abstract: We propose an automatic approach to soft color segmentation, which produces soft color segments with an appropriate amount of overlapping and transparency essential to synthesizing natural images for a wide range of image-based applications. Although many state-of-the-art and complex techniques are excellent at partitioning an input image to facilitate deriving a semantic description of the scene, to achieve seamless image synthesis, we advocate a segmentation approach designed to maintain spatial and color coherence among soft segments while preserving discontinuities by assigning to each pixel a set of soft labels corresponding to their respective color distributions. We optimize a global objective function, which simultaneously exploits the reliability given by global color statistics and flexibility of local image compositing, leading to an image model where the global color statistics of an image is represented by a Gaussian mixture model (GMM), whereas the color of a pixel is explained by a local color mixture model where the weights are defined by the soft labels to the elements of the converged GMM. Transparency is naturally introduced in our probabilistic framework, which infers an optimal mixture of colors at an image pixel. To adequately consider global and local information in the same framework, an alternating optimization scheme is proposed to iteratively solve for the global and local model parameters. Our method is fully automatic and is shown to converge to a good optimal solution. We perform extensive evaluation and comparison and demonstrate that our method achieves good image synthesis results for image-based applications such as image matting, color transfer, image deblurring, and image colorization.

title: Dynamic range independent image quality assessment
abstract: The diversity of display technologies and introduction of high dynamic range imagery introduces the necessity of comparing images of radically different dynamic ranges. Current quality assessment metrics are not suitable for this task, as they assume that both reference and test images have the same dynamic range. Image fidelity measures employed by a majority of current metrics, based on the difference of pixel intensity or contrast values between test and reference images, result in meaningless predictions if this assumption does not hold. We present a novel image quality metric capable of operating on an image pair where both images have arbitrary dynamic ranges. Our metric utilizes a model of the human visual system, and its central idea is a new definition of visible distortion based on the detection and classification of visible changes in the image structure. Our metric is carefully calibrated and its performance is validated through perceptual experiments. We demonstrate possible applications of our metric to the evaluation of direct and inverse tone mapping operators as well as the analysis of the image appearance on displays with various characteristics.

title: Bilateral filtering for gray and color images
abstract: Bilateral filtering smooths images while preserving edges, by means of a nonlinear combination of nearby image values. The method is noniterative, local, and simple. It combines gray levels or colors based on both their geometric closeness and their photometric similarity, and prefers near values to distant values in both domain and range. In contrast with filters that operate on the three bands of a color image separately, a bilateral filter can enforce the perceptual metric underlying the CIE-Lab color space, and smooth colors and preserve edges in a way that is tuned to human perception. Also, in contrast with standard filtering, bilateral filtering produces no phantom colors along edges in color images, and reduces phantom colors where they appear in the original image.

title: Multiscale structural similarity for image quality assessment
abstract: The structural similarity image quality paradigm is based on the assumption that the human visual system is highly adapted for extracting structural information from the scene, and therefore a measure of structural similarity can provide a good approximation to perceived image quality. This paper proposes a multiscale structural similarity method, which supplies more flexibility than previous single-scale methods in incorporating the variations of viewing conditions. We develop an image synthesis method to calibrate the parameters that define the relative importance of different scales. Experimental comparisons demonstrate the effectiveness of the proposed method.

title: Image quality assessment: from error visibility to structural similarity
abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

title: High dynamic range video
abstract: Typical video footage captured using an off-the-shelf camcorder suffers from limited dynamic range. This paper describes our approach to generate high dynamic range (HDR) video from an image sequence of a dynamic scene captured while rapidly varying the exposure of each frame. Our approach consists of three parts: automatic exposure control during capture, HDR stitching across neighboring frames, and tonemapping for viewing. HDR stitching requires accurately registering neighboring frames and choosing appropriate pixels for computing the radiance map. We show examples for a variety of dynamic scenes. We also show how we can compensate for scene and camera movement when creating an HDR still from a series of bracketed still photographs.