Quality Assessment in Computer Graphics

In this chapter, we review the existing works regarding visual quality assessment in computer graphics. This broad area of research includes many sub-domains which make intensive use of quality assessment and/or artifact visibility evaluation: geometry processing, rendering, HDR imaging, tone-mapping, and stereo vision. For each of these sub-domains, we present the existing objective quality metrics, the subjective quality experiments as well as an evaluation and comparison of their performance. We broadly classify these existing works into image-based (i.e., evaluating artifacts introduced in 2D rendered images and videos) and model-based approaches (i.e., artifacts introduced on the 3D models themselves). Finally, the last part presents the emerging trends and main future directions.

Related papers:

title: Robust and blind mesh watermarking based on volume moments
abstract: This paper presents a robust and blind watermarking algorithm for three-dimensional (3D) meshes. The watermarking primitive is an intrinsic 3D shape descriptor: the analytic and continuous geometric volume moment. During watermark embedding, the input mesh is first normalized to a canonical and robust spatial pose by using its global volume moments. Then, the normalized mesh is decomposed into patches and the watermark is embedded through a modified scalar Costa quantization of the zero-order volume moments of some selected candidate patches. Experimental results and comparisons with the state of the art demonstrate the effectiveness of the proposed approach.

title: Color compatibility from large datasets
abstract: This paper studies color compatibility theories using large datasets, and develops new tools for choosing colors. There are three parts to this work. First, using on-line datasets, we test new and existing theories of human color preferences. For example, we test whether certain hues or hue templates may be preferred by viewers. Second, we learn quantitative models that score the quality of a five-color set of colors, called a color theme. Such models can be used to rate the quality of a new color theme. Third, we demonstrate simple proto-types that apply a learned model to tasks in color design, including improving existing themes and extracting themes from images.

title: Visual equivalence: towards a new standard for image fidelity
abstract: Efficient, realistic rendering of complex scenes is one of the grand challenges in computer graphics. Perceptually based rendering addresses this challenge by taking advantage of the limits of human vision. However, existing methods, based on predicting visible image differences, are too conservative because some kinds of image differences do not matter to human observers. In this paper, we introduce the concept of visual equivalence, a new standard for image fidelity in graphics. Images are visually equivalent if they convey the same impressions of scene appearance, even if they are visibly different. To understand this phenomenon, we conduct a series of experiments that explore how object geometry, material, and illumination interact to provide information about appearance, and we characterize how two kinds of transformations on illumination maps (blurring and warping) affect these appearance attributes. We then derive visual equivalence predictors (VEPs): metrics for predicting when images rendered with transformed illumination maps will be visually equivalent to images rendered with reference maps. We also run a confirmatory study to validate the effectiveness of these VEPs for general scenes. Finally, we show how VEPs can be used to improve the efficiency of two rendering algorithms: Light-cuts and precomputed radiance transfer. This work represents some promising first steps towards developing perceptual metrics based on higher order aspects of visual coding.

title: Perceptually Adaptive Graphics
abstract: In recent years, the Graphics community has come to realise the importance of taking human perception into account when striving for realism in images, animations and Virtual Environments. In May 2001, a EUROGRAPHICS/SIGGRAPH Campfire brought together a group of researchers from various fields, including computer graphics and visualisation, psychology, neuroscience, eye-movements and medicine to discuss the future and current state of the field. Since then, many researchers have been very active in furthering the field of perceptually adaptive graphics. In this report, we outline the state of the art as discussed at that event and the progress that has been made since.

title: A frequency based ray tracer
abstract: A ray tracer has been developed that synthesizes images directly into the frequency domain. This makes it possible to use a simple vision model to control where rays are cast into a scene and to decide how rays should be spawned once an object is intersected. In this manner the most visible artifacts can be removed first and noise can be channeled into those areas of an image where it is least noticeable. The resulting image is produced in a format that is consistent with many image compression and transmission schemes.

title: A model of visual masking for computer graphics
abstract: In this paper we develop a computational model of visual masking based on psychophysical data. The model predicts how the presence of one visual pattern affects the detectability of another. The model allows us to choose texture patterns for computer graphics images that hide the effects of faceting, banding, aliasing, noise and other visual artifacts produced by sources of error in graphics algorithms. We demonstrate the utility of the model by choosing a texture pattern to mask faceting artifacts caused by polygonal tesselation of a flat-shaded curved surface. The model predicts how changes in the contrast, spatial frequency, and orientation of the texture pattern, or changes in the tesselation of the surface will alter the masking effect. The model is general and has uses in geometric modeling, realistic image synthesis, scientific visualization, image compression, and image-based rendering. CR Categories: I.3.0 [Computer Graphics]: General;

title: A perceptually based adaptive sampling algorithm
abstract: A perceptually based approach for selecting image samples has been developed. An existing image processing vision model has been extended to handle color and has been simplified to run efficiently. The resulting new image quality model was inserted into an image synthesis program by first modifying the rendering algorithm so that it computed a wavelet representation. In addition to allowing image quality to be determined as the image was generated, the wavelet representation made it possible to use statistical information about the spatial frequency distribution of natural images to estimate values where samples were yet to be taken. Tests on the image synthesis algorithm showed that it correctly handled achromatic and chromatic spatial detail and that it was able predict and compensate for masking effects. The program was also shown to produce images of equivalent visual quality while using different rendering techniques.

title: The Visible Differences Predictor: Applications to Global Illumination Problems
abstract: In this study of global illumination computations, we investigate the applications of the perceptually-based Visual Difference Predictor (VDP) developed by Daly . First, we validate the performance of this predictor in shadow masking by texture and luminance contrast experiments. We also experiment with Contrast Sensitivity Functions (CSFs) derived from the results of various psychophysical experiments, various spatial frequency and orientation channel decomposition schemes, and contrast definitions, in order to check predictor integrity and sensitivity to differing models of visual mechanisms. We show applications of the VDP to monitor the perceived quality of the progressive radiosity and Monte Carlo solutions, and decide upon their stopping conditions. Also, based on the local error metric provided by the predictor we show some initial attempts to drive adaptive mesh subdivision in radiosity computations.

title: A perceptually based physical error metric for realistic image synthesis
abstract: We introduce a new concept for accelerating realistic image synthesis algorithms. At the core of this procedure is a novel physical error metric that correctly predicts the perceptual threshold for detecting artifacts in scene features. Built into this metric is a computational model of the human visual system's loss of sensitivity at high background illumination levels, high spatial frequencies, and high contrast levels (visual masking). An important feature of our model is that it handles the luminance-dependent processing and spatiallydependent processing independently. This allows us to precompute the expensive spatially-dependent component, making our model extremely efficient. We illustrate the utility of our procedure with global illumination algorithms used for realistic image synthesis. The expense of global illumination computations is many orders of magnitude higher than the expense of direct illumination computations and can greatly benefit by applying our perceptually based technique. Results show our method preserves visual quality while achieving significant computational gains in areas of images with high frequency texture patterns, geometric details, and lighting variations.

title: Perceptually-Informed Accelerated Rendering of High Quality Walkthrough Sequences
abstract: In this paper, we consider accelerated rendering of walkthrough animation sequences using combination of ray tracing and Image-Based Rendering (IBR) techniques. Our goal is to derive as many pixels as possible using inexpensive IBR techniques without affecting the animation quality. A perception-based spatio-temporal Animation Quality Metric (AQM) is used to automatically guide such a hybrid rendering. The Pixel Flow (PF) obtained as a by-product of the IBR computation is an integral part of the AQM. The final animation quality is enhanced by an efficient spatio-temporal antialiasing, which utilize the PF to perform a motion-compensated filtering.

title: Perception-guided global illumination solution for animation rendering
abstract: We present a method for efficient global illumination computation in dynamic environments by taking advantage of temporal coherence of lighting distribution. The method is embedded in the framework of stochastic photon tracing and density estimation techniques. A locally operating energy-based error metric is used to prevent photon processing in the temporal domain for the scene regions in which lighting distribution changes rapidly. A perception-based error metric suitable for animation is used to keep noise inherent in stochastic methods below the sensitivity level of the human observer. As a result a perceptually-consistent quality across all animation frames is obtained. Furthermore, the computation cost is reduced compared to the traditional approaches operating solely in the spatial domain.

title: Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments
abstract: We present a method to accelerate global illumination computation in prerendered animations by taking advantage of limitations of the human visual system. A spatiotemporal error tolerance map, constructed from psychophysical data based on velocity dependent contrast sensitivity, is used to accelerate rendering. The error map is augmented by a model of visual attention in order to account for the tracking behavior of the eye. Perceptual acceleration combined with good sampling protocols provide a global illumination solution feasible for use in animation. Results indicate an order of magnitude improvement in computational speed.

title: Perceptually Driven Simplification Using Gaze-Directed Rendering
abstract: We present a unique polygonal simplification method grounded in rigorous perceptual science. Local simplification operations are driven directly by perceptual metrics, rather than the geometric metrics common to other algorithms. The effect of each operation on the final image is considered in terms of the contrast the operation will induce in the image and the spatial frequency of the resulting change. Equations derived from psychophysical studies determine whether the simplification operation will be perceptible; the operation is performed only if its effect is judged imperceptible. To increase the range of simplification permitted, we incorporate gaze-directed rendering. A commercial eye tracker monitors the direction of the user’s gaze, allowing the image to be simplified more aggressively in the periphery than at the center of vision. Our perceptual model addresses many interesting topics in polygonal simplification, including gaze-directed rendering, silhouette preservation, and imperceptible simplification. We describe two user studies to evaluate our model, and address the shortcomings as well as the potential of our approach.

title: Perceptually-driven decision theory for interactive realistic rendering
abstract: In this paper we introduce a new approach to realistic rendering at interactive rates on commodity graphics hardware. The approach uses efficient perceptual metrics within a decision theoretic framework to optimally order rendering operations, producing images of the highest visual quality within system constraints. We demonstrate the usefulness of this approach for various applications such as diffuse texture caching, environment map prioritization and radiosity mesh simplification. Although here we address the problem of realistic rendering at interactive rates, the perceptually-based decision theoretic methodology we introduce can be usefully applied in many areas of computer graphics.

title: Using Perceptual Texture Masking for Efficient Image Synthesis
abstract: Texture mapping has become indispensable in image synthesis as an inexpensive source of rich visual detail. Less obvious, but just as useful, is its ability to mask image errors due to inaccuracies in geometry or lighting. This ability can be used to substantially accelerate rendering by eliminating computations when the resulting errors will be perceptually insignificant.

title: Motion and vision. II. Stabilized spatio-temporal threshold surface.
abstract: The stabilized contrast-sensitivity function measured at a constant retinal velocity is tuned to a particular spatial frequency, which is inversely related to the velocity chosen. The Fourier transforms of these constant-velocity passbands have the same form as retinal receptive fields of various sizes. At low velocities, in the range of the natural drift motions of the eye, the stabilized contrast-sensitivity function matches the normal, unstablized result. At higher velocities (corresponding to motions of objects in the environment), this curve maintains the same shape but shifts toward lower spatial frequencies. The constant-velocity passband is displaced across the spatio-temporal frequency domain in a manner that is almost symmetric about the constant-velocity plane at v = 2 deg/s. Interpolating these diagonal profiles by a suitable analytic expression, we construct the spatio-temporal threshold surface for stabilized vision, and display its properties in terms of the usual frequency parameters; e.g., at low spatial frequencies, the temporal response becomes nearly independent of spatial frequency, while at low temporal frequencies, the spatial response becomes independent of temporal frequency.

title: The contrast sensitivity of human colour vision to red‐green and blue‐yellow chromatic gratings.
abstract: A method of producing red‐green and blue‐yellow sinusoidal chromatic gratings is used which permits the correction of all chromatic aberrations. A quantitative criterion is adopted to choose the intensity match of the two colours in the stimulus: this is the intensity ratio at which contrast sensitivity for the chromatic grating differs most from the contrast sensitivity for a monochromatic luminance grating. Results show that this intensity match varies with spatial frequency and does not necessarily correspond to a luminance match between the colours. Contrast sensitivities to the chromatic gratings at the criterion intensity match are measured as a function of spatial frequency, using field sizes ranging from 2 to 23 deg. Both blue‐yellow and red‐green contrast sensitivity functions have similar low‐pass characteristics, with no low‐frequency attenuation even at low frequencies below 0.1 cycles/deg. These functions indicate that the limiting acuities based on red‐green and blue‐yellow colour discriminations are similar at 11 or 12 cycles/deg. Comparisons between contrast sensitivity functions for the chromatic and monochromatic gratings are made at the same mean luminances. Results show that, at low spatial frequencies below 0.5 cycles/deg, contrast sensitivity is greater to the chromatic gratings, consisting of two monochromatic gratings added in antiphase, than to either monochromatic grating alone. Above 0.5 cycles/deg, contrast sensitivity is greater to monochromatic than to chromatic gratings.

title: A Model of Saliency-Based Visual Attention for Rapid Scene Analysis
abstract: A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail.

title: Detail to Attention: Exploiting Visual Tasks for Selective Rendering
abstract: The perceived quality of computer graphics imagery depends on the accuracy of the rendered frames, as well as the capabilities of the human visual system. Fully detailed, high fidelity frames still take many minutes even hours to render on today's computers. The human eye is physically incapable of capturing a moving scene in full detail. We sense image detail only in a 2° foveal region, relying on rapid eye movements, or saccades, to jump between points of interest. Our brain then reassembles these glimpses into a coherent, but inevitably imperfect, visual percept of the environment. In the process, we literally lose sight of the unimportant details. In this paper, we demonstrate how properties of the human visual system, in particular inattentional blindness, can be exploited to accelerate the rendering of animated sequences by applying a priori knowledge of a viewer's task focus. We show in a controlled experimental setting how human subjects will consistently fail to notice degradations in the quality of image details unrelated to their assigned task, even when these details fall under the viewers' gaze. We then build on these observations to create a perceptual rendering framework that combines predetermined task maps with spatiotemporal contrast sensitivity to guide a progressive animation system which takes full advantage of image-based rendering techniques. We demonstrate this framework with a Radiance ray-tracing implementation that completes its work in a fraction of the normally required time, with few noticeable artifacts for viewers performing the task.

title: The cortex transform: rapid computation of simulated neural images
abstract: This paper describes a transform which maps an image into a set of images that vary in resolution and orientation. Each pixel in the output may be regarded as the simulated response of a neuron in human visual cortex. The transform is invertible through simple expansion and addition operations. The transform is designed as a tool for image processing, machine vision, and modeling human vision.

title: A transducer function for threshold and suprathreshold human vision
abstract: A nonlinear function is derived to describe the contrast transduction process for human visual mechanisms. This function is sigmoid in form, having an accelerating nonlinearity at low contrasts and a compressive nonlinearity at high contrasts. The resulting formulation is consistent with both signal detection theory and with Quick's (1974) equation for probability summation. Similarities between the present description of human vision and properties of complex cells in cat visual cortex are noted.

title: Perceptual Metric for Production Testing
abstract: This paper describes a perceptually based image comparison processtliat can be used to tell when images are perceptually identical even though they contain imperceptible numerical differences. The technique has shown much utility in the production testing of rendering software.

title: Predicting visible differences in high dynamic range images: model and its calibration
abstract: New imaging and rendering systems commonly use physically accurate lighting information in the form of high-dynamic range (HDR) images and video. HDR images contain actual colorimetric or physical values, which can span 14 orders of magnitude, instead of 8-bit renderings, found in standard images. The additional precision and quality retained in HDR visual data is necessary to display images on advanced HDR display devices, capable of showing contrast of 50,000:1, as compared to the contrast of 700:1 for LCD displays. With the development of high-dynamic range visual techniques comes a need for an automatic visual quality assessment of the resulting images. In this paper we propose several modifications to the Visual Difference Predicator (VDP). The modifications improve the prediction of perceivable differences in the full visible range of luminance and under the adaptation conditions corresponding to real scene observation. The proposed metric takes into account the aspects of high contrast vision, like scattering of the light in the optics (OTF), nonlinear response to light for the full range of luminance, and local adaptation. To calibrate our HDR VDP we perform experiments using an advanced HDR display, capable of displaying the range of luminance that is close to that found in real scenes.

title: HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions
abstract: Visual metrics can play an important role in the evaluation of novel lighting, rendering, and imaging algorithms. Unfortunately, current metrics only work well for narrow intensity ranges, and do not correlate well with experimental data outside these ranges. To address these issues, we propose a visual metric for predicting visibility (discrimination) and quality (mean-opinion-score). The metric is based on a new visual model for all luminance conditions, which has been derived from new contrast sensitivity measurements. The model is calibrated and validated against several contrast discrimination data sets, and image quality databases (LIVE and TID2008). The visibility metric is shown to provide much improved predictions as compared to the original HDR-VDP and VDP metrics, especially for low luminance conditions. The image quality predictions are comparable to or better than for the MS-SSIM, which is considered one of the most successful quality metrics. The code of the proposed metric is available on-line.

title: TID2008 – A database for evaluation of full-reference visual quality assessment metrics
abstract: In this paper, a new image database, TID2008, for evaluation of full-reference visual quality assessment metrics is described. It contains 1700 test images (25 reference images, 17 types of distortions for each reference image, 4 different levels of each type of distortion). Mean Opinion Scores (MOS) for this database have been obtained as a result of more than 800 experiments. During these tests, observers from three countries (Finland, Italy, and Ukraine) have carried out about 256000 individual human quality judgments. The obtained MOS can be used for effective testing of different visual quality metrics as well as for the design of new metrics. Using the designed image database, we have tested several known quality metrics. The designed test image database is freely available for downloading and utilization in scientific investigations.

title: A no-reference metric for evaluating the quality of motion deblurring
abstract: Methods to undo the effects of motion blur are the subject of intense research, but evaluating and tuning these algorithms has traditionally required either user input or the availability of ground-truth images. We instead develop a metric for automatically predicting the perceptual quality of images produced by state-of-the-art deblurring algorithms. The metric is learned based on a massive user study, incorporates features that capture common deblurring artifacts, and does not require access to the original images (i.e., is "noreference"). We show that it better matches user-supplied rankings than previous approaches to measuring quality, and that in most cases it outperforms conventional full-reference image-similarity measures. We demonstrate applications of this metric to automatic selection of optimal algorithms and parameters, and to generation of fused images that combine multiple deblurring results.

title: Extending quality metrics to full luminance range images
abstract: Many quality metrics take as input gamma corrected images and assume that pixel code values are scaled perceptually uniform. Although this is a valid assumption for darker displays operating in the luminance range typical for CRT displays (from 0.1 to 80 cd/m2), it is no longer true for much brighter LCD displays (typically up to 500 cd/m2), plasma displays (small regions up to 1000 cd/m2) and HDR displays (up to 3000 cd/m2). The distortions that are barely visible on dark displays become clearly noticeable when shown on much brighter displays. To estimate quality of images shown on bright displays, we propose a straightforward extension to the popular quality metrics, such as PSNR and SSIM, that makes them capable of handling all luminance levels visible to the human eye without altering their results for typical CRT display luminance levels. Such extended quality metrics can be used to estimate quality of high dynamic range (HDR) images as well as account for display brightness.

title: Image quality assessment: from error visibility to structural similarity
abstract: Objective methods for assessing perceptual image quality traditionally attempted to quantify the visibility of errors (differences) between a distorted image and a reference image using a variety of known properties of the human visual system. Under the assumption that human visual perception is highly adapted for extracting structural information from a scene, we introduce an alternative complementary framework for quality assessment based on the degradation of structural information. As a specific example of this concept, we develop a structural similarity index and demonstrate its promise through a set of intuitive examples, as well as comparison to both subjective ratings and state-of-the-art objective methods on a database of images compressed with JPEG and JPEG2000. A MATLAB implementation of the proposed algorithm is available online at http://www.cns.nyu.edu//spl sim/lcv/ssim/.

title: Veiling glare: the dynamic range limit of HDR images
abstract: High Dynamic Range (HDR) images are superior to conventional images. However, veiling glare is a physical limit to HDR image acquisition and display. We performed camera calibration experiments using a single test target with 40 luminance patches covering a luminance range of 18,619:1. Veiling glare is a scene-dependent physical limit of the camera and the lens. Multiple exposures cannot accurately reconstruct scene luminances beyond the veiling glare limit. Human observer experiments, using the same targets, showed that image-dependent intraocular scatter changes identical display luminances into different retinal luminances. Vision's contrast mechanism further distorts any correlation of scene luminance and appearance. There must be reasons, other than accurate luminance, that explains the improvement in HDR images. The multiple exposure technique significantly improves digital quantization. The improved quantization allows displays to present better spatial information to humans. When human vision looks at high-dynamic range displays, it processes them using spatial comparisons.

title: Contrast constancy: deblurring in human vision by spatial frequency channels.
abstract: The perception of contrast was measured in humans by a technique of subjective contrast‐matching, and was compared with contrast sensitivity as defined by threshold measures. 2. Contrast‐matching between different spatial frequencies was performed correctly (especially at frequencies above 5 c/deg) despite the attenuation by optical and neural factors which cause large differences in contrast thresholds. 3. Contrast‐matching between single lines of different widths was also veridical, and was not limited by the spatial integration (Ricco's Law) present at threshold. Adaptation to gratings altered the appearance of lines, and this could be best understood in Fourier terms. 4. The generality of these results was shown by matching the contrast of pictures which had been filtered so that each contained a one octave band of spatial frequencies. 5. Within the limits imposed by threshold and resolution, contrast‐matching was largely independent of luminance and position on the retina. 6. Six out of eleven astigmatic observers showed considerable suprathreshold compensation for their orientation‐specific neural deficit in contrast sensitivity. 7. These results define a new property of vision: contrast constancy. It is argued that spatial frequency channels in the visual cortex are organized to compensate for earlier attenuation. This achieves a dramatic 'deblurring' of the image, and optimizes the clarity of vision.

title: A standard model for foveal detection of spatial contrast.
abstract: The ModelFest data set was created to provide a public source of data to test and calibrate models of foveal spatial contrast detection. It consists of contrast thresholds for 43 foveal achromatic contrast stimuli collected from each of 16 observers. We have fit these data with a variety of simple models that include one of several contrast sensitivity functions, an oblique effect, a spatial sensitivity aperture, spatial frequency channels, and nonlinear Minkowski summation. While we are able to identify one model, with particular parameters, as providing the lowest overall residual error, we also note that the differences among several good-fitting models are small. We find a strong reciprocity between the size of the spatial aperture and the value of the summation exponent: both are effective means of limiting the extent of spatial summation. The results demonstrate the power of simple models to account for the visibility of a wide variety of spatial stimuli and suggest that special mechanisms to deal with special classes of stimuli are not needed. But the results also illustrate the limited power of even this large data set to distinguish among similar competing models. We identify one model as a possible standard, suitable for simple theoretical and applied predictions.

title: Measurements of achromatic and chromatic contrast sensitivity functions for an extended range of adaptation luminance
abstract: Inspired by the ModelFest and ColorFest data sets, a contrast sensitivity function was measured for a wide range of adapting luminance levels. The measurements were motivated by the need to collect visual performance data for natural viewing of static images at a broad range of luminance levels, such as can be found in the case of high dynamic range displays. The detection of sine-gratings with Gaussian envelope was measured for achromatic color axis (black to white), two chromatic axes (green to red and yellow-green to violet) and two mixed chromatic and achromatic axes (dark-green to light-pink, and dark yellow to light-blue). The background luminance varied from 0.02 to 200 cd/m2. The spatial frequency of the gratings varied from 0.125 to 16 cycles per degree. More than four observers participated in the experiments and they individually determined the detection threshold for each stimulus using at least 20 trials of the QUEST method. As compared to the popular CSF models, we observed higher sensitivity drop for higher frequencies and significant differences in sensitivities in the luminance range between 0.02 and 2 cd/m2. Our measurements for chromatic CSF show a significant drop in sensitivity with luminance, but little change in the shape of the CSF. The drop of sensitivity at high frequencies is significantly weaker than reported in other studies and assumed in most chromatic CSF models.

title: Dynamic range independent image quality assessment
abstract: The diversity of display technologies and introduction of high dynamic range imagery introduces the necessity of comparing images of radically different dynamic ranges. Current quality assessment metrics are not suitable for this task, as they assume that both reference and test images have the same dynamic range. Image fidelity measures employed by a majority of current metrics, based on the difference of pixel intensity or contrast values between test and reference images, result in meaningless predictions if this assumption does not hold. We present a novel image quality metric capable of operating on an image pair where both images have arbitrary dynamic ranges. Our metric utilizes a model of the human visual system, and its central idea is a new definition of visible distortion based on the detection and classification of visible changes in the image structure. Our metric is carefully calibrated and its performance is validated through perceptual experiments. We demonstrate possible applications of our metric to the evaluation of direct and inverse tone mapping operators as well as the analysis of the image appearance on displays with various characteristics.

title: Video quality assessment for computer graphics applications
abstract: Numerous current Computer Graphics methods produce video sequences as their outcome. The merit of these methods is often judged by assessing the quality of a set of results through lengthy user studies. We present a full-reference video quality metric geared specifically towards the requirements of Computer Graphics applications as a faster computational alternative to subjective evaluation. Our metric can compare a video pair with arbitrary dynamic ranges, and comprises a human visual system model for a wide range of luminance levels, that predicts distortion visibility through models of luminance adaptation, spatiotemporal contrast sensitivity and visual masking. We present applications of the proposed metric to quality prediction of HDR video compression and temporal tone mapping, comparison of different rendering approaches and qualities, and assessing the impact of variable frame rate to perceived quality.

title: Objective Quality Assessment of Tone-Mapped Images
abstract: Tone-mapping operators (TMOs) that convert high dynamic range (HDR) to low dynamic range (LDR) images provide practically useful tools for the visualization of HDR images on standard LDR displays. Different TMOs create different tone-mapped images, and a natural question is which one has the best quality. Without an appropriate quality measure, different TMOs cannot be compared, and further improvement is directionless. Subjective rating may be a reliable evaluation method, but it is expensive and time consuming, and more importantly, is difficult to be embedded into optimization frameworks. Here we propose an objective quality assessment algorithm for tone-mapped images by combining: 1) a multiscale signal fidelity measure on the basis of a modified structural similarity index and 2) a naturalness measure on the basis of intensity statistics of natural images. Validations using independent subject-rated image databases show good correlations between subjective ranking score and the proposed tone-mapped image quality index (TMQI). Furthermore, we demonstrate the extended applications of TMQI using two examples - parameter tuning for TMOs and adaptive fusion of multiple tone-mapped images.

title: Evaluation of HDR tone mapping methods using essential perceptual attributes
abstract: The problem of reproducing high dynamic range images on media with restricted dynamic range has gained a lot of interest in the computer graphics community. There exist various approaches to this issue, which span several research areas including computer graphics, image processing, color vision, physiological aspects, etc. These approaches assume a thorough knowledge of both the objective and subjective attributes of an image. However, no comprehensive overview and analysis of such attributes has been published so far. In this contribution, we present an overview about the effects of basic image attributes in high dynamic range tone mapping. Furthermore, we propose a scheme of relationships between these attributes, leading to the definition of an overall image quality measure. We present results of subjective psychophysical experiments that we have performed to prove the proposed relationship scheme. Moreover, we also present an evaluation of existing tone mapping methods (operators) with regard to these attributes. Finally, the execution of with reference and without a real reference perceptual experiments gave us the opportunity to relate the obtained subjective results. Our effort is not just useful to get into the tone mapping field or when implementing a tone mapping method, but it also sets the stage for well-founded quality comparisons between tone mapping methods. By providing good definitions of the different attributes, user-driven or fully automatic comparisons are made possible.

title: Contrast Restoration by Adaptive Countershading
abstract: The ABSTRACT is to be in fully‐justified italicized text, between two horizontal lines, in one‐column format, below the author and affiliation information. Use the word “Abstract” as the title, in 9‐point Times, boldface type, left‐aligned to the text, initially capitalized. The abstract is to be in 9‐point, single‐spaced type. The abstract may be up to 3 inches (7.62 cm) long. Leave one blank line after the abstract, then add the subject categories according to the ACM Classification Index (see http://www.acm.org/class/1998/).

title: Unsharp Masking, Countershading and Halos: Enhancements or Artifacts?
abstract: Countershading is a common technique for local image contrast manipulations, and is widely used both in automatic settings, such as image sharpening and tonemapping, as well as under artistic control, such as in paintings and interactive image processing software. Unfortunately, countershading is a double‐edged sword: while correctly chosen parameters for a given viewing condition can significantly improve the image sharpness or trick the human visual system into perceiving a higher contrast than physically present in an image, wrong parameters, or different viewing conditions can result in objectionable halo artifacts. In this paper we investigate the perception of countershading in the context of a novel mask‐based contrast enhancement algorithm and analyze the circumstances under which the resulting profiles turn from image enhancement to artifact for a range of parameters and viewing conditions. Our experimental results can be modeled as a function of the width of the countershading profile. We employ this empirical function in a range of applications such as image resizing, view dependent tone mapping, and countershading analysis in photographs and works of fine art.

title: Perceptual models of viewpoint preference
abstract: The question of what are good views of a 3D object has been addressed by numerous researchers in perception, computer vision, and computer graphics. This has led to a large variety of measures for the goodness of views as well as some special-case viewpoint selection algorithms. In this article, we leverage the results of a large user study to optimize the parameters of a general model for viewpoint goodness, such that the fitted model can predict people's preferred views for a broad range of objects. Our model is represented as a combination of attributes known to be important for view selection, such as projected model area and silhouette length. Moreover, this framework can easily incorporate new attributes in the future, based on the data from our existing study. We demonstrate our combined goodness measure in a number of applications, such as automatically selecting a good set of representative views, optimizing camera orbits to pass through good views and avoid bad views, and trackball controls that gently guide the viewer towards better views.

title: Perceptual Metrics for Static and Dynamic Triangle Meshes
abstract: Almost all mesh processing procedures cause some more or less visible changes in the appearance of objects represented by polygonal meshes. In many cases, such as mesh watermarking, simplification or lossy compression, the objective is to make the change in appearance negligible, or as small as possible, given some other constraints. Measuring the amount of distortion requires taking into account the final purpose of the data. In many applications, the final consumer of the data is a human observer, and therefore the perceptibility of the introduced appearance change by a human observer should be the criterion that is taken into account when designing and configuring the processing algorithms. In this review, we discuss the existing comparison metrics for static and dynamic (animated) triangle meshes. We describe the concepts used in perception‐oriented metrics used for 2D image comparison, and we show how these concepts are employed in existing 3D mesh metrics. We describe the character of subjective data used for evaluation of mesh metrics and provide comparison results identifying the advantages and drawbacks of each method. Finally, we also discuss employing the perception‐correlated metrics in perception‐oriented mesh processing algorithms.

title: Decimation of triangle meshes
abstract: The polygon remains a popular graphics primitive for computer graphics application. Besides having a simple representation, computer rendering of polygons is widely supported by commercial graphics hardware and software. However, because the polygon is linear, often thousands or millions of primitives are required to capture the details of complex geometry. Models of this size are generally not practical since rendering speeds and memory requirements are proportional to the number of polygons. Consequently applications that generate large polygonal meshes often use domain-specific knowledge to reduce model size. There remain algorithms, however, where domainspecific reduction techniques are not generally available or appropriate. One algorithm that generates many polygons is marching cubes. Marching cubes is a brute force surface construction algorithm that extracts isodensity surfaces from volume data, producing from one to five triangles within voxels that contain the surface. Although originally developed for medical applications, marching cubes has found more frequent use in scientific visualization where the size of the volume data sets are much smaller than those found in medical applications. A large computational fluid dynamics volume could have a finite difference grid size of order 100 by 100 by 100, while a typical medical computed tomography or magnetic resonance scanner produces over 100 slices at a resolution of 256 by 256 or 512 by 512 pixels each. Industrial computed tomography, used for inspection and analysis, has even greater resolution, varying from 512 by 512 to 1024 by 1024 pixels. For these sampled data sets, isosurface extraction using marching cubes can produce from 500k to 2,000k triangles. Even today’s graphics workstations have trouble storing and rendering models of this size. Other sampling devices can produce large polygonal models: range cameras, digital elevation data, and satellite data. The sampling resolution of these devices is also improving, resulting in model sizes that rival those obtained from medical scanners. This paper describes an application independent algorithm that uses local operations on geometry and topology to reduce the number of triangles in a triangle mesh. Although our implementation is for the triangle mesh, it can be directly applied to the more general polygon mesh. After describing other work related to model creation from sampled data, we describe the triangle decimation process and its implementation. Results from two different geometric modeling applications illustrate the strengths of the algorithm.

title: Metro: Measuring Error on Simplified Surfaces
abstract: This paper presents a new tool, Metro, designed to compensate for a deficiency in many simplification methods proposed in literature. Metro allows one to compare the difference between a pair of surfaces (e.g. a triangulated mesh and its simplified representation) by adopting a surface sampling approach. It has been designed as a highly general tool, and it does no assumption on the particular approach used to build the simplified representation. It returns both numerical results (meshes areas and volumes, maximum and mean error, etc.) and visual results, by coloring the input surface according to the approximation error.

title: A Comparison of Perceptually-Based Metrics for Objective Evaluation of Geometry Processing
abstract: Recent advances in 3-D graphics technologies have led to an increasing use of processing techniques on 3-D meshes, such as filtering, compression, watermarking, simplification, deformation, and so forth. Since these processes may modify the visual appearance of the 3-D objects, several metrics have been introduced to properly drive or evaluate them, from classic geometric ones such as Hausdorff distance, to more complex perceptually-based measures. This paper presents a survey on existing perceptually-based metrics for visual impairment of 3-D objects and provides an extensive comparison between them. In particular, different scenarios which correspond to different perceptual and cognitive mechanisms are analyzed. The objective is twofold: 1) catching the behavior of existing measures to help Perception researchers for designing new 3-D metrics and 2) providing a comparison between them to inform and help computer graphics researchers for choosing the most accurate tool for the design and the evaluation of their mesh processing algorithms.

title: Dihedral Angle Mesh Error: a fast perception correlated distortion measure for fixed connectivity triangle meshes
abstract: In computer graphics, triangle meshes are ubiquitous as a representation of surface models. Processing of this kind of data, such as compression or watermarking, often involves an unwanted distortion of the surface geometry. Advanced processing algorithms are continuously being proposed, aiming at improving performance (compression ratio, watermark robustness and capacity), while minimizing the introduced distortion.

title: Spectral compression of mesh geometry
abstract: We show how spectral methods may be applied to 3D mesh data to obtain compact representations. This is achieved by projecting the mesh geometry onto an orthonormal basis derived from the mesh topology. To reduce complexity, the mesh is partitioned into a number of balanced submeshes with minimal interaction, each of which are compressed independently. Our methods may be used for compression and progressive transmission of 3D content, and are shown to be vastly superior to existing methods using spatial techniques, if slight loss can be tolerated.

title: High-Pass Quantization for Mesh Encoding
abstract: Any quantization introduces errors. An important question is how to suppress their visual effect. In this paper we present a new quantization method for the geometry of 3D meshes, which enables aggressive quantization without significant loss of visual quality. Conventionally, quantization is applied directly to the 3-space coordinates. This form of quantization introduces high-frequency errors into the model. Since high-frequency errors modify the appearance of the surface, they are highly noticeable, and commonly, this form of quantization must be done conservatively to preserve the precision of the coordinates. Our method first multiplies the coordinates by the Laplacian matrix of the mesh and quantizes the transformed coordinates which we call "δ-coordinates". We show that the high-frequency quantization errors in the δ-coordinates are transformed into low-frequency errors when the quantized δ-coordinates are transformed back into standard Cartesian coordinates. These low-frequency errors in the model are much less noticeable than the high-frequency errors. We call our strategy high-pass quantization, to emphasize the fact that it tends to concentrate the quantization error at the low-frequency end of the spectrum. To allow some control over the shape and magnitude of the low-frequency quantization errors, we extend the Laplacian matrix by adding a number of spatial constraints. This enables us to tailor the quantization process to specific visual requirements, and to strongly quantize the δ-coordinates.

title: Perceptually driven 3D distance metrics with application to watermarking
abstract: This paper presents an objective structural distortion measure which reflects the visual similarity between 3D meshes and thus can be used for quality assessment. The proposed tool is not linked to any specific application and thus can be used to evaluate any kinds of 3D mesh processing algorithms (simplification, compression, watermarking etc.). This measure follows the concept of structural similarity recently introduced for 2D image quality assessment by Wang et al.1 and is based on curvature analysis (mean, standard deviation, covariance) on local windows of the meshes. Evaluation and comparison with geometric metrics are done through a subjective experiment based on human evaluation of a set of distorted objects. A quantitative perceptual metric is also derived from the proposed structural distortion measure, for the specific case of watermarking quality assessment, and is compared with recent state of the art algorithms. Both visual and quantitative results demonstrate the robustness of our approach and its strong correlation with subjective ratings.

title: A Multiscale Metric for 3D Mesh Visual Quality Assessment
abstract: Many processing operations are nowadays applied on 3D meshes like compression, watermarking, remeshing and so forth; these processes are mostly driven and/or evaluated using simple distortion measures like the Hausdorff distance and the root mean square error, however these measures do not correlate with the human visual perception while the visual quality of the processed meshes is a crucial issue. In that context we introduce a full‐reference 3D mesh quality metric; this metric can compare two meshes with arbitrary connectivity or sampling density and produces a score that predicts the distortion visibility between them; a visual distortion map is also created. Our metric outperforms its counterparts from the state of the art, in term of correlation with mean opinion scores coming from subjective experiments on three existing databases. Additionally, we present an application of this new metric to the improvement of rate‐distortion evaluation of recent progressive compression algorithms.

title: A Curvature Tensor Distance for Mesh Visual Quality Assessment
abstract: This paper presents a new objective metric for assessing the visual difference between a reference or 'perfect' mesh and its distorted version. The proposed metric is based on the measurement of a distance between curvature tensors of the two triangle meshes under comparison. Unlike existing methods, our algorithm uses not only eigenvalues but also eigenvectors of the curvature tensor to derive a perceptually-oriented distance. Our metric also accounts for some important properties of the human visual system. Experimental results show good coherence between the proposed objective metric and subjective assessments.

title: Watermarked 3-D Mesh Quality Assessment
abstract: This paper addresses the problem of assessing distortions produced by watermarking 3D meshes. In particular, a new methodology for subjective evaluation of the quality of 3D objects is proposed and implemented. Two objective metrics derived from measures of surface roughness are then proposed and their efficiency to predict the perceptual impact of 3D watermarking is assessed and compared with the state of the art. Results obtained show good correlations between the proposed objective metrics and subjective assessments by human observers

title: Perceptual Quality Metrics for 3D Meshes: Towards an Optimal Multi-attribute Computational Model
abstract: 3D graphical data, commonly represented using triangular meshes, are deployed in a wide range of application processes including compression, filtering, watermarking, and simplification. These processes often introduce geometric distortions which affect the visual quality of the ultimate data visualization. In order to accurately evaluate perceptual impacts caused by the distortions, assessment metrics on 3D Mesh Visual Quality (MVQ) have been extensively discussed in the literature. Researchers recommended various metrics to predict the adverse effects that visual artifacts can have in applications. Most of these metrics are based on geometric attributes, conventional geometric distance, Laplacian coordinates, different types of curvature computation, and dihedral angles. We hypothesize that an optimal combination of multiple attributes associated with a 3D mesh surface can contribute to better perceptual prediction than single attributes used separately. In this paper, we use two user studies to validate our hypothesis. Our contributions are: (1) providing a detailed analysis of the most relevant geometric attributes for mesh quality assessment, and (2) introducing a new perceptual evaluation metric based on multiple attributes, with the optimal combination determined through machine learning techniques. Statistical quantitative analysis shows that our metric delivers better results than other state-of-the-art approaches. The proposed method is simple to implement and fast in execution. Moreover, our framework can easily be expanded to accommodate additional surface attributes.

title: FQM: a fast quality measure for efficient transmission of textured 3D models
abstract: In this paper, we propose an efficient transmission method to stream textured 3D models. We develop a bit-allocation algorithm that distributes the bit budget between the geometry and the mapped texture to maximize the quality of the model displayed on the client's screen. Both the geometry and the texture are progressively and independently compressed. The resolutions for the geometry and the texture are selected to maximize the quality for a given bitrate. We further propose a novel and fast quality measure (FQM) to quantify the perceptual fidelity of the simplified model. Experimental results demonstrate the effectiveness of the proposed bit-allocation algorithm using FQM. For example, when the bit budget is 10KB, the quality of the <sc>Zebra</sc> model is improved by 15% using the proposed method compared to distributing the bit budget equally between the geometry and the texture.

title: Quality metric for approximating subjective evaluation of 3-D objects
abstract: Many factors, such as the number of vertices and the resolution of texture, can affect the display quality of three-dimensional (3-D) objects. When the resources of a graphics system are not sufficient to render the ideal image, degradation is inevitable. It is, therefore, important to study how individual factors will affect the overall quality, and how the degradation can be controlled given limited resources. In this paper, the essential factors determining the display quality are reviewed. We then integrate two important ones, resolution of texture and resolution of wireframe, and use them in our model as a perceptual metric. We assess this metric using statistical data collected from a 3-D quality evaluation experiment. The statistical model and the methodology to assess the display quality metric are discussed. A preliminary study of the reliability of the estimates is also described. The contribution of this paper lies in: 1) determining the relative importance of wireframe versus texture resolution in perceptual quality evaluation and 2) proposing an experimental strategy for verifying and fitting a quantitative model that estimates 3-D perceptual quality. The proposed quantitative method is found to fit closely to subjective ratings by human observers based on preliminary experimental results.

title: Compression of soft-body animation sequences
abstract: We describe a compression scheme for the geometry component of 3D animation sequences. This scheme is based on the principle component analysis (PCA) method, which represents the animation sequence using a small number of basis functions. Second-order linear prediction coding (LPC) is applied to the PCA coefficients in order to further reduce the code size by exploiting the temporal coherence present in the sequence. Our results show that applying LPC to the PCA scheme results in significant performance improvements relative to other coding methods. The use of these codes will make animated 3D data more accessible for graphics and visualization applications.

title: A Perception Correlated Comparison Method for Dynamic Meshes
abstract: There are multiple areas of computer graphics where triangular meshes are being altered in order to reduce their size or complexity, while attempting to preserve the original shape of the mesh as closely as possible. Recently, this area of research has been extended to cover even a dynamic case, i.e., surface animations which are compressed and simplified. However, to date very little effort has been made to develop methods for evaluating the results, namely the amount of distortion introduced by the processing. Even the most sophisticated compression methods use distortion evaluation by some kind of mean squared error while the actual relevance of such measure has not been verified so far. In this paper, we point out some serious drawbacks of the existing error measures. We present results of the subjective testing that we have performed, and we derive a new measure called Spatiotemporal edge difference (STED) which is shown to provide much better correlation with subjective opinions on mesh distortion.

title: BTF‐CIELab: A Perceptual Difference Measure for Quality Assessment and Compression of BTFs
abstract: Driven by the advances in lossy compression of bidirectional texture functions (BTFs), there is a growing need for reliable methods to numerically measure the visual quality of the various compressed representations. Based on the CIE ΔE00 colour difference equation and concepts of its spatio‐temporal extension ST‐CIELab for video quality assessment, this paper presents a numerical quality measure for compressed BTF representations. By analysing the BTF in its full six‐dimensional (6D) space, light and view transition effects are integrated into the measure. In addition to the compressed representation, the method only requires the source BTF images as input and thus aids the objective evaluation of different compression techniques by means of a simple numerical comparison. By separating the spatial and angular components of the difference measure and linearizing each of them, the measure can be incorporated into any linear or multi‐linear compression technique. Using a per‐colour‐channel principal component analysis (PCA), compression rates of about 500:1 can be achieved at excellent visual quality.

title: Image-driven simplification
abstract: We introduce the notion of image-driven simplification, a framework that uses images to decide which portions of a model to simplify. This is a departure from approaches that make polygonal simplification decisions based on geometry. As with many methods, we use the edge collapse operator to make incremental changes to a model. Unique to our approach, however, is the use at comparisons between images of the original model against those of a simplified model to determine the cost of an ease collapse. We use common graphics rendering hardware to accelerate the creation of the required images. As expected, this method produces models that are close to the original model according to image differences. Perhaps more surprising, however, is that the method yields models that have high geometric fidelity as well. Our approach also solves the quandary of how to weight the geometric distance versus appearance properties such as normals, color, and texture. All of these trade-offs are balanced by the image metric. Benefits of this approach include high fidelity silhouettes, extreme simplification of hidden portions of a model, attention to shading interpolation effects, and simplification that is sensitive to the content of a texture. In order to better preserve the appearance of textured models, we introduce a novel technique for assigning texture coordinates to the new vertices of the mesh. This method is based on a geometric heuristic that can be integrated with any edge collapse algorithm to produce high quality textured surfaces.

title: Perceptually Guided Polygon Reduction
abstract: The properties of the human visual system are taken into account, along with the geometric aspects of an object, in a new surface remeshing algorithm and a new mesh simplification algorithm. Both algorithms have a preprocessing step and are followed by the remeshing or mesh simplification steps. The preprocessing step computes an importance map that indicates the visual masking potential of the visual patterns on the surface. The importance map is then used to guide the remeshing or mesh simplification algorithms. Two different methods are proposed for computing an importance map that indicates the masking potential of the visual patterns on the surface. The first one is based on the Sarnoff visual discrimination metric, and the second one is inspired by the visual masking tool available in the current JPEG2000 standard. Given an importance map, the surface remeshing algorithm automatically distributes few samples to surface regions with strong visual masking properties due to surface texturing, lighting variations, bump mapping, surface reflectance and inter-reflections. Similarly, the mesh simplification algorithm simplifies more aggressively where the light field of an object can hide more geometric artifacts.

title: SCROOGE:Perceptually‐Driven Polygon Reduction
abstract: Many real‐time 3D graphics renderers represent each object as a collection of simple polygons. The complexity of this polygon structure is of practical relevance because it can manifestly affect the performance of the graphics system. It is therefore common place to find techniques to reduce the polygonal complexity of a model with the ultimate aim of improving the interactivity of the application. In the past, many of these schemes have not been concerned with the perceptual side‐effects of this reduction and as a result a number of visual incongruities are often perceivable when these correspondingly‐reduced representations are employed. As an attempt to circumvent these problems, this paper presents a methodology for reducing the polygonal complexity of a model, whilst retaining a degree of perceptual predictability. This allows the visual consequences of the degradation to be quantified and accurately modelled.

title: Perceptually Driven Interactive Rendering
abstract: We present a framework for accelerating interactive rendering, grounded in psychophysical models of visual perception. This framework is applicable to rendering techniques that regulate resolution—and hence rendering load—using a hierarchy of local simplification operations. Our method drives those local operations directly by perceptual metrics: the effect of each simplification on the final image is considered in terms of the contrast the operation will induce in the image and the spatial frequency of the resulting change. A simple and conservative perceptual model determines under what conditions the simplification operation will be perceptible, enabling imperceptible simplification in which operations are performed only when judged imperceptible. Alternatively, simplifications may be ordered according to their perceptibility, providing a principled approach to best-effort rendering. Our approach addresses many interesting topics in the acceleration of interactive rendering, including imperceptible simplification, silhouette preservation, and gazedirected rendering. We demonstrate our framework applied to two quite different multiresolution rendering paradigms: viewdependent polygonal simplification and the QSplat point-based rendering system of Rusinkiewicz and Levoy [26].

title: Perceptually guided simplification of lit, textured meshes
abstract: We present a new algorithm for best-effort simplification of polygonal meshes based on principles of visual perception. Building on previous work, we use a simple model of low-level human vision to estimate the perceptibility of local simplification operations in a view-dependent Multi-Triangulation structure. Our algorithm improves on prior perceptual simplification approaches by accounting for textured models and dynamic lighting effects. We also model more accurately the scale of visual changes resulting from simplification, using parametric texture deviation to bound the size (represented as spatial frequency) of features destroyed, created, or altered by simplifying the mesh. The resulting algorithm displays many desirable properties: it is view-dependent, sensitive to silhouettes, sensitive to underlying texture content, and sensitive to illumination (for example, preserving detail near highlight and shadow boundaries, while aggressively simplifying washed-out regions). Using a unified perceptual model to evaluate these effects automatically accounts for their relative importance and balances between them, overcoming the need for ad hoc or hand-tuned heuristics.

title: Towards Perceptual Simplification of Models with Arbitrary Materials
abstract: Real‐time rendering of models with high polygon count is still an important issue in interactive computer graphics. A common way to improve rendering performance is to generate different levels of detail of a model. These are mostly computed using polygonal simplification techniques, which aim to reduce the number of polygons without significant loss of visual fidelity. Most existing algorithms use geometric error bounds, which are well‐suited for silhouette preservation. They ignore the fact that a much more aggressive simplification is possible in low‐contrast areas inside the model. The main contribution of this paper is an efficient simplification algorithm based on the human visual system. The key idea is to move the domain of error computation from image‐space to vertex‐space to avoid a costly per‐pixel comparison. This way the error estimation of a simplification operation can be accelerated significantly. To account for the human vision, we introduce a perceptually based metric depending on the contrast and spatial frequency of the model at a single vertex. Finally, we validate our approach with a user study.

title: Efficient method for paired comparison
abstract: This paper describes a more efficient paired comparison method that reduces the number of trials necessary for converting a table of paired comparisons into scaler data. Instead of comparing every pair of samples (the complete method), a partial method is used that makes more comparisons between closer samples than between more distant samples. A sorting algorithm is used to effi- ciently order the samples with paired comparisons, and each com- parison is recorded. When the sorting is completed, more trials will have been conducted between closer samples than between distant samples. Regression is used to scale the resulting comparison ma- trix into a one dimensional perceptual quality estimate.

title: New measurements reveal weaknesses of image quality metrics in evaluating graphics artifacts
abstract: Reliable detection of global illumination and rendering artifacts in the form of localized distortion maps is important for many graphics applications. Although many quality metrics have been developed for this task, they are often tuned for compression/transmission artifacts and have not been evaluated in the context of synthetic CG-images. In this work, we run two experiments where observers use a brush-painting interface to directly mark image regions with noticeable/objectionable distortions in the presence/absence of a high-quality reference image, respectively. The collected data shows a relatively high correlation between the with-reference and no-reference observer markings. Also, our demanding per-pixel image-quality datasets reveal weaknesses of both simple (PSNR, MSE, sCIE-Lab) and advanced (SSIM, MS-SSIM, HDR-VDP-2) quality metrics. The most problematic are excessive sensitivity to brightness and contrast changes, the calibration for near visibility-threshold distortions, lack of discrimination between plausible/implausible illumination, and poor spatial localization of distortions for multi-scale metrics. We believe that our datasets have further potential in improving existing quality metrics, but also in analyzing the saliency of rendering distortions, and investigating visual equivalence given our with- and no-reference data.

title: Measuring and predicting visual fidelity
abstract: This paper is a study of techniques for measuring and predicting visual fidelity. As visual stimuli we use polygonal models, and vary their fidelity with two different model simplification algorithms. We also group the stimuli into two object types: animals and man made artifacts. We examine three different experimental techniques for measuring these fidelity changes: naming times, ratings, and preferences. All the measures were sensitive to the type of simplification and level of simplification. However, the measures differed from one another in their response to object type. We also examine several automatic techniques for predicting these experimental measures, including techniques based on images and on the models themselves. Automatic measures of fidelity were successful at predicting experimental ratings, less successful at predicting preferences, and largely failures at predicting naming times. We conclude with suggestions for use and improvement of the experimental and automatic measures of visual fidelity.

title: Evaluation of Tone Mapping Operators for HDR-Video
abstract: Eleven tone-mapping operators intended for video processing are analyzed and evaluated with camera-captured and computer-generated high-dynamic-range content. After optimizing the parameters of the operators in a formal experiment, we inspect and rate the artifacts (flickering, ghosting, temporal color consistency) and color rendition problems (brightness, contrast and color saturation) they produce. This allows us to identify major problems and challenges that video tone-mapping needs to address. Then, we compare the tone-mapping results in a pair-wise comparison experiment to identify the operators that, on average, can be expected to perform better than the others and to assess the magnitude of differences between the best performing operators.

title: Perceptual evaluation of tone mapping operators with real-world scenes
abstract: A number of successful tone mapping operators for contrast compression have been proposed due to the need to visualize high dynamic range (HDR) images on low dynamic range devices. They were inspired by fields as diverse as image processing, photographic practice, and modeling of the human visual systems (HVS). The variety of approaches calls for a systematic perceptual evaluation of their performance. We conduct a psychophysical experiment based on a direct comparison between the appearance of real-world scenes and HDR images of these scenes displayed on a low dynamic range monitor. In our experiment, HDR images are tone mapped by seven existing tone mapping operators. The primary interest of this psychophysical experiment is to assess the differences in how tone mapped images are perceived by human observers and to find out which attributes of image appearance account for these differences when tone mapped images are compared directly with their corresponding real-world scenes rather than with each other. The human subjects rate image naturalness, overall contrast, overall brightness, and detail reproduction in dark and bright image regions with respect to the corresponding real-world scene. The results indicate substantial differences in perception of images produced by individual tone mapping operators. We observe a clear distinction between global and local operators in favor of the latter, and we classify the tone mapping operators according to naturalness and appearance attributes.

title: A reality check for tone-mapping operators
abstract: A large number of high-quality tone-mapping operators is currently available. In addition to inherent practical value, comparing their performance is necessary to further advance the field and can provide better understanding of visual realism. In this paper, we show that it becomes very difficult to meaningfully judge relative performance of modern tone-mapping techniques with existing comparison methods and demonstrate that using real environments is crucial in such experiments. We report results of a new study comparing five recent tone-mapping techniques using this approach.

title: Analysis of Reproducing Real‐World Appearance on Displays of Varying Dynamic Range
abstract: We conduct a series of experiments to investigate the desired properties of a tone mapping operator (TMO) and to design such an operator based on subjective data. We propose a novel approach to the tone mapping problem, in which the tone mapping parameters are determined based on the data from subjective experiments, rather than an image processing algorithm or a visual model. To collect this data, a series of experiments are conducted in which the subjects adjust three generic TMO parameters: brightness, contrast and color saturation. In two experiments, the subjects are to find a) the most preferred image without a reference image (preference task) and b) the closest image to the real‐world scene which the subjects are confronted with (fidelity task). We analyze subjects’ choice of parameters to provide more intuitive control over the parameters of a tone mapping operator. Unlike most of the researched TMOs that focus on rendering for standard low dynamic range monitors, we consider a broad range of potential displays, each offering different dynamic range and brightness. We simulate capabilities of such displays on a high dynamic range (HDR) display. This allows us to address the question of how tone mapping needs to be adjusted to accommodate displays with drastically different dynamic ranges.

title: Psychovisual Assessment of Tone-Mapping Operators for Global Appearance and Colour Reproduction
abstract: To identify people’s preference, psychovisual tests are carried out in virtual environment. Images of scenes used for psychovisual tests are natively High Dynamic Range (HDR) images. However, in order to allow a Low Dynamic Range (LDR) display device to project HDR pictures, their dynamic range must be compressed with a tone-mapping operator (TMO). Thus, before tests are carried out, selection of the most suitable TMO, perceptually speaking, is required to present images faithfully, depending on the aim of the application. In this paper, three different experimental protocols are proposed for assessing the applications of a display device used for psychovisual tests and selecting the most suitable TMO from six candidates. An additional goal is to give a first idea of the applications that can be addressed by this device. Two subjective experiments were conducted and are presented in the paper. The first one is divided in two steps. The results of three different protocols (two protocols in the first test and the third in the second test) for the identification of the preferred TMO for five different scenes are compared and LDR image defects noticed by observers are highlighted in the paper.

title: Evaluation of tone mapping operators using a High Dynamic Range display
abstract: Tone mapping operators are designed to reproduce visibility and the overall impression of brightness, contrast and color of the real world onto limited dynamic range displays and printers. Although many tone mapping operators have been published in recent years, no thorough psychophysical experiments have yet been undertaken to compare such operators against the real scenes they are purporting to depict. In this paper, we present the results of a series of psychophysical experiments to validate six frequently used tone mapping operators against linearly mapped High Dynamic Range (HDR) scenes displayed on a novel HDR device. Individual operators address the tone mapping issue using a variety of approaches and the goals of these techniques are often quite different from one another. Therefore, the purpose of this investigation was not simply to determine which is the "best" algorithm, but more generally to propose an experimental methodology to validate such operators and to determine the participants' impressions of the images produced compared to what is visible on a high contrast ratio display.

title: Perceptual image quality: Effects of tone characteristics
abstract: Tone mapping refers to the conversion of luminance values recorded by a digital camera or other acquisition device, to the luminance levels available from an output device, such as a monitor or a printer. Tone mapping can improve the appearance of rendered images. Although there are a variety of algorithms available, there is little information about the image tone characteristics that produce pleasing images. We devised an experiment where preferences for images with different tone characteristics were measured. The results indicate that there is a systematic relation between image tone characteristics and perceptual image quality for images containing faces. For these images, a mean face luminance level of 46-49 CIELAB L* units and a luminance standard deviation (taken over the whole image) of 18 CIELAB L* units produced the best renderings. This information is relevant for the design of tone-mapping algorithms, particularly as many images taken by digital camera users include faces.

title: Do HDR displays support LDR content?: a psychophysical evaluation
abstract: The development of high dynamic range (HDR) imagery has brought us to the verge of arguably the largest change in image display technologies since the transition from black-and-white to color television. Novel capture and display hardware will soon enable consumers to enjoy the HDR experience in their own homes. The question remains, however, of what to do with existing images and movies, which are intrinsically low dynamic range (LDR). Can this enormous volume of legacy content also be displayed effectively on HDR displays? We have carried out a series of rigorous psychophysical investigations to determine how LDR images are best displayed on a state-of-the-art HDR monitor, and to identify which stages of the HDR imaging pipeline are perceptually most critical. Our main findings are: (1) As expected, HDR displays outperform LDR ones. (2) Surprisingly, HDR images that are tone-mapped for display on standard monitors are often no better than the best single LDR exposure from a bracketed sequence. (3) Most importantly of all, LDR data does not necessarily require sophisticated treatment to produce a compelling HDR experience. Simply boosting the range of an LDR image linearly to fit the HDR display can equal or even surpass the appearance of a true HDR image. Thus the potentially tricky process of inverse tone mapping can be largely circumvented.

title: Assessment of video tone-mapping: Are cameras' S-shaped tone-curves good enough?
abstract: The performance of video tone-mapping operators is investigated in a rating experiment using two criteria: overall quality and fidelity to real-world experience. The study includes a tone-curve used in commercial cameras, rarely considered in tone-mapping evaluation studies. The quality is measured for a range of parameter settings, revealing the importance of parameter fine-tuning and often unsatisfactory results of the default operator parameters. In order to explain what makes best performing operators better, the results are analysed in relation to image statistics and the characteristics of the tone-mapping function. Our observations are: state-of-the-art tone mapping produces measurably better results than camera's S-shaped curve for high dynamic range scenes with important content spanned across a wide dynamic range; differences in colour reproduction strongly affect the results; fidelity and quality criteria produce similar results when no reference is present; and state-of-the-art operators produce the results of comparable quality when their parameters are well selected.

title: A Psychophysical Evaluation of Inverse Tone Mapping Techniques
abstract: In recent years inverse tone mapping techniques have been proposed for enhancing low‐dynamic range (LDR) content for a high‐dynamic range (HDR) experience on HDR displays, and for image based lighting. In this paper, we present a psychophysical study to evaluate the performance of inverse (reverse) tone mapping algorithms. Some of these techniques are computationally expensive because they need to resolve quantization problems that can occur when expanding an LDR image. Even if they can be implemented efficiently on hardware, the computational cost can still be high. An alternative is to utilize less complex operators; although these may suffer in terms of accuracy. Our study investigates, firstly, if a high level of complexity is needed for inverse tone mapping and, secondly, if a correlation exists between image content and quality. Two main applications have been considered: visualization on an HDR monitor and image‐based lighting.

title: Evaluation of reverse tone mapping through varying exposure conditions
abstract: Most existing image content has low dynamic range (LDR), which necessitates effective methods to display such legacy content on high dynamic range (HDR) devices. Reverse tone mapping operators (rTMOs) aim to take LDR content as input and adjust the contrast intelligently to yield output that recreates the HDR experience. In this paper we show that current rTMO approaches fall short when the input image is not exposed properly. More specifically, we report a series of perceptual experiments using a Brightside HDR display and show that, while existing rTMOs perform well for under-exposed input data, the perceived quality degrades substantially with over-exposure, to the extent that in some cases subjects prefer the LDR originals to images that have been treated with rTMOs. We show that, in these cases, a simple rTMO based on gamma expansion avoids the errors introduced by other methods, and propose a method to automatically set a suitable gamma value for each image, based on the image key and empirical data. We validate the results both by means of perceptual experiments and using a recent image quality metric, and show that this approach enhances visible details without causing artifacts in incorrectly-exposed regions. Additionally, we perform another set of experiments which suggest that spatial artifacts introduced by rTMOs are more disturbing than inaccuracies in the expanded intensities. Together, these findings suggest that when the quality of the input data is unknown, reverse tone mapping should be handled with simple, non-aggressive methods to achieve the desired effect.

title: A comparative study of image retargeting
abstract: The numerous works on media retargeting call for a methodological approach for evaluating retargeting results. We present the first comprehensive perceptual study and analysis of image retargeting. First, we create a benchmark of images and conduct a large scale user study to compare a representative number of state-of-the-art retargeting methods. Second, we present analysis of the users' responses, where we find that humans in general agree on the evaluation of the results and show that some retargeting methods are consistently more favorable than others. Third, we examine whether computational image distance metrics can predict human retargeting perception. We show that current measures used in this context are not necessarily consistent with human rankings, and demonstrate that better results can be achieved using image features that were not previously considered for this task. We also reveal specific qualities in retargeted media that are more important for viewers. The importance of our work lies in promoting better measures to assess and guide retargeting algorithms in the future. The full benchmark we collected, including all images, retargeted results, and the collected user data, are available to the research community for further investigation at http://people.csail.mit.edu/mrub/retargetme.

title: Comparison of Deghosting Algorithms for Multi-exposure High Dynamic Range Imaging
abstract: High dynamic range (HDR) images can be generated by capturing a sequence of low dynamic range (LDR) images of the same scene with different exposures and then merging those images to create an HDR image. During capturing of LDR images, any changes in the scene or slightest camera movement results in ghost artifacts in the resultant HDR image. Over the past few years many algorithms have been proposed to produce ghost free HDR images of dynamic scenes. In this study we performed subjective psychophysical experiments to evaluate four algorithms for removing ghost artifacts in the final HDR image. To our best knowledge, no evaluation of deghosting algorithms for HDR imaging has been published. Thus, the aim of this paper is not only to evaluate different ghost removal algorithms but also to introduce a methodology to evaluate such algorithms and to present some of the challenges that exist in evaluating ghost removal algorithms in HDR images. Optical flow algorithms have been shown to produce successful results in aligning input images before merging them into an HDR image. As a result one of the state-of-the-art deghosting algorithm for HDR image alignment is based on optical flow. To test the limits of the evaluated deghosting algorithms the scenes used in our experiments were selected following the criteria proposed by Baker et al. [2011], which is considered as de facto standard for evaluating optical flow methodologies. The scenes used in the experiments serve to provide challenges that need to be dealt with by not only algorithms based on optical flow methodologies but also by other ghost removal algorithms for HDR imaging. The results reveal the scenes for which the evaluated algorithms fail and may serve as a guide for future research in this area.

title: Perceptual evaluation of tone-reproduction operators using the Cornsweet--Craik--O'Brien illusion
abstract: High dynamic-range images cannot be directly displayed on conventional display devices, but have to be tone-mapped first. For this purpose, a large set of tone-reproduction operators is currently available. However, it is unclear which operator is most suitable for any given task. In addition, different tasks may place different requirements upon each operator. In this paper we evaluate several tone-reproduction operators using a paradigm that does not require the construction of a real high dynamic-range scene, nor does it require the availability of a high dynamic-range display device. The user study involves a task that relates to the evaluation of contrast, which is an important attribute that needs to be preserved under tone reproduction.

title: Influence of Context and Content on Tone-mapping Operators
abstract: High Dynamic Range (HDR) imaging is one of the technologies that are changing photo, TV, and film industries. The popularity and full public adoption of HDR content is however hindered by the lack of standards in evaluation of quality, file formats, and compression, as well as large legacy base of Low Dynamic Range (LDR) displays that are unable to render HDR. Many tone-mapping operators, aiming to generating viewable LDR content from HDR, were developed to resolve the legacy hardware and software problem. However, there is no consensus on which tone-mapping to use and under which conditions. This paper, via a series of subjective evaluations, demonstrates the dependency of the perceptual quality of the tone-mapped LDR images on the context, such as environmental factors and display parameters, and image content. The results of subjective tests indicate a significant influence of context and content on performance of tone-mapping, and, therefore, both should be considered in applications generating LDR from HDR.

title: Temporal Coherency in Video Tone Mapping, A Survey
abstract: High Dynamic Range (HDR) video acquisition and display is now achievable thanks to recent advances 
in the HDR field. Tone Mapping Operators (TMOs) convert HDR images to Low Dynamic Range 
(LDR) ones. However, these operators do not take into account the temporal correlation present in 
a video sequence. We first present in this article, two main issues that arise when dealing with video 
tone mapping: flickering and temporal brightness incoherency. Flickering corresponds to rapid changes 
of either the tone map curve or the scene content. As for brightness incoherency, it occurs when tone 
mapping frames of a video sequence separately. We then propose a survey of the techniques that solve 
these issues. Results show that while some issues have been well studied and solved, others still need 
further work.

title: Perceptual issues in substituting texture for geometry
abstract: An important goal in interactive computer graphics is to allow the user to interact dynamically with three-dimensional objects. The computing resources required to represent, transmit and display a three dimensional object depends on the number of polygons used to represent it. Many geometric simplification algorithms have been developed to represent the geometry with as few polygons as possible, without substantially changing the appearance of the rendered object. A popular method for achieving geometric simplification is to replace fine scale geometric detail with texture images mapped onto the simplified geometry. However the effectiveness of replacing geometry with texture has not been explored experimentally. In this paper we describe a visual experiment in which we examine the perceived quality of various representations of textured, geometric objects, viewed under direct and oblique illumination. We used a pair of simple large scale objects with different fine-scale geometric detail. For each object we generated many representations, varying the resources allocated to geometry and texture. The experimental results show that while replacing geometry with texture can be very effective, in some cases the addition of texture does not improve perceived quality, and can sometimes reduce the perceived quality.

title: Evaluation of supra-threshold perceptual metrics for 3D models
abstract: Measures of dissimilarity of 3D models are necessary in a wide range of applications such as geometry compression, simplification, and 3D model retrieval. In many cases a metric that models perceptual dissimilarity is desirable. Recently, metrics for 3D models have been evaluated in that respect using concepts such as just noticeable differences, rankings, and others. We propose a simple experimental setup for evaluating supra-threshold perception of 3D models in which users select models at equal perceptual distance to given pairs of models. We discuss the advantages of our approach and report the results of a field study comparing six objective distance measures applied to palettes of simplified reference models. We found that the objective measures are biased, and generally image-based metrics perform better than metrics based on the original 3D geometry.

title: A Perceptual Data Repository for Polygonal Meshes
abstract: A repository containing perceived quality data for polygonal meshes, obtained through observer studies, is presented. It includes information regarding the experimental methodology, protocol and models used, with the purpose of allowing researchers to use it, e.g., for a faster preliminary assessment of their perceived quality metrics without the overhead of designing and performing an observer study.

title: A local roughness measure for 3D meshes and its application to visual masking
abstract: 3D models are subject to a wide variety of processing operations such as compression, simplification or watermarking, which may introduce some geometric artifacts on the shape. The main issue is to maximize the compression/simplification ratio or the watermark strength while minimizing these visual degradations. However few algorithms exploit the human visual system to hide these degradations, while perceptual attributes could be quite relevant for this task. Particularly, the masking effect defines the fact that one visual pattern can hide the visibility of another. In this context we introduce an algorithm for estimating the roughness of a 3D mesh, as a local measure of geometric noise on the surface. Indeed, a textured (or rough) region is able to hide geometric distortions much better than a smooth one. Our measure is based on curvature analysis on local windows of the mesh and is independent of the resolution/connectivity of the object. The accuracy and the robustness of our measure, together with its relevance regarding visual masking have been demonstrated through extensive comparisons with state-of-the-art and subjective experiment. Two applications are also presented, in which the roughness is used to lead (and improve) respectively compression and watermarking algorithms.

title: A spatial extension of CIELAB for digital color‐image reproduction
abstract: We describe a spatial extension to the CIELAB color metric that is useful for measuring color reproduction errors of digital images. To compute the error, digital color images are spatially filtered using a pattern-color separable method and then converted into the CIELAB representation. Over patterned regions of the image, the reproduction errors measured using the spatial extension of CIELAB correspond to perceived color errors better than errors computed without the spatial extension. Over uniform spatial regions of the image, errors computed with the extension are equal to errors computed using the standard CIELAB formulae.

title: On the Efficiency of Image Metrics for Evaluating the Visual Quality of 3D Models
abstract: 3D meshes are deployed in a wide range of application processes (e.g., transmission, compression, simplification, watermarking and so on) which inevitably introduce geometric distortions that may alter the visual quality of the rendered data. Hence, efficient model-based perceptual metrics, operating on the geometry of the meshes being compared, have been recently introduced to control and predict these visual artifacts. However, since the 3D models are ultimately visualized on 2D screens, it seems legitimate to use images of the models (i.e., snapshots from different viewpoints) to evaluate their visual fidelity. In this work we investigate the use of image metrics to assess the visual quality of 3D models. For this goal, we conduct a wide-ranging study involving several 2D metrics, rendering algorithms, lighting conditions and pooling algorithms, as well as several mean opinion score databases. The collected data allow (1) to determine the best set of parameters to use for this image-based quality assessment approach and (2) to compare this approach to the best performing model-based metrics and determine for which use-case they are respectively adapted. We conclude by exploring several applications that illustrate the benefits of image-based quality assessment.