Full‐reference tone‐mapped images quality assessment

IET Image ProcessingVolume 15, Issue 3 p. 763-773 ORIGINAL RESEARCH PAPEROpen Access Full-reference tone-mapped images quality assessment Mohammad Reza Faraji, Corresponding Author Mohammad Reza Faraji M.Faraji@iasbs.ac.ir Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences, Zanjan, Iran Correspondence Mohammad Reza Faraji, Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences, Zanjan, Iran. Email: M.Faraji@iasbs.ac.irSearch for more papers by this author Mohammad Reza Faraji, Corresponding Author Mohammad Reza Faraji M.Faraji@iasbs.ac.ir Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences, Zanjan, Iran Correspondence Mohammad Reza Faraji, Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences, Zanjan, Iran. Email: M.Faraji@iasbs.ac.irSearch for more papers by this author First published: 26 December 2020 https://doi.org/10.1049/ipr2.12060AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Various tone mapping operators have been proposed to convert the high dynamic range images to low dynamic ranges to improve visualization on low dynamic range displays. This paper presents a full-reference objective quality assessment index to evaluate the perceived quality of tone-mapped images. The proposed method seamlessly employs the multi-scale structural fidelity, statistical naturalness, colourfulness and the multi-scale free energy of the image to create a similarity score between a produced low dynamic range image and its high dynamic range image. The extensive experiments on three publicly available datasets using Spearman's rank-order correlation coefficient, Kendall's rank-order correlation coefficient and receiver operating characteristics analyses indicate the proposed tone-mapped quality index is superior to recently proposed state-of-the-art objective quality indices. 1 INTRODUCTION Traditional imaging methods, known as low dynamic range (LDR) imagery, only handle 8 bits or less per colour channel per pixel, whereas high dynamic range (HDR) imagery records very dark and bright areas of a scene at the same time while avoiding under-exposed and over-exposed areas [1]. Most display devices are not able to display HDR content. As a result, in the last two decades, various researchers have spent significant time and effort and employed the so-called tone mapping operations to compress the HDR images and videos in a way that the results are visualized, more naturally, on LDR displays. A typical tone mapping operator (TMO) keeps characteristics such as local and global contrast and details of the HDR content to some extent and, mathematically, is defined as follows [1]: T ( I ) : R i w × h × c → D o w × h × c (1)where T is the TMO operator, I is the image, w and h are the width and height of the image, c is the number of colour bands in I (for RGB colour space, c = 3 ), R i ⊂ R and D o ⊂ R i (for normal LDR monitors, D o =[0, 255]). TMOs can be categorized into global, local, frequency/gradient and segmentation operators [1]. Global operators perform the same operation to all pixels. Local operators, for each pixel, perform the operation on a neighbourhood of the pixel. Frequency operators first separate image's low and high frequencies and then apply the operator on low frequencies while keeping the high frequency to preserve the fine details. Segmentation operators segment the image in broad regions and then apply a different mapping to each region [1]. Some representative examples include adaptive logarithmic [2], contrast-based scale factor [3] and quantization techniques [4] as global operators; photographic tone reproduction [5] and TMO for high contrast images [6] as local operator; low curvature image simplifiers [7], trilateral filtering [8] and image colour appearance model (iCAM06) [9] as frequency operator; and interactive manipulation (IM) [10] and lightness perception [11] as segmentation operator. Reducing the dynamic range of the image would inevitably result in losing some information. Therefore, a TMO aims at producing a natural-looking realistic LDR image by preserving the structural information of a given HDR image [12]. TMOs can act differently on different HDR images, which means not every TMO performs well for any given HDR image [13]. Similar to image quality assessments (IQA) in other areas, TMO methods and their effectiveness on producing high-quality LDR images can be evaluated using two different approaches: traditional subjective IQA (SIQA) and objective IQA (OIQA). In SIQA, which relies on human subjective evaluations, a number of subjects are asked to rate LDR images produced using different TMOs based on their overall quality, brightness, contrast, naturalness, detail reproduction and colour [12]. The mean opinion score (MOS) obtained from SIQA is usually regarded as the best approach for IQA [14]. However, they are expensive, time-consuming and difficult to employ for improving TMOs, and some subjects may miss some valuable structures contained in HDR images [12]. OIQA designs computational models to predict the perceived image quality accurately and automatically [14]. Although various OIQA methods are proposed in different areas of IQA, only a few full-reference assessment methods are developed for HDR images. Mantiuk et al. [15] proposed to use HDR visible difference predictor (HDR-VDP) to distinguish between visible and invisible distortions. HDR-VDP is not specifically designed to compare HDR images with their LDRs as it only predicts the visibility of differences between two HDR images of the same dynamic range. Aydin et al. [16] further improved HDR-VDP to be dynamic independent, but their improved method does not result in a single quality score for a given image, which makes it almost impossible to be validated with subjective evaluations of overall image quality [12]. One of the first OIQA methods, specifically designed for evaluating TMOs, is the tone-mapped quality index (TMQI) [12]. Inspired by structural similarity (SSIM) approach [17], its multi-scale derivations [18, 19] and the natural scene statistics (NSS) approach [20], TMQI combines a multi-scale structural fidelity measure and a statistical naturalness measure to produce an overall quality score [12]. Ma et al. [21] later investigate each component of TMQI (i.e. the structural fidelity and statistical naturalness components) and improved each term separately. The improved index is known as TMQI-II. While both TMQI and TMQI-II yield a decent quality score compared with the subjective score of a tone-mapped image, they do not consider the colourfulness of the image which indicates the image's colour vividness and the preserved colour information after the tone-mapping operation. Feature similarity index for tone-mapped images (FSITM) [13] is another OIQA metric which compares the locally weighted mean phase angle map of the HDR image with its converted LDR image. The phase angle in FSITM can be calculated using each of the three red (FSITMR), green (FSITMG) and blue (FSITMB) channels. The final FSITM score, obtained by any of the three channels, can be combined with the TMQI score to further improve the performance of each of the two methods. FSITM effectively uses the gradient domain to capture the image quality, but similar to TMQI and TMQI-II, it does not also consider the colour information of the tone-mapped image. In addition, in real application, one challenge of using FSITM is to decide which colour channel should be employed as different channels (i.e. R, G or B) yield a different quality score. This paper, inspired by TMQI, presents a new quality assessment metric for tone-mapped images, which combines a multi-scale structural fidelity, a statistical naturalness, colourfulness and image's free energy measures to produce a quality index. The quality index is called SNCF which is the acronym of the following four components: Sstructural fidelity, naturalness, colourfulness and free energy). The proposed index first computes the structural fidelity of the image in different scales to produce the multi-scale structural fidelity measure which is weighted based on the psychophysical experiment results. Then, it measures statistical naturalness and colourfulness of the image and combines them together to produce the colour quality measure. Next, SNCF employs the free energy principle to model the perception and understanding of an image by brain to produce a psychovisual quality metric. Finally, these metrics are combined together to produce the final tone-mapped image quality SNCF score. The rest of the paper is organized as follows. Section 2 presents the methodology, including the four main features of the proposed measure. Section 3 shows the experimental results on the three publicly available databases and compares the proposed SNCF index with some of the recently proposed state-of-the-art methods. Finally, Section 4 draws the conclusion. 2 METHODOLOGY TMOs are not able to preserve all structural information since they reduce the dynamic range of HDR images [22]. Therefore, assessing the preserved structural information is a key element in evaluating a TMO. Other key element in assessing TMOs and LDR images is naturalness of the LDR image [12]. Naturalness is the correspondence degree between human perception and reality world [23]. Yeganeh and Wang show some LDR images that maintain the structural information of their corresponding HDR images well look overly dark [22]. Their proposed index, TMQI, attempts to make a balance between the two factors (i.e. structural fidelity preservation and high naturalness). However, these two factors are not the only factors under influence of the tone-mapping operations. Colourfulness which presents the degree of colour vividness, and image's free energy which tries to infer the meaningful part from the visual stimuli, are two other important factors that need to be considered when evaluating the image quality [23, 24]. Generally, colourfulness and naturalness are both decisive elements when images are judged for their perceived quality of colour reproduction [25]. Image's free energy, on the other hand, models the perception and understanding of the image with an active inference process in which the brain tries to explain the scene using an internal generative model [26]. It increases with presence of the detail and decreases with presence of the blurriness [27]. The following subsections first present these four factors and then propose the TMQI SNCF. 2.1 Structural fidelity SSIM method is one of fundamental methods to measure structural fidelities, from the image formation point of view [17]. SSIM combines structure s, luminance l and contrast c comparisons of two images (considering one image has perfect quality, then SSIM quantifies the quality of the second image). l, c, s and SSIM for two image patches, and x and y are defined as follows [17]: (2) (3) (4) (5)where μ x and μ y are the mean intensity values of the two signal images. σ x , σ y and σ x y are the standard deviations and cross-correlation between the two images. C l , C c and C s are constant values which are used to avoid instability in each component, and α, β and γ are positive values which are used to adjust the relative importance of each component. The overall SSIM score for a given image is computed as the mean value of the SSIM values of all pixels. The dynamic range of a LDR image totally changes after conversion from its respective HDR. Therefore, the SSIM's luminance component is not proper to be considered for quality assessment of TMOs [12]. The SSIM's contrast component penalizes any change in signal strength, while differences between HDR and LDR images should be only penalized when one of the two signal strengths is above and the other is below visibility threshold [12]. This led researchers to modify the contrast component and use the psychometric function Galton's ogive [28], which is similar to the cumulative normal distribution function, to nonlinearly map signals above and below visibility threshold to 1 and 0, respectively. This mapping function is defined as follows: σ ′ = 1 2 π θ σ ∫ − ∞ σ exp − ( x − t σ ) 2 2 θ σ 2 d x (6)where t σ is the modulation threshold, and θ σ is the standard deviation of the normal distribution which controls the slope of detection probability variation [12]. t σ is set to be μ ¯ ( 2 ) λ A , where μ ¯ is the mean intensity value, λ is a constant value and A is the contrast sensitivity function. θ σ is then set to be t σ k , where k is roughly a constant with a value between 2.3 and 4. The structural fidelity for each pixel for a HDR and LDR image is then defined as follows: S F ( x , y ) = σ x y + C s σ x σ y + C s 2 σ x ′ σ y ′ + C c σ x ′ 2 + σ y ′ 2 + C c (7)Finally, the mean value of the SF values of all pixels can be considered as the overall SF value. The structural fidelity can be computed in different scales to ensure the images details are visible [12, 18]. To this end, the HDR and LDR images are iteratively low-pass filtered, and down-sampled before their corresponding overall SF value is calculated. The final multi-scale SF score MSF can be computed as follows: MSF = ∏ l = 1 L S F l w l (8)where L is the total number of scales, and S F l and w l are the overall structural fidelity score and its respective weight in lth scale. To compute the structural fidelity for an RGB colour image, it is first converted to Y x y space and then the structural fidelity of the Y component is measured (x and y are the chrominance channels and Y is the luminance channel). 2.2 Naturalness and colourfulness While naturalness is a subjective quantity, TMO methods try to simulate the naturalness of reference images to obtain realistic and natural low dynamic images. Studies show that brightness and contrast are highly correlated with perceived naturalness, and brightness mapping is an especially inevitable issue in all tone-mapping operations [12]. Therefore, statistical naturalness models based on these two factors are generally used to obtain the perceptual naturalness [12, 21]. In [12], Yeganeh et al. built a model by analyzing the histograms of the means and standard deviations of about 3000 grey scale image and found these histograms can be properly fitted using a Gaussian and a beta probability density functions, G P and B P , as follows: G P ( m ) = 1 2 π σ m e x p − ( m − μ m ) 2 2 σ m 2 (9)and B P ( d ) = ( 1 − d ) β d − 1 d α d − 1 B ( α d , β d ) (10)where m and d represent the mean and standard deviation, and B(., .) is the beta function. The values for μ m , σ m , α d and β d are, respectively, found to be 115.94, 27.99, 4.4 and 10.1 using regression [12]. Then, the NSS measure is computed by NSS = 1 K × G P ( m ) × B P ( d ) (11)where K is a normalization factor, set to max ( G P ( m ) , B P ( d ) ) , to ensure the NSS is bounded between 0 and 1. The naturalness is also applied on the Y channel of the Y x y colour space. On the other hand, colourfulness, which presents the degree of colour vividness, determines the attribute of colour appearance responsible for the subjective chromatic response's strength by which the hue is recognized [25]. When the dynamic range of HDR images are compressed to produce their corresponding LDR images, colour information get lost to some extent [29]. Therefore, one factor in evaluating TMOs can be measuring how much colour information is preserved by the LDR image. Computing the perceived colourfulness of a natural scene is a challenging task. As a result, researchers have proposed different techniques to measure the image's overall colourfulness [23, 25, 29, 30]. Inspired by the colourfulness indices proposed by [29, 30], the index to compute the perceived colourfulness of the LDR image is defined as the follows. First, the RGB colour space is converted into opponent colour space as follows: (12) (13)Then, the colourfulness index (CI) is computed as follows: CI = log σ r g 2 | μ r g | C C I log σ y b 2 | μ y b | C C I (14)where σ r g , σ y b , μ r g and μ y b are the standard deviation and mean values of their respective r g and y b colour channels, and | . | indicates the absolute value operation. C C I is a constant value which is used to avoid instability in the CI index. Based on my experiment, images with more natural colour have CI values between 3 and 3.5 and the CI value for other images is either less than 3 or bigger than 3.5. The experiment for this setting is done using 20 collected natural colour images and adding different types of noise to these images. Then, the degrees of the colourfulness for each of the reference images and their respective noisy images are computed and scaled using different pair of values. As a result, 3 and 3.5 are the pair of values which give the 20 original natural images the best colourfulness score compared with the noisy images. To ensure the CI values are bounded between 0 and 1, the bell-shaped (i.e. Gaussian) multi-options function is used to scale them. This function sets the scaled CI for the images which their CI index is between [3 3.5] and 1, and for CI values less than 3 and higher than 3.5 uses the left and right sides of the function. The multi-options to calculate the scaled colourfulness index SCI is as follows: SCI = exp − ( C I − p l ) 2 2 σ 2 C I < p l 1 p l ≤ C I ≤ p r exp − ( C I − p r ) 2 2 σ 2 C I > p r (15)where p l = 3 , p r = 3.5 and σ is a constant value that determines how the CI values, out of the [3 3.5] range, are scaled. Figure 1 plots the general form of the SCI measure for σ = 1 . The figure clearly indicates that the colourfulness of every image with a CI value out of the [3 3.5] range is proportional to its distance from this range. FIGURE 1Open in figure viewerPowerPoint The CI values scaled in accordance with the plot Finally, the colourfulness and naturalness of the scene can be linearly combined to form the colour quality of the image NC as follows: NC = w × N S I + ( 1 − w ) × S C I (16)where w is a weighting parameter to compromise between the naturalness and colourfulness constraints. I set w = 0.75 to be consistent with the Yendrikhovski et al. [25] research in which their proposed colourfulness and naturalness indices were linearly combined and the weighting parameter experimentally set to be 0.75. 2.3 Image free energy Free-energy (FE) principle unifies several brain theories in biological and physical sciences about human action, perception and learning [26]. Since all adaptive biological agents resist the natural tendency to disorder in an ever-changing environment, FE principle avoids encountering ‘surprise’ to keep the agents' internal states at the low entropy level under different environments and maintain the states within some physiological bounds. This ensures that the biological agents can violate the second law of thermodynamics (which states a system's entropy that is not at its equilibrium tends to increase over time, approaching a maximum value at the equilibrium [26]). The biological agent cannot measures or avoids ‘surprise’ directly and free energy is usually used to upper bound it. The minimization of free energy implicitly minimizes ‘surprise’. On the other hand, the Bayesian brain theory indicates that the human visual system uses an internal generative mechanism (IGM) for visual perception while some structural uncertainties are avoided for understanding [31]. FE minimization is highly related to the predictive coding [26]. Thus, if we consider IGM to be a linear autoregressive (AR) model which separates the disorderly uncertainty from the input scene, then the process of FE minimization is equivalent to encoding the input visual signal I with the minimum number of bits based on the AR model [24, 26, 31, 32]. To minimize the coding length, the piecewise AR model is used, which ensures the model parameters adjust on a pixel-by-pixel basis [26]. As a result, the total description length of the image I with the kth-order AR model is defined as follows [24, 26]: L ( θ ) = − log P ( I | θ ) + k 2 log N (17)where θ is the parameter vector, N is the number of pixels and the model is selected by minimizing L ( θ ) . In addition, for the large samples, when N → ∞ , the FE ℑ ( θ ) is the total description length: ℑ ( θ ) = − log P ( I | θ ) + k 2 log N , with N → ∞ (18) This means, total description length of the image data computed using the AR model is an approximation of the image's FE. In other words, the entropy of the prediction residuals plus the model cost can be used as the image's FE estimation. In practice, a fixed-model order such as k = 8 is used to simplify the complicated model selection process and ignoring the second term k 2 log N [24, 26]. In addition, changes in visual appearance such as direction, scale and frequency lead to images with different structures. Therefore, I propose to estimate the image's free energy in different scales. As a result, the free energy for a given image I is defined as follows: F E ( I ) = f e 1 + f e 2 f e 4 + f e 8 (19)where f e i represents the approximated image's FE at a decreased resolution in which each non-overlapping i × i patch of the image I is replaced by the local mean value of the patch. Equation (19) not only guarantees employing the estimated FE in multi-scales, but also results in a more robust value by considering a proportion that approximately indicates the slope of changes in the FE in different scales. Finally, the four discussed features (i.e. Sstructural fidelity, naturalness, colourfulness and free energy) are combined together to propose the TMO quality assessment index SNCF as follows: SNCF = ( a × MSF ) + ( b × NC ) + ( c × FE ) (20)where a, b and c are real values between 0 and 1 to adjust the relative importance of the three components. 2.4 Implementation The proposed SNCF method has several parameters that need to be set to compute the final tone-mapped image quality score. Some parameters, which are somehow related to previous researches, are set using their corresponding recommended values and some others are set empirically. Following the psychophysical experiment results in [18], the number of scales L and the weights w l are set to be 5 and {0.0448, 0.2856, 0.3001,0.2363, 0.1333}, respectively [12, 18]. C l and C s in the structural fidelity index are set to be 0.01 and 10, respectively. C C I in the colourfulness index is set to be 0.25. The real values a, b and c in Equation (20) are empirically set to be 0.7742, 0.1290 and 0.0968, respectively. As it will be discussed shortly, three datasets are used to evaluate the proposed metric. All the empirically set values are investigated using only one of these datasets (the Yeganeh&Wang dataset), and so the results on other two datasets can verify the robustness of these parameters. The algorithmic view of the proposed index is illustrated in Algorithm 1. ALGORITHM 1. Algorithmic view of the proposed tone-mapped image quality score Input: The reference HDR image and the respective LDR image. Output: The final tone-mapped image quality score. 1: Compute the multi-scale structural fidelity score using Equation (8). 2: Compute the natural scene statistics (NSS) measure using Equation (11). 3: Compute the colourfulness index (CI) and its scaled version using Equations (14) and (15). 4: Compute the combined colourfulness and naturalness score using Equation (16). 5: Compute the image's free energy using Equation (19). 6: Set the final tone-mapped image quality score using Equation (20). 3 EXPERIMENTAL RESULTS In this section, the proposed index is compared with some of the recently proposed state-of-the-art methods. The compared methods include TMQI, TMQI-II, the three variants of FSITM method (i.e. FSITMR, FSITMG and FSITMB) and the three variants of FSITM _TMQI (i.e. FSITMR-TMQI, FSITMG-TMQI and FSITMB-TMQI). All of these methods are implemented in MATLAB using the code available at the authors' websites. Moreover, for each metric, the parameters are set as recommended by the respective researchers. These TMO quality assessment metrics are evaluated by conducting experiments on three publicly available datasets. The three datasets are Yeganeh&Wang [12], MMSPG [33] and TMQID [34] datasets. For the first two datasets, since their MOSs are available, the Spearman's rank-order correlation coefficient (SRCC) and the Kendall's rank-order correlation coefficient (KRCC) are employed to objectively evaluate the performance of the proposed method and other state-of-the-art methods. SRCC and KRCC are defined as follows: (21) (22)where r s and r o are the subjective and objective rank scores of the ith image, respectively. N is the number of LDR images, and N c and N d are the numbers of concordant rank order and discordant rank order pairs in the data set, respectively. For the third dataset, the authors in [34] provided two new measures to answer two questions in evaluating the metric performance. The two questions are as follows [34]: Q1) ‘Can the metrics successfully predict which images are perceived to be statistically different by the observers?’, and Q2) ‘If the two images are different (with statistical significance), how well can the metrics determine which one is of better quality?’. To answer these questions, based on the scores obtained by a metric, a binary classification problem is developed where its decision ability (and therefore effectiveness of the respective metric) is evaluated by receiver operating characteristics (ROC) analyses and the area under curve (AUC) values. Answering each question results in different ROC analysis: different vs. similar ROC analysis for Q1 with AUC values called AUC-DS values, and better vs. worse ROC analysis for Q2 with AUC values called AUC-BW values. Higher AUC-DS and AUC-BW values for a metric indicate its better performance compared with its peer [34]. It should be mentioned that these measures do not need subjective rating/ranking of the images. 3.1 Y e g a n e h & W a n g dataset The Yeganeh&Wang dataset is introduced in [12] and contains 15 HDR images. For each HDR image, there are eight LDR images. Five of these eight LDR images are created using the TMOs developed by Reinhard et al. [5], Drago et al. [2], Durand and Dorsey [35], Mantiuk et al. [36] and Pattanaik et al. [37], and other three created by Exposure and Gamma, Equalize Histogram and Local Adaptation built-in TMOs in Adobe Photoshop. The HDR and LDR images were given to 20 subjects for their subjective quality assessment. For each HDR image, the observers rank the eight images and the respective average rank for each LDR image across all 20 subjects is considered as mean ranking score for each LDR image. Table 1 summarizes the SRCC and KRCC values for the proposed metric together with the TMQI TMQI-II, and different channels of FSITM and FSITM _TMQI metrics. While the first column indicates the compared methods, columns 2–16 give the obtained SRCC and KRCC values for each HDR image. Column 17 provides the mean SRCC and KRCC value across all 15 HDR images together with the respective standard deviations reported in the parentheses. To better see the changes in the obtained values, the median value of the 15 SRCC and KRCC value is given in the last column of the table. Clearly, the proposed index achieves the highest mean SRCC and KRCC values of 0.8733 and 0.7652 as well as the highest median values of 0.9048 and 0.7857. For example, compared with the eight state-of-the-art methods, the proposed index improves the SRCC's mean value of TMQI, TMQI-II, FSITMR, FSITMG, FSITMB, FSITMR-TMQI, FSITMG-TMQI and FSITMB-TMQI by 8.80%, 13.62%, 7.22%, 9.56%, 23.21%, 2.03%, 3.67% and 7.85%. TABLE 1. Performance comparison of the five methods using the SRCC and KRCC values for the Yeganeh&Wang dataset SRCC Method/image set 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Mean Median TMQI [12] 0.9048 0.7857 0.8095 0.9048 0.7619 0.9524 0.6429 0.6667 0.6905 0.9286 0.8810 0.7143 0.6826 0.7381 0.9762 0.8027(0.11) 0.7857 TMQI-II [21] 0.9048 0.5000 0.6905 0.6905 0.6667 0.9762 0.8333 0.6667 0.8095 0.9048 0.8333 0.5952 0.7904 0.7619 0.9048 0.7686(0.13) 0.7904 FSITMR [13] 1.0000 0.7619 0.6429 0.9048 0.7143 0.6667 0.9048 05714 0.9286 0.7857 0.9762 0.857