Quality assessment tool for performance measurement of image contrast enhancement methods

IET Image ProcessingVolume 13, Issue 5 p. 833-842 Research ArticleFree Access Quality assessment tool for performance measurement of image contrast enhancement methods Mohsen Abdoli, Corresponding Author Mohsen Abdoli mosen.abd@gmail.com orcid.org/0000-0002-9308-4156 ATEMEn, Rennes, France L2S Laboratory of Signal and Systems, CNRS UMR 8506, CentraleSupélec, Gif-sur-Yvette, France Université Paris-Saclay, Paris, FranceSearch for more papers by this authorFatemeh Nasiri, Fatemeh Nasiri National Institute of Applied Sciences INSA, Rennes, France Institute of Research and Technology, IRT b-com, Cesson Sévigné, France Aviwest, Saint-Grégoire, FranceSearch for more papers by this authorPatrice Brault, Patrice Brault L2S Laboratory of Signal and Systems, CNRS UMR 8506, CentraleSupélec, Gif-sur-Yvette, France Université Paris-Saclay, Paris, FranceSearch for more papers by this authorMohammad Ghanbari, Mohammad Ghanbari Department of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran School of Computer Science and Electronic Engineering, University of Essex, Colchester, UKSearch for more papers by this author Mohsen Abdoli, Corresponding Author Mohsen Abdoli mosen.abd@gmail.com orcid.org/0000-0002-9308-4156 ATEMEn, Rennes, France L2S Laboratory of Signal and Systems, CNRS UMR 8506, CentraleSupélec, Gif-sur-Yvette, France Université Paris-Saclay, Paris, FranceSearch for more papers by this authorFatemeh Nasiri, Fatemeh Nasiri National Institute of Applied Sciences INSA, Rennes, France Institute of Research and Technology, IRT b-com, Cesson Sévigné, France Aviwest, Saint-Grégoire, FranceSearch for more papers by this authorPatrice Brault, Patrice Brault L2S Laboratory of Signal and Systems, CNRS UMR 8506, CentraleSupélec, Gif-sur-Yvette, France Université Paris-Saclay, Paris, FranceSearch for more papers by this authorMohammad Ghanbari, Mohammad Ghanbari Department of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran School of Computer Science and Electronic Engineering, University of Essex, Colchester, UKSearch for more papers by this author First published: 21 March 2019 https://doi.org/10.1049/iet-ipr.2018.5520Citations: 7AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract An objective image quality assessment tool is proposed to measure image enhancement quality with emphasis on contrast. In the proposed tool, which is based on maximizing contrast with minimum artefact (MCMA), local and global properties of an image are measured through pixel-wise and histogram-wise features, respectively. To this aim, three sub-measures are introduced, each of which able to detect one contrast-related quality aspect: (i) low dynamic range of image; (ii) histogram shape preservation during image enhancement process; and (iii) local pixel diversity. These sub-measures are combined through a subjective test to adapt them to the mean opinion scores (MOSs) of a diverse set of training contrast-enhanced images. A regression algorithm performs the adaptation by fitting the three sub-measures to the MOS values and finding an optimal linear combination by maximizing the Pearson correlation. In order to evaluate the performance of the MCMA algorithm, another independent, subsequent, subjective test was performed on a set of images enhanced by various known contrast enhancement algorithms to obtain MOS values and to compare them with the output of the proposed MCMA method. The experimental results show that MCMA has the highest correlation to the MOS when compared to the existing tested contrast measurement tools. 1 Introduction Demands for enhancing images taken by digital cameras in poor lighting conditions [1], low contrast medical [2] or remote sensing images [3] have encouraged researchers to devise image contrast enhancement algorithms. In the past few years, various contrast enhancement techniques have been introduced [1, 4-7]. However, like other image restoration tasks, there is always a risk of quality degradation during the contrast enhancement process. The effectiveness of avoiding this degradation, which mostly appears in the form of image information loss or introducing new artefacts (or magnifying the existing ones), discriminates between good and bad enhancement methods. For instance, one of the most common artefacts that appears in a contrast enhanced image is over-enhancement, which is sometimes referred to as 'saturation' [8, 9]. As the name suggests, it happens when a contrast enhancement method substantially enlarges the difference between neighbouring pixels' intensity levels. In the domain of automatic image restoration, such degradations should be avoided to achieve the best enhancement result. The first prerequisite to this goal is to enable the enhancement algorithms to detect occurrences of enhancement side-effects. This can be done by an efficient and consistent automated contrast-based objective measure to properly quantify contrast-based quality during or after enhancement. Unlike image quality assessment (IQA) in general, which has been thoroughly studied in the literature, focusing on contrast for designing objective IQA tools has been mostly overlooked. The necessity of an efficient contrast-based IQA tool could be well perceived by noting that most authors present the performance of their enhancement methods by visual assessment instead of objective measures, which makes the results unreliable and difficult to verify. In addition, a good contrast-based IQA tool can also act as an objective function to optimise in parametric image auto-correction applications. As will be discussed, most of the existing contrast assessment tools are unable to properly correlate to the human quality perception, due to their simplicity. Studies on the human visual system (HVS) and its properties have shown that human quality perception is too complex to be modelled by simple plots [10, 11]. Instead, it requires considering various aspects of quality perception and their proper combinations. This is the main reason why advanced IQA algorithms try to extract as many features as possible from the image to combine them and introduce an effective measure. However, in the area of contrast measurement, the feature extraction task is not clearly obvious and one must be aware of different artefacts and their impacts on the human quality perception. Another important challenge in the contrast-based IQA is the use of an original low quality image as the reference. Unlike applications such as image compression where an excellent reference image (e.g. uncompressed image) is used to evaluate the quality of the output image (e.g. compressed image), the input image in the contrast enhancement tasks is mostly degraded and untrustworthy. While in the conventional image quality metering, one tries to make the processed image as close as possible to the reference image, in the contrast-based one it is aimed to be as different as possible. Also, since dissimilar images could include irrelevant ones, this makes the metering task complicated. Therefore, using the conventional reference-based IQA tools [e.g. peak signal to noise ratio (PSNR) and structural similarity (SSIM)] with low contrast input image as a reference would lead to ineffective quality measurement. Nonetheless, ignoring the input image would also keep us from detecting enhancement artefacts. This is due to the fact that, despite the low contrast, the input image still carries useful information that is actually the only clue for artefact detection. Therefore, it can be concluded that the low contrast input image needs to be 'carefully' exploited in an efficient objective quality assessment [12]. Considering this fact, another drawback in some of the existing objective measurement tools is that they quantify the contrast in a 'no-reference' (NR) manner. Although the NR property is sometimes advantageous (e.g. when the reference signal is not available) [13-15], it spoils any opportunity to detect artefacts added during the enhancement process [5]. In this study, the proposed maximum contrast with minimum artefact (MCMA) IQA tool tries to measure the performance of a contrast enhancement method by considering two factors: visual quality improvement and information preservation. 2 Related works The problem definition of IQA is manifold in nature due to the complexity of the HVS. In fact, a proper HVS model must deal with biological and psychological processes that have not been fully understood yet [11, 16]. The interpretation of this complicated nature in terms of IQA is that practically there are different ways to degrade the quality of an image, each of which focusing on one of the HVS aspects (e.g. different noises, blurriness, blockiness artefacts due to compression etc.). The consequence of this complexity is reflected in the literature, where the existing IQA tools either focus merely on a single aspect or consider all of them together and introduce a general IQA tool [11]. A handful of objective IQA measures have been proposed to measure the contrast-based quality perceived by HVS. Like other IQA problems, the existing tools are either full-reference (i.e. another version of the image with perfect quality is available as the reference) [12], NR (i.e. the reference image is not available) [5, 13-16] or reduced-reference (i.e. selected features of the reference image are available) [9, 17]. Here some of the most popular contrast measurement tools are briefly reviewed. In the early contrast enhancement methods, the histogram mean displacement problem was common. That is to say, if a naturally over-exposed/under-exposed image is given as the input, the enhancement algorithms tend to darken/brighten it, respectively. Therefore, one of the first contrast measures called absolute mean brightness error (AMBE) was introduced to measure this very basic artefact [17]. The main problem of the AMBE is that it tends to give a higher quality index to images that have undergone minor or no enhancement. The entropy (H) of pixels is another indicator of image contrast in the literature [18]. The entropy measure is able to properly reflect the contrast drop, but not contrast enhancement. This is especially true when the number of unique intensity levels of image histogram remains the same before and after enhancement, which is the case for the majority of histogram-based contrast enhancement methods. In these methods, the entropy remains almost the same after the enhancement, since all that is done is shifting pixels intensities to cover the entire dynamic range of the histogram, which theoretically does not alter the entropy. The measurement of enhancement (EME) and the EME by entropy (EMEE) are two parametric measurement tools which quantify the pixel diversity in small blocks and are then extended to the whole image by averaging [19]. The pixel-wise logarithmic approach to assess the contrast in each block is helpful to avoid a huge drop in the plain areas of the image. However, they both have a weak performance when a highly detailed image is over-enhanced. In this case, each block contains both extreme dark and extreme bright pixels and deserves a penalty. However, the measures would consider this block as a good contrast content. The Michelson contrast measure (AME) [20] and its advanced logarithmic successor, the LogAMEE (logarithmic AME) [21] are two other measures very similar to the EME and the EMEE where the same block partitioning approach is applied to assess local pixel diversity. They address some of the weaknesses in the EME and EMEE, especially with the use of parameterised logarithmic image processing operations in LogAMEE [22]. However, the main problem of over-enhancement for highly detailed images still remains. The histogram flatness measure (HFM) and histogram spread (HS) are two other histogram-based measures that use the descriptive statistics of the enhanced histogram to determine its contrast [23]. Although these approaches help considering the shape of the histogram which is assumed as a significant indicator of contrast, none of them considers the content of the image in their calculations. Therefore, they are unable to detect any artefact that has no impact on the histogram, yet distorts the image in the pixel domain. With a quick look at the literature, it can be observed that recent contrast measures are more advanced by utilising contrast-correlated statistics of the image. The reduced reference image quality metric for contrast (RIQMC) measure simply builds the measure by weighted averaging of five statistical features [5]: entropy-based similarity, mean-based comfort, variance-based comfort, the skewness-based comfort, and the kurtosis-based comfort. This measure has an excellent correlation to the human quality perception [e.g. mean opinion score (MOS)]. NR quality metric for contrast-distorted images (NR-CDIQA) is another measurement tool, which uses the same set of statistical features as the RIQMC. In NR-CDIQA, the goal is to quantify contrast-distortion regarding the naturalness of image [24]. This goal is achieved by using a natural image dataset to fit the above-mentioned features' curve on the Gaussian, inverse-Gaussian and Gumbel distribution functions. The performance of NR-CDIQA is sometimes comparable with RIQMC. In [25], a blind contrast assessment measure is introduced where both contrast/spatial distributions of the image are taken into account. In this measure, multi-threshold tetra patterns and Weber–Laplacian of Gaussian operator are used to extract spatial/contrast features. These features are then combined to build the final IQA measure. In addition to the aforementioned measures, there are also other contrast measures which are not used in the experimental results of this study. One of the most important measures is the combination of a set of sub-measures namely the distribution separation measure, the target to background contrast enhancement measure based on entropy and the target to background contrast enhancement measure based on standard deviation [25]. This measure merely focuses on the mammographic images hence it is not considered here. Moreover, it was observed that in some previous studies [26-29], the performances of contrast enhancement tools are evaluated using the so-called (PSNR) [30]. However, in a contrast enhancement process, like in almost all other image restoration problems, the original image is not perfect and therefore it is improper to be referred to. The rest of this paper is organised as follows. In Section 3, the three proposed features are introduced, each of which try to model one aspect of the perceived image quality. In Section 4, the adopted subjective test method is described step by step. A regression process is provided in Section 5, which finds a combination of the features that optimally fit the results of the subjective test. In Section 6, the performance of the proposed measurement tool is compared against the others with a benchmark of the subjective test results. Finally, Section 7 concludes the paper. 3 MCMA quality assessment tool The MCMA tries to obtain as much information as possible about the factors affecting human quality perception and the way these factors influence quality. In this section, three sub-measures are introduced, each of which considers specific contrast-based visual quality degradations: (i) When the global pixel diversity is not sufficient. This problem mostly happens when a picture is taken in an imperfect lighting condition and the picture degradation is reflected in a narrow band histogram. (ii) When the global shape of the histogram is distorted during the enhancement. In this situation, the relative distances of the pixel values in the locally spatial neighbourhoods abnormally change and the HVS perceives it as an artefact (e.g. saturation or abnormal pixel values). (iii) When the local pixel diversity in small neighbourhoods is not sufficient. This problem causes edge vanishing and can exist even when the global pixel diversity is satisfactory [25]. 3.1 Dynamic range occupation (DRO) It is well known that the weak dynamic range of an image, reflected in the narrowness of the histogram, is the main characteristic of a low contrast image and therefore, almost all enhancement methods aim at broadening the histogram. The DRO is used as the first and most simple contrast indicator for MCMA. As its name suggests, the DRO is a ratio that represents the currently occupied portion of the dynamic range (i.e. for a bit depth of m). To improve the performance of the DRO, a de-noising step is needed to remove outlier values from both ends of the histogram (i.e. salt-pepper noise). To eliminate outlier values from the histogram, the first and the last α percentile of the histogram are ignored. Assuming that ifirst and ilast are the indices of the first and the last non-zero entries of the histogram after the outlier removal phase, the NR DRO is calculated as (1)The importance of the DRO is magnified when a conservative enhancement method does not increase the contrast sufficiently while the goal is to avoid the over-enhancement artefact. In this situation, the resultant image which has not undergone 'enough' enhancement, receives a low quality index from DRO. Fig. 1 shows three different levels of contrast for the 'statue' image with their corresponding DRO values. Fig. 1Open in figure viewerPowerPoint Relation between different DRO values and the actual contrast-based quality of an image 3.2 Histogram shape deformation (HSD) In previous work, the authors have shown that the histogram shape preservation is always a necessary condition for having a desirable enhanced image [31]. In other words, when the shape of an original low contrast histogram is modified during the enhancement, the resultant enhanced image would surely be degraded. This degradation could occur either in the form of pixel saturation or by introducing abnormal pixel values. Fig. 2 shows examples of these two artefacts along with their corresponding histograms. As can be seen, the histograms of the poorly enhanced images in Fig. 2b are deformed in two different ways, which have resulted in two different artefacts. To quantify the amount of shape modification, a RR sub-measure called HSD is proposed. Fig. 2Open in figure viewerPowerPoint Impact of HSD (a-b) Original low contrast images with their narrow histograms:diversity of pixel intensities is not sufficient and almost nodetails can be recognised, (c-d)Poorly enhanced images with their highly deformed histograms: incameraman, an unusual concentration in the middle of the histogramhas introduced abnormal intensities in the face and the sky. In F16,the gap in the middle of the histogram has introduced saturation bymaking dark pixels too darker and bright pixels too brighter,(e-f) Properly enhancedhistograms with preserved shapes: differences in the intensitylevels of neighbouring pixels are adequate to reveal the details The main objective of the HSD is to quantify the similarity between two histograms: input histogram and enhanced histogram. In most cases of contrast enhancement, the enhanced histogram is sparsely widened (i.e. made of non-zero entries, surrounded by zero entries). Therefore, using a point-to-point histogram similarity measures (e.g. sum of squared errors) is pointless. The main problem in these measures is their inaccuracy in case of sparse histograms. For instance, if one takes a sparse histogram and shifts its entire entries one unit to the right, then the shifted histogram would subjectively look quite the same as the input histogram. However, since all the non-zero entries are now shifted to zero entries, then the point-to-point similarity measure would recognise the input histogram and its shifted version as two very dissimilar histograms. This problem is due to the extremely localised nature of the point-to-point error calculation approaches. To address the above problem, the HSD sub-measure proposes a less local approach to calculate the difference between the histograms' shapes. The first step in the calculation of HSD is to uniformly stretch the original narrow histogram to occupy the entire dynamic range of the histogram. To do so, a stretching factor is defined as and is used in a stretching function which smoothly moves non-zero entries towards the proper end of the dynamic range (i.e. 0 or ). The stretching process is carried out in three steps: (i) Dilation: re-locate the non-zero entries in a way that they become equally distanced. (ii) Interpolation: replace zero entries by linearly interpolating the closest re-located non-zero entries. (iii) Normalisation: divide each non-zero entry to the sum of non-zero entries to preserve the histogram property (i.e. sum heights equalling 1). Fig. 3 summarises the process of stretching a narrow histogram with stretching factor . Fig. 3Open in figure viewerPowerPoint Process of stretching a narrow histogram by applying three steps: (1) dilation, (2)interpolation, and (3) normalisation To calculate the HSD sub-measure, the stretching function is applied to both reference and enhanced histograms to ensure full DRO. In the next formulas, the stretched and interpolated versions of the low contrast original histogram and the enhanced histogram are referred to as and , respectively. The third step in the HSD calculation is to define a 1D moving window with length d to independently quantify the local similarity of and for all neighbourhoods of length d. This window moves from zero towards and in each position , the local error between and is calculated as (2)In fact, (2) considers as a perfect reference histogram and locally quantifies its similarity to the imperfectly enhanced histogram . The role of the moving window is to ignore minor histogram displacements. Fig. 4 shows an example of calculating the local error by putting a moving window of size d = 8 on position pos = 10. Fig. 4Open in figure viewerPowerPoint Calculating the local error function with a moving window of size 8 After calculating the local errors for all , the deformation value can be calculated by averaging on local errors over all positions of the moving window (3)The final deformation sub-measure is obtained after normalising the output of (3) by its maximum possible value MAXDEF. (4)Theoretically, the maximum possible value of the deformation is reached when non-zero entries of the two histograms have absolutely no overlap in all windows. In this situation, (3) would result in 2 × d × R × C, where R and C are the number of rows and columns of the image, respectively. This number is actually the sum of two non-normalised histograms times the length of the window. Fig. 5 shows how the histogram deformation during enhancement is reflected in the HSD sub-measure. Here, Fig. 5a is a low contrast reference image and Figs. 5b–i are eight different enhanced versions of it. The HSD values besides the equivalent histograms of the nine images are depicted in Fig. 5j where the histogram of the reference image is shown in the dark background. It can be seen, as the shape of the enhanced histogram distorts, the value of the HSD sub-measure increases. Fig. 5Open in figure viewerPowerPoint Impact of different levels of enhancement on the histogram shape (a) Original low contrast image of Astro, (b)–(i) Eight different enhanced versions of the original image with the increasing level of contrast, (j) Values of the HSD sub-measure for (a)–(i) with their corresponding deformed histograms The choice of window size d is critical in the performance of the HSD. In order to choose the optimum d, different values were tested and integrated into the rest of the proposed method. Fig. 6 shows how different window sizes impact the performance of the final measure. Fig. 6Open in figure viewerPowerPoint Impact of using different window sizes d on the final performance of the proposed measure in terms of correlation with the MOS values It is also important to note that the main purpose of using HSD is to detect the occurrence of 'over-enhancement'. In fact, the HSD is not able to penalise the enhancement in case of 'under-enhancement'. By reminding that the DRO sub-measure can detect under-enhancement and is not able to measure over-enhancement, it can be concluded that a proper combination of DRO and HSD can perform more effectively than the sub-measures individually. 3.3 Pixel uniformity (PU) In terms of histogram representation (i.e. 1D luminance-frequency domain), contrast degradation is reflected in the narrow shape of the histogram and the equivalent impact in the spatial domain (i.e. 2D luminance-position domain) is a lack of pixel diversity. Thus, the PU sub-measure aims at representing pixels diversity in a NR interpretable sub-measure. Fig. 7 shows a small neighbourhood from the cameraman image, including the ear and the hair. As can be seen, in Fig. 7a, pixels of skin and hair areas are too similar to be correctly discriminated because of low inter-region diversity or poor contrast. In Fig. 7b on the other hand, these two areas are easily discriminable; however, their inner details have disappeared due to the pixel saturation that causes low inner-regional diversity or high saturation. Actually, the reason that Fig. 7c is a properly enhanced image, is because of its suitably balanced contrast and saturation (i.e. inter- and inner-region pixel diversity). As a result, in almost all areas, the visible details are preserved in Fig. 7c. Fig. 7Open in figure viewerPowerPoint Spatial domain diversity of pixel intensities and its effect on the contrast of the 'ear' and 'hair' regions (a) Low contrast, (b) Saturated, (c) Proper contrast A logical way of simultaneously considering contrast and saturation is to look at the relationship between every 'target' pixel with its neighbours. A large luminance difference indicates a higher contrast and a small difference either reflects saturation or a contrast drop. A simple way of reflecting these two parameters into an equation is to define it for a block of 3 × 3 pixels (i.e. namely an 8-connexity). In this model, the level of luminance distance of the target pixel from its eight neighbours can be a measure of contrast. Correspondingly, its degree of closeness can be interpreted either as saturation or degradation of contrast. Given an input image I with R rows and C columns, the local PU value for each 3 × 3 pixels block centred at pixel (i,j) is formulated as (5)where I(i,j) is the target pixel in the centre of the block and I(p,k) values are the neighbours in the 8-connexity. By averaging the values for each target pixel, using (6), over the entire image gives the global PU sub-measure. It is important to note that the image must be padded prior to this step, in order to take the border pixels into consideration (6) 4 Subjective test The goal of the objective quality assessment task in all visual multimedia processing applications is to propose a numeric formula, which best models the human quality perception. Therefore the only way of evaluating the performance of different visual quality assessment tools is to compare them against the output of a common subjective quality assessment method. For this purpose, a set of images with different levels of contrast and quality were generated, and then a double-stimulus continuous quality scale (DSCQS) method was performed on them [32]. In the DSCQS method, the images are presented in pairs: the reference image and its enhanced one. Subjects are given unlimited time to compare the enhanced image to its reference and grade them. For this purpose, a graphical user interface (GUI) was developed to show the images and take the grades. Fig. 8 shows a screen shot of this tool. In order to provide the subjects the test images in their full resolutions, this GUI was shown in two high-definition screens. Fig. 8Open in figure viewerPowerPoint Subjective quality assessment GUI In order to provide diversity in terms of content and image format, a total of 36 low contrast images in three classes of resolutions (1920 × 1280, 960 × 640 and 512 × 512 pixels) were prepared, making a total of 8 × 36 = 288 different images. These images are shown in Fig. 9. Fig. 9Open in figure viewerPowerPoint Three classes of test images for the subjective quality assessment test To produce the test set, each image was first modified by a parametric contrast adjustment software to produce a very low contrast version. This low contrast image was then enhanced several times with different parameter values of the same contrast adjustment tool. This resulted in eight different enhanced versions from each low contrast image. Fig. 10 shows these progressively enhanced images from one of the samples. The enhancement parameter of each image was carefully selected so that each set contains all possible outputs including low contrast, proper contrast and then over-enhanced images. Fig. 10Open in figure viewerPowerPoint Example dataset image before and after contrast modification (a) Original low contrast image fro