Enhanced image no‐reference quality assessment based on colour space distribution

IET Image ProcessingVolume 14, Issue 5 p. 807-817 Research ArticleFree Access Enhanced image no-reference quality assessment based on colour space distribution Hao Liu, Hao Liu College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorCe Li, Corresponding Author Ce Li xjtulice@gmail.com College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorDong Zhang, Dong Zhang College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorYannan Zhou, Yannan Zhou College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorShaoyi Du, Shaoyi Du Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, People's Republic of ChinaSearch for more papers by this author Hao Liu, Hao Liu College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorCe Li, Corresponding Author Ce Li xjtulice@gmail.com College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorDong Zhang, Dong Zhang College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorYannan Zhou, Yannan Zhou College of Electrical and Information Engineering, Lanzhou University of Technology, Lanzhou, People's Republic of ChinaSearch for more papers by this authorShaoyi Du, Shaoyi Du Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University, Xi'an, People's Republic of ChinaSearch for more papers by this author First published: 19 February 2020 https://doi.org/10.1049/iet-ipr.2019.0856Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this study, the authors investigate the problem of enhanced image no-reference (NR) quality assessment. For resolving the problem of the enhanced images, it is difficult to obtain reference images, this study proposes an NR image quality assessment (IQA) model based on colour space distribution. Given an enhanced image, our method first uses a gist to select a clear target image in which the scene, colour and quality are similar to the hypothetical reference images. And then, the colour transfer is used between the input images and target images to construct the reference image. Next, the appropriate IQA method is used to assess enhanced image quality. The absolute colour difference and feature similarity (FSIM) are used to measure the colour and grey-scale image quality, respectively. Extensive experiments demonstrate that the proposed method is good at evaluating enhanced image quality for X-ray, dust, underwater and low-light images. The experimental results are consistent with human subjective evaluation and achieve good assessment effects. 1 Introduction As a result of the rapid development of image acquisition equipment, the amount of images produced to save and collect information in life is constantly growing. So a lot of low-quality images appear, such as X-ray, dust, underwater and low-light images. These low-quality images have the colour inharmonious problem. For some reasons such as health, weather, environment and time, it is difficult to obtain clear images of such images after repeated collection. Many enhancement algorithms are proposed to process these images. Hence, how the quality of these enhanced images is judged becomes a serious problem. In the human visual system, it only takes a few seconds for people to determine the quality of images. A lot of psychological experiments show that in the first 20 s, the colour account for about 80% in perception, an image is observed [1]. So colour has a profound influence on the image quality evaluation. Through analysing the low-quality images which have the colour inharmonious problem, we find that enhanced images only improve the image colour information in most cases. We have no wish for any change in the image structure information after enhanced, which will seriously affect the image subsequent process. It also demonstrates the importance of colour for image quality. So, what is the colour in a high-quality image? In photography, the harmony and richness of the colour are important to measure the image quality, which means colour cast and too bright or too weak luminance degrade the image quality. In RGB colour model, it means the components of RGB colour space display strong correlations in the high-quality images of natural scene [2]. In most cases, the R, G, B channels of a clear natural image are positively correlated at the same brightness, as shown in Fig. 1. When we look at representative low-quality images which have the colour inharmonious problem, such as X-ray, dust, underwater and low-light, it is easy to identify that there are different inharmonious colours in these images. The dust images tend to be yellowish, and the underwater images appear too bluish. The brightness of X-ray and low-light images is on the low side. In Fig. 1, taking the case of underwater image, there is no positive correlation between R, G and B channels. This phenomenon reduces the harmony and richness of colour obviously. Similarly, the correlation of these three channels of dust images is also on the low side. The RGB and grey curves of the low-light and X-ray image are concentrated on the low-light areas which also decrease the image quality. Therefore, we come to a conclusion by analysing the generality of these low-quality images, when the R, G, B are positively correlated under the same brightness and not concentrated in the lower light areas, there is no colour cast and low brightness problems in the image. The better the harmony and richness of the colour are, the higher the image quality will be. Fig. 1Open in figure viewerPowerPoint Natural image, low-quality images and corresponding RGB curves Based on the above analysis, we find that the reference image can be constructed by adjusting the RGB curve of the low-quality image. It means the reference image's R, G, B curve is positively correlated at the same brightness and does not concentrate on the brightest or darkest areas. At last, we can construct reference images and realise the enhanced image quality assessment (IQA). Therefore, we propose an enhanced image no-reference (NR) quality assessment based on colour space distribution. By transferring the harmony and richness colour from high-quality images to low-quality images, this algorithm constructs a reference image. And then we compare the reference image with the enhanced image through the image evaluation index, so as to obtain the final quality assessment result. In this paper, the proposed method is verified through X-ray, dust, underwater and low-light images. The proposed algorithm is divided into two parts: construct reference image and quality assessment. First, we assume every enhanced image has a clear reference image. And we use the gist to select clear target images. The target images scene, colour and quality are similar to the hypothetical reference images. And then, the colour transfer is used between input images and target images to construct reference image. It means the reference image with similar quality as the hypothetical reference image is obtained. Then, we choose the right quality assessment index to compare the enhanced image and reference image. Finally, we get an enhanced IQA result. There are three advantages of this algorithm: first, our algorithm does not need a large number of data sets for training. It is only necessary to use low-quality images, enhanced images and clear images. Secondly, the quality assessment results of the proposed method are more inclined to evaluate the image colour information, which is in concert with the colour psychology experimental results and image quality subjective results. Finally, the proposed algorithm has a wide range of application that can measure X-ray, dust, underwater and low-light enhanced images quality without reference image. The remaining part of this paper is organised as follows. The related work is explained in Section 2. Section 3 briefly reviews the main block of the proposed method. The experimental results are shown in Section 4. In Section 5, we discussed the proposed method disadvantages. Finally, a conclusion is given in Section 6. 2 Related work According to the dependency of reference images, IQA methods are usually divided into three categories: Full-Reference IQA (FR-IQA), Reduced-Reference IQA (RR-IQA) and NR-IQA. In FR-IQA and RR-IQA, we need the whole or partial information of reference image to assess the image quality. Such as FSIM [3], mean squared error (MSE) [4], peak signal to noise ratio (PSNR) [5], structural similarity index (SSIM) [6], dual-path deep convolutional neural network (DCNN) [7] and reduced-reference image quality metric for contrast change (RIQMC) [8]. However, it is hard for low-quality images to supply clear information as a reference. Therefore, neither of these algorithms are suitable for straight evaluation of the enhanced images quality. NR-IQA does not need any information from the reference image, which enhances its applicability remarkably and renders it significant in practice. It can be concluded that NR-IQA is the best choice to assess the quality of enhanced images, but there still exist many problems to be solved. Most of the NR-IQA algorithms focus on the investigation of natural scene statistics and deep learning. Sheikh et al. [9] extracted effective statistical features from natural images to evaluate image quality without reference images. Zhang et al. [10] proposed an NR-IQA algorithm based on semantic information. They extracted two types of features, one to measure the semantic obviousness of the image and the other to discover local characteristic. Then, the two kinds of features were combined for image quality estimation. Marziliano et al. [11] detected vertical Sobel edge and obtained the edge width by searching the local extremum of grey value. The average edge width of all edge pixels was used to estimate the image quality. Wu et al. [12] proposed a local learning method for NR-IQA to improve quality prediction performance. Gu et al. [13] proposed the IQA model which performs efficiently with convolution operations at multiscale, gradient magnitude (GM), and colour information similarity, and a perceptual-based pooling. At the same time, by the guidance of the proposed NR-IQA measure, a robust image enhancement framework is established. Next year, Gu et al. [14] extracted 17 features through analysis of contrast, sharpness, brightness and more, and then yielded a measure of visual quality using a regression module, which is learned with big-data training samples that are much bigger than the size of relevant image data sets. These algorithms based on the features of the image itself and achieved good results. However, these algorithms are not suitable for the enhanced X-ray, dust, underwater and low-light. Aim at these types of images, we only consider the colour information except texture information. The above mentioned methods are too complicated for these images. In addition, the convolution neural network was first used to extract the features of different distortion types by Bianco et al. [15]. Liu et al. [16] learned the ranking features from different quality images through siamese network. Chen et al. [17] realised an NR-IQA algorithm through multitask network which learned the quality information from image regions. Bare et al. [18] used residual network knowledge to achieve NR image quality evaluation. Ren et al. [19] proposed restorative adversarial net, a GAN-based model for NR-IQA, which mimics the process of HVS, consists of three components: a restorator, a discriminator and an evaluator. Ma et al. [20] proposed a deep bilinear model to detect the quality of synthetically distorted images and real distorted images. Jiang et al. [21] devised a novel quadratic optimised model based on the DCNN (QODCNN) for FR and NR screen content image quality assessment. Gu et al. [22] developed an NR-IQA model for accessing the perceptual quality of screen content pictures with big data learning. Compared with the previous algorithms, these algorithms greatly improve the accuracy of IQA, but the amount of computation in deep learning is staggering. A large amount of training data is the basis of these algorithms. On small sample data, such as the low-quality images which have the colour inharmonious problem, it is difficult to provide enough training data for deep learning. In fact, with the development of IQA, more and more scholars began to evaluate the quality of specific images (enhanced images, pansharpened images) and specific features (sharpness, contrast). Gu et al. [23] designed an automatic parameter contrast enhancement technology with a complete histogram modification framework and an automatic parameter selector. This framework combines the original image, its histogram equalised product, and its visually pleasing version created by a sigmoid transfer function. Zhou et al. [24] proposed an NR-IQA method to blindly predict the quality of pansharpened images via opinion-unaware learning. For image sharpness, Gu et al. [25] proposed a blind sharpness metric in the autoregressive (AR) parameter which is established via the analysis of AR model parameters space. In addition to brightness, the algorithm further considers the effect of colour information on image sharpness. In 2017, Gu et al. [26] devised a new blind metric for IQA of contrast distortion. This model generates an overall quality estimation of a contrast-distorted image by properly combining local and global considerations. However, these algorithms achieved good results for specific problems, but there are still many limitations on IQA of the enhanced X-ray, dust, underwater and low-light images. Through deeply researching the existing algorithms of IQA, NR-IQA algorithms supplied at present cannot satisfy what we need. Hence, we propose an enhanced image NR quality assessment based on colour space distribution. By transferring the harmony and richness colour from high-quality images to low-quality images, this algorithm constructs a reference image. And then we compare the reference image with the enhanced image through the image evaluation index, so as to obtain the final quality assessment result. 3 Proposed method The main block of the proposed method is shown in Fig. 2 that is divided into two parts: construct reference image and quality assessment. First, we use target image and colour transfer to correct the inharmonious colour in low-quality images, meanwhile, obtain the reference image. Next, the right quality assessment index is chosen to compare the enhanced image and reference image. Finally, we get the enhanced IQA result. Fig. 2Open in figure viewerPowerPoint Block of enhanced image NR quality assessment based on colour space distribution 3.1 Colour transfer model Reinhard et al. [27] proposed a method about colour transfer between two different images. The method effectively transfers the target image colour to the input image in CIELab colour space. This method is divided into three steps: Step 1: In order to remove the correlation between different colour channels, we convert the target image from RGB colour space to CIELab colour space. Step 2: We calculate the mean and variance of each colour channel in the target image. Continue doing this work for the input image. Mean represents images detail features and variance represents the hue information of the image. Step 3: According to (1), we restore the input image (1)In this model, represents three colour channels. is the k channel of the input image, is the k channel of the target image, and are the variance of the k channel of the input image and target image, respectively, and are the mean of the k channel of the input image and target image, respectively. The experimental results are shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Experimental results of colour transfer (a) The input image, target image and image after colour transfer from left to right, (b) The RGB curves corresponding to the first line, (c) The grey images corresponding to the first line, (d) The grey curves corresponding to the third line Taking dust image as an example, we use colour transfer [27] to transfer colour features in colour and grey images. As shown in Fig. 3, the dust image has colour cast and low-light problems inevitably. In the RGB curve, the R, G, B channels of dust images are separated, which are disproportionate to each other and have a poor correlation. After colour transfer, the trouble with colour cast of dust image is solved. At the same time, the R,G,B channels are basically proportional, the correlation is enhanced, the subjective quality of the image is improved. The grey level range of the image after colour transfer also increases in the grey curve. The brightness of the image has been improved obviously. Then again, target image has similar quality and colour distribution with the hypothetical reference image in normal people's eyes. Experimental results show that the colour distribution of the low-quality image can be changed effectively by colour transfer, so as to improve the image quality. Hence, we can use colour transfer between input low-quality image and the target image to get the reference image which has a similar scene, colour and quality with the hypothetical reference image. It means the image after colour transfer can be taken as a reference image in subsequent processing. 3.2 Construct reference image The reference image can be constructed through transferring the colour distribution from the target image to the low-quality image. The colour distribution of the image after colour transfer gets close to the hypothetical reference image. Then, the image after colour transfer can be taken as a reference in IQA. Therefore, how to select the target image for colour transfer is a key issue. The images with similar scenes have similar colour distribution. Hence, we use scene gist feature to select the target image which has the highest similarity scene with the input image in the clear images database [28]; then, the colour transfer algorithm is used to transfer the colour information in target image to the input image and get the reference image. The main frame of the algorithm in this section is shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Reference image is constructed based on gist 3.3 Quality assessment algorithm In view of the difference between colour and grey image, choosing a reasonable quality assessment algorithm is helpful to get accurately IQA results. 3.3.1 Absolute colour difference In order to enhance dust, underwater and low-light images, we often enhance colour features, like brightness, contrast and colour, instead of texture features. At present, the mean difference is used to measure the image colour similarity. To a certain extent, this algorithm can reflect image colour similarity. However, in different pixel positions, both positive and negative values are found in the colour differences. This problem reduced the accuracy of mean difference. Further, this method needs reference image to calculate. Therefore, we propose absolute colour difference (ACD) to measure image colour similarity. This method calculates absolute difference value in each colour channel between the enhanced image and the reference image. Then, we get ACD value after calculating the mean of the absolute value. The calculated equation of the algorithm is shown in (2). (2)where M is the absolute colour difference, c represents R, G, B three channels of images, is the original image, is the result image. It can be seen from (2) that a smaller ACD represents a smaller colour difference between two images. It also means the smaller the ACD is, the better the enhanced image quality is. In order to verify the objective of ACD, we use photoshop software to add a yellow mask to Fig. 5a. The opaque degrees of masks are 30, 60, and 90%, respectively. Then, the ACD is used to calculate the colour similarity between these images with the original image. The results are shown in Table 1. From this experiment, with the increase of mask opaque degree, the ACD also increases, and the colour difference between original image and mask image becomes greater. Hence, ACD can effectively evaluate image colour similarity. Table 1. ACD results of Fig. 5a and opacity masks images Image Fig. 5b Fig. 5c Fig. 5d ACD 23.67 46.00 67.67 Fig. 5Open in figure viewerPowerPoint Images with different levels of mask (a) Original image, (b) 30% opacity mask image, (c) 60% opacity mask image, (d) 90% opacity mask image 3.3.2 FSIM Different with colour images, the X-ray images do not have clear image database. So, we use bidirectional mapping to construct the reference images which is described in Section 4.1. In this processing, the structure of the reference images will be changed. Therefore, a quality assessment algorithm which takes brightness and texture into account can get better IQA results. FSIM [3] is an FR-IQA method. The main parameters about FSIM are phase congruency and GM. Based on these parameters, FSIM can evaluate image structure and colour quality. Hence, FSIM can help us get the enhanced X-ray image quality result. Furthermore, the pseudo code of the algorithm proposed in this paper is shown as follows Algorithm 1 (see Fig. 6). Fig. 6Open in figure viewerPowerPoint Algorithm 1: Enhanced image NR quality assessment based on colour space distribution 4 Experimental results and analysis In order to verify the effectiveness and reliability of the proposed algorithm, we devise a series of experiments on low-quality images that have the colour inharmonious problem. First, we evaluate the quality of the enhanced X-ray images to prove the effectiveness of the proposed algorithm for grey enhanced images quality assessment. Secondly, the reliability of the reference image in the proposed algorithm is verified through the fake dust image. Then, utilising the enhanced underwater image to prove the proposed algorithm can effectively achieve the quality assessment of the colour-enhanced image. Finally, considering that the constructed reference images are not always consistent with the hypothetical reference images, we use the low-light image to illustrate that the proposed algorithm still has good performance. 4.1 Quality assessment results of X-ray enhanced images On account of the particularity of X-ray image structure, it is extremely difficult to find clear images that have a similar structure with input image. Therefore, we use bidirectional mapping to construct two different reference images and compare with the input images, respectively. Next, the two comparison results are confirmed with each other and get the final results. The processing of reference image constructed can be divided into 'transfer after enhance' and 'enhance after transfer'. Specifically, the details are summarised as follows. 4.1.1 'Transfer after enhance' experiment In 'transfer after enhance' experiment, we choose a nature image as input, the enhanced X-ray image as target to achieve colour transfer. The main frame is shown in Fig. 7. First, we use histogram equalisation method (HEM) [29], parametric indices of fuzziness (PIFM) [30], maximum Shannon entropy method (MSNM) [31] and Li et al. [32] to enhance low quality X-ray images. Next, colour transfer is used to implement grey mapping for enhanced images. Finally, we use FSIM [3], MSE [4], PSNR [5], SSIM [6], RIQMC [8], blind image quality measure of enhanced images (BIQME) [14], discriminable image pairs image quality (dipIQ) [33], integrated local natural image quality evaluation (IL-NIQE) [34] and perceptual similarity (PSIM) [13] to get the image quality. The experimental results are shown in Fig. 8. After grey mapping, we get four images through 'transfer after enhance' that are shown in line 3 of Fig. 8. Next, we measure these images quality through the above IQA methods, the objective results are shown in Table 2. Table 2. 'Village+Neck' image quality objective results Method HEM [29] PIFM [30] MSNM [31] Li et al. [32] MSE [4] 11,642.9988 11,688.9443 4213.4411 1477.3304 PSNR [5] 7.4702 7.4531 11.8844 16.4360 SSIM [6] 0.6238 0.4758 0.7098 0.7879 FSIM [3] 0.7726 0.8069 0.8109 0.8428 RIQMC [8] 5.5374 5.4707 3.3640 7.9729 BIQME [14] 0.3256 0.3208 0.4995 0.5920 dipIQ [33] 15.1733 16.4851 14.8816 14.6976 IL-NIQE [34] 66.0671 66.1634 69.1971 61.7997 PSIM [13] 0.9938 0.9937 0.9971 0.9972 Bold values indicate the best results. Fig. 7Open in figure viewerPowerPoint Reference images is constructed based on 'transfer after enhance' Fig. 8Open in figure viewerPowerPoint 'Village+Neck' experimental results of 'transfer after enhance' From objective IQA results, most of the IQA methods consider the enhanced image which is obtained through Li et al. [32] is better. In order to further verify the consistency between experimental results and subjective evaluation results, we ask 20 people unfamiliar with image processing to evaluate the quality of these images. After scrambling the order of the images, these persons rate the images quality on a scale of 0 to 10. The highest image quality means 10, and the worst image quality means 0. The subjective results are shown in Fig. 9. The ordinate represents the opinion scores of 20 evaluators, and the abscissa is the number of persons. Same with the objective results, the enhanced images which are obtained through Li et al. [32] get the highest scores, in most of the compared IQA methods. These enhanced images subjective scores are shown by the yellow line in Fig. 9. Fig. 9Open in figure viewerPowerPoint 'Village+Neck' experimental subjective scores of 'transfer after enhance' Figs. 10 and 11 are other experimental results. To measure the effect of light intensity on the quality evaluation, an indoor portrait is selected as the nature image. Table 3 shows the image quality objective comparison results of each algorithm. The subjective results are obtained through the method in Fig. 9. The experimental results confirmed that the accuracy of the algorithm is not affected by different light intensity. It means the proposed method is common to all images which are under different luminance. This algorithm is verified from subjective visual effects and objective IQA, experimental results show that the enhanced images which are obtained through Li et al. [32] get the highest quality. Table 3. 'Arabian woman+Vertebra' image quality objective results Method HEM [29] PIFM [30] MSNM [31] Li et al. [32] MSE [4] 1429.2915 5919.1637 1762.537 864.3215 PSNR [5] 11.9056 10.4082 15.6694 18.7641 SSIM [6] 0.7316 0.6815 0.7736 0.8159 FSIM [3] 0.8310 0.8609 0.8299 0.8796 RIQMC [8] 4.8956 4.6271 0.7644 4.1638 BIQME [14] 0.6187 0.4260 0.5917 0.6313 dipIQ [33] 0.8622 1.5117 0.4044 0.6413 IL-NIQE [34] 101.4497 99.6318 110.9020 121.2315 PSIM [13] 0.9995 0.9981 0.9988 0.9989 Bold values indicate the best results. Fig. 10Open in figure viewerPowerPoint 'Arabian woman+Vertebra' experimental subjective scores of 'transfer after enhance' Fig. 11Open in figure viewerPowerPoint 'Arabian woman+Vertebra' experimental subjective scores of 'transfer after enhance' After different 'transfer after enhance' experiments, both subjective and objective evaluation demonstrate Li et al. [32] is the better enhancement algorithm than other comparison method for X-ray images. Next, we use 'enhance after transfer' experiment to illustrate the effectiveness of the proposed algorithm. 4.1.2 'Enhance after transfer' experiment In 'enhance after transfer' experiment, we first use colour transfer to realise grey mapping for input nature images. Next, the images after colour transfer are enhanced through HEM [29], PIFM [30], MSNM [31] and Li et al. [32]. The main frame is shown in Fig. 12. Fig. 12Open in figure viewerPowerPoint Reference images is constructed based on 'enhance after transfer' The experimental results are shown in Fig. 13. In this experiment, we still choose the natural image in Fig. 8 as the input image. Table 4 and Fig. 14 are the objective and subjective quality assessment results. The subjective results are obtained through the method in Fig. 9. Table 4. 'Village+Neck' image quality objective results Method HEM [29] PIFM [30] MSNM [31] Li et al. [32] MSE [4] 3203.7977 1734.6501 4266.0675 1374.6149 PSNR [5] 13.0742 15.7387 11.8305 16.7490 SSIM [6] 0.6022 0.7026 0.5437 0.7508 FSIM [3] 0.7143 0.7590 0.7410 0.7630 RIQMC [8] 8.8185 14.5777 5.028