Image quality assessment via spatial‐transformed domains multi‐feature fusion

IET Image ProcessingVolume 14, Issue 4 p. 648-657 Research ArticleFree Access Image quality assessment via spatial-transformed domains multi-feature fusion Miaomiao Yu, Miaomiao Yu Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorYuanlin Zheng, Corresponding Author Yuanlin Zheng zhengyuanlin@xaut.edu.cn Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorKaiyang Liao, Kaiyang Liao Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorZhisen Tang, Zhisen Tang Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this author Miaomiao Yu, Miaomiao Yu Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorYuanlin Zheng, Corresponding Author Yuanlin Zheng zhengyuanlin@xaut.edu.cn Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorKaiyang Liao, Kaiyang Liao Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this authorZhisen Tang, Zhisen Tang Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an, 710048 People's Republic of China Key Lab of Printing and Packaging Engineering of Shanxi Province, Xi'an, 710048 People's Republic of ChinaSearch for more papers by this author First published: 18 February 2020 https://doi.org/10.1049/iet-ipr.2018.6417Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract The basis of image processing is to evaluate and monitor image quality using algorithms rather than subjective methods. Conventional gradient operators have been popularly used in previous image quality assessment tasks to reflect the edge contour of an image, while there are some obvious defects in terms of the selection of mask scale and direction. Some improved versions are also less than ideal since they fail to consider the gradient information of the same pixel in different directions at the same time. The authors adopt a powerful gradient operator that can simultaneously capture edge information in all four directions at the same pixel point with more relevant values being considered instead of selecting the maximum in these four directions. Furthermore, four complementary types of features extracted from the spatial and transform domains are considered. A set of 12-dimensional feature vectors is generated for each image by multi-feature fusion. Ultimately, random forest regression technique is employed to train their model and then map the distortion effects to the prediction scores. The experimental results show that the proposed FVC-G has better overall performance, more powerful cross-database operation capability, and higher visual consistency than other advanced methods. 1 Introduction The intention of image quality assessment (IQA) is to quantify human's visual perception of image's quality, which can provide valuable theoretical guidance of image encoding compression, parameters optimisation of image processing algorithms and even video quality monitoring of user terminals [1]. The IQA tasks can generally be classified into subjective evaluation and objective evaluation. The former method is not applicable to automation systems or any other real-time ones since the evaluation process needs to be done manually. Conversely, the objective evaluation method is to establish the visual model by computers and simulate the information processing process of human visual system (HVS) so as to realise the objective evaluation of image quality. This type of method has been widely used because it not only inherits the advantages of subjective ones but also compensates for their shortcomings. According to the degree of dependence on the reference image information, the objective evaluation method can be further divided into full-reference IQA (FR-IQA), reduced-reference IQA and no-reference IQA (NR-IQA). The first two types can utilise all or part of the information in a reference image, whereas the last one has no access to any information for reference. Since no pristine information is available, the design of the NR-IQA method has become more complicated and its prediction performance is normally worse than that of FR-IQA metric. Thus, in this study, we continue to explore a novel FR-IQA method in an effort to achieve new breakthroughs in terms of prediction accuracy. The existing FR-IQA metrics are classified into traditional methods and learning-based methods. The earlier FR-IQA metrics, represented by the peak signal-to-noise ratio and the mean square error (MSE), are designed to evaluate image quality by calculating pixel-level differences between a distorted image and the corresponding pristine version. However, these methods assume that the pixels are independent of each other, which is inconsistent with the perception mechanism of HVS (HVS are more sensitive to the structural information of a visual scene than a single pixel [2]). Subsequently, structural similarity (SSIM) [3], the cornerstone for promoting the development of IQA studies, plays a pivotal role in this subject. Most of the methods, such as multi-scale SSIM (MS-SSIM) [4], information-content weighted SSIM (IW-SSIM) [5], and gradient similarity metric (GSM) [6], are the extended version of SSIM, which get further improvements and can be better aligned with human visual perception. Furthermore, many IQA models tend to extract low-level features and design image quality prediction function to simulate the function of the HVS from various aspects, e.g. the visual saliency (VS), which extracts significant areas of an image by simulating human visual characteristics, and the contrast sensitivity function (CSF), which reflects the ability of the HVS to distinguish objects with blurred boundaries. In [7], the author extracted phase congruency and gradient features and chose the former one as the weighting function to emphasise its importance in the whole image. The VS-based index (VSI) [8] adds VS feature and considers image's edge feature meanwhile. The VS here is used as one of the features that characterise the local quality of an image, and also as a weighting function in the pooling stage to emphasis on the importance of local region in the whole image. Another VS-based method called VSM [9], an improved version of VSI, has made some improvements in the calculation of VS and gradient features and shows better quality prediction performance. The gradient magnitude similarity deviation (GMSD) [10] measures the gradient variation of the local quality maps, and the global quality score is obtained by adopting the standard deviation pooling strategy. To simulate the behaviour of HVS, the structural contrast-quality index (SC-QI) [11] combines structural contrast index, a metric which is used to assess the perceptual complexity of image texture patterns, with CSF to achieve the perfect representation of the visual perception of local image feature and various types of distortion. In addition, many methods [12-17] that are designed for texture recognition have achieved great success in the field of computer vision. Although most of them are dedicated to synthetic aperture radar images, apparently, they are quite effective for texture recognition in any type of image as well as in IQA tasks. Specifically, the method proposed in [12] is the most well-known one, which regards wavelet transform as the tool of extracting texture feature and then uses feature vector and level set function to segment different texture regions. More importantly, the study of learning-based IQA algorithms has become increasingly popular, which aims to train the model by merging representative image features and mapping the feature vectors into perceived scores through an appropriate regression strategy. Currently, many regression techniques have been developed, such as support vector regression (SVR) [18], random forest (RF) [19], and so on. Pei and Chen [20] built an RF-based model by extracting image features in different Gaussian bands, then mapping the feature vectors to the perceived scores through a trained regression model to simulate the brain mechanisms. The proposed difference of Gaussian (DOG) model can be combined with other existing algorithms to significantly improve their performance. Combined multiple object image quality models, such as context-dependent multi-method fusion (CD-MMF) [21], use SVR to evaluate image quality. Furthermore, singular value decomposition (SVD)-based [22] IQA metric also obtains good experimental results. The SVR tool is used in [23] to train the model and for quality mapping. The local linear model (LLM) [24] obtains higher prediction accuracy than other metrics on some databases by using a distortion-specific compensation strategy. Different from the existing regression-based approaches, blind image quality assessment via deep learning (DLIQA-R) [25] adopts a classification framework, which is robust to small sample size problems and has been widely used in current IQA system. Many recent studies [26-28] have shown that the learning-based IQA approach represented by a convolutional neural network (CNN) also shows superior predictive capability over traditional methods. The deep similarity metric (DeepSim) [29] applies the visual geometry group net model trained in the task of image classification to the FR-IQA framework and can be regarded as a true deep neural network architecture. To some extent, the performance of IQA metrics improves with the depth of the neural network model, whereas the increase of parameters will obviously affect the complexity of the algorithm. According to the summarisation of previous studies, one can see that the gradient operator is widely used in IQA to describe structural information that can well reflect the feature of the sharply transformed edge portions of an image. The kernels used in traditional gradient operators are too small to contain enough adjacent information between pixels, and only the horizontal and vertical gradient features are considered. Therefore, the description of edge details inevitably lacks accuracy, and the correlation with the HVS's perception mechanism needs to be further improved. Later, high-pass filter convolutional masks with a scale of 5 × 5 are used to extract edge figures in four directions of 0°, 45°, 90°, and 135°. However, this operation captures only the maximum of the four directions at the same pixel point, resulting in insufficient information in the other three directions. To address the above problems, we take the gradient information of a pixel in four directions into consideration at the same time, which can maximise the adjacent relationship between the centre pixel and its surrounding pixels. Our previous research [30] has found that extracting features in both spatial and frequency domains are quite effective for IQA tasks. In that paper, a nine-dimensional (9D) feature vector is generated for each image and then RF is adopted to learn a mapping from the feature vectors to quality scores. Although such method is obviously superior to other state-of-the-art models since it attains the best results on average comparison; however, it is not sensible to extract gradient feature using traditional way which is difficult to capture more adjacent information between pixels, and there is still a lot of room for improvement in its prediction performance on IQA datasets, especially on the LIVE [31] database. Consequently, we want to follow the idea of extracting features from spatial and transformed domains and build a more robust and stable model with better generalisation capabilities. In this study, a learning-based FR-IQA framework is proposed, the main contributions of which are as follows. First, the gradient features of 0°–90° and 45°–135° are superimposed to simultaneously capture the edge information in all four directions at the same pixel point instead of selecting the maximum value from four directions, which can maximise the adjacent relationship between the centre pixel and its surrounding pixels. Second, four complementary features extracted from spatial and transformed domains are used to represent the whole information of an image. The experimental results show that the proposed framework is high in prediction accuracy, which indicates that these combined features are valid enough to maximise the representation of its original image. The method of multi-feature fusion can largely overcome the common limitation of most existing FR-IQA metrics in depicting image quality only in spatial or transformed domains. Third, the generated easy-to-manage feature vectors and the corresponding subjective mean opinion scores (MOS) or differential MOS (DMOS) are used as input of the RF regression technique to train our model and ultimately mapped to objective scores. The limitation of conventional pooling is that summation or multiplication operation may result in a linear relationship between the distortions and prediction score, which can be overcome by an RF-based pooling strategy. Compared with other advanced models, our proposed approach has a good correlation with subjective perception, better prediction performance, and more robust cross-database capability. The rest of this paper is arranged as follows: Section 2 briefly reviews the feature extracted steps from spatial and transformed domains and summarises various pooling strategies; Section 3 describes the details of each process of our proposed method; Section 4 presents experimental setups and experimental results; and Section 5 is a summary of the present paper. 2 Related work FR-IQA methods consist of feature extraction and pooling strategy. This section introduces the studies most relevant to our study and briefly reviews the image gradient and VS features extracted from the spatial domain, the other visual perception-related features extracted from the transformed domain, as well as their respective existing applications in IQA. Then, on the basis of previous research studies, various pooling strategies are summarised. 2.1 Extraction of feature in the spatial domain Since digital images are represented by 2D digital groups, in many cases, the extraction of the representative feature that contains distorted information in the spatial domain tends to focus more on the relationship between pixels. When observing an image of a real scene, the human eyes seem to be more interested in object-like regions; hence, the distortion of these regions is more noticeable than that of the background ones. In general, transformation of pixel values in the background areas is relatively smooth, so it is easier to distinguish the background areas from the object-like ones by measuring whether the changes of edge pixels are drastic. As the place where the variation of greyscale values is the most significant and dramatic, the edge is considered to be the most important image feature in spatial domain. For IQA, image gradient magnitude, which is sensitive to structural distortion, is the most commonly used index for describing the edge features of an image. The conventional gradient extraction methods include the Sobel, Prewitt, and Scharr operator. What they have in common is the use of two filtered templates to calculate the horizontal and vertical directions partial derivatives separately, which we denote here as and , respectively. Then the gradient feature is obtained as (1)Although these conventional gradient operators have been popularly used in existing IQA methods, there are still some defects in terms of the selection of mask scale and direction. Different from previous approaches [7, 8, 10], the methods proposed in [6, 9, 32] make an improvement by using four larger gradient masks to convolve the image in four directions and then choosing the maximum of these four directions as the final gradient feature. Unfortunately, although the above methods enlarge the mask scale and direction, they are also less than ideal since they fail to consider the gradient information in different directions at the same pixel. Specifically, it is not the best choice to select the maximum of the four directions as the extracted gradient feature, because the gradient figures in other directions are also important for the formation of edge, thus leading to insufficient accuracy of the described gradient information. Therefore, in our proposed framework, further improvements based on the above gradient algorithms are made to simultaneously capture gradient information in all directions at the same pixel so that the edge of an image can be described more comprehensively. One thing all primates have in common when performing advanced visual processing is that they often process a sub-region of the image in depth, which reduces the complexity of the real scene. Such a characteristic can be represented by VS models. Since both IQA and VS are related to HVS, it is appropriate to integrate VS model into the IQA algorithms to effectively improve their prediction performance. Many existing VS models [33-35] assume that visual attention is driven by low-level visual stimuli in the scene, such as intensity, direction, and colour. VSM [9] processes each image patch and introduces a novel VS model by fusing pixel-level dissimilarity, central bias, and spatial distance. The Fourier transform-based method [36] is designed to extract spectral residuals of the image in the spectral domain and establish corresponding saliency maps in the spatial domain. The model (here we named it SO-RB [37]) uses boundary connectivity background measure and principled optimisation framework to detect salient object, which provides cleaner and more uniform saliency maps than most advanced VS models. Moreover, the saliency detection method by combining simple priors (SPSD) [38] is the most classic and widely used salient object detector, which effectively extracts the salient regions of the image. Based on the cognition of the high correlation between the VS model and HVS, the VS feature of the image is integrated into our approach. In addition, owning to the fact that the variation of contrast cannot be comprehensively reflected on VS maps, the combination and supplement of the gradient and VS features become more feasible so that the image can be better described. Most of the currently used digital images are coloured, so it is advisable to consider the chromatic feature when designing an IQA algorithm, which is more accurate for predicting colour distorted images. 2.2 Extraction of features in the transformed domain Since most of the energy of an image is concentrated in the low-frequency (LF) regions, it is not enough to well describe the whole image by only extracting the representative feature in the spatial domain. It is significantly indispensable for the IQA task to convert spatial domain of an image into transformed domain to extract more useful features. The model in [39] calculates the correlation among each sub-band in the discrete cosine transform (DCT) domain and then applies the Gaussian weighted function for each sub-band to simulate the important functions of HVS. It is true that most of the image encoders tend to deal with each spatial frequency separately, thus leading to a wide range of distortion intensity in different frequency sub-bands, so all distortion information can be captured efficiently by calculating the similarity of each DCT sub-band separately. SC-QI computes the similarity of the LF, medium-frequency (MF) and high frequency (HF) regions from each DCT block from the image so as to simulate the CSF of HVS. Moreover, DOG-SSIM [20] extracts feature from different Gaussian bands decomposed by the DOG model. 2.3 Pooling strategy When it comes to traditional FR-IQA methods, each metric would generate representative feature vectors after the step of feature extraction. Since low-dimensional vector space is easy to manage, the pooling process is extremely necessary and pretty important. Pooling can map high-dimensional vector (usually refers to the similarity of the extracted features) into a single quality score. The pooling methods for IQA can generally be divided into six types, including average pooling, weighted average pooling, percentile pooling, standard deviation pooling, mean absolute deviation pooling, and full deviation pooling. Many state-of-the-art IQA metrics [3, 4, 6, 7] are based on the first two pooling strategies. However, a recent study [10] shows that because average pooling does not fully consider the spread of distortion, it only has good prediction accuracy for one distortion type of image, but fails to perform well on the entire dataset. The percentile pooling strategy is used in [40], and this strategy can adjust the degree of distortion or discard less or undistorted pixels. In [6, 10], standard deviation pooling is applied to obtain the final quality scores, which can not only reduce the complexity of the algorithm but also achieve better performance. The mean absolute deviation pooling strategy obtains the objective scores by calculating the average absolute difference between each quality prediction value and average value. The full deviation pooling method intelligently combines standard deviation pooling and mean absolute deviation pooling strategy with a ratio control value and shows superior results. The specific calculation processes of several pooling strategies are introduced in detail in [29]. Lately, an increasing number of researchers tend to explore the learning-based IQA methods, which definitely outperform most traditional pooling-based methods. The feature extraction step of this type of method is followed by the process of training hand-crafted features using a suitable regression technique. Narwaria and Lin [22] used the distortion of a singular vector pair out of SVD as the pooled image feature. The MMF model [21] is constructed to further improve the predictability of images of all distortion types by SVR. In terms of computational complexity, many additional parameters need to be tuned when SVR and CNN are adopted, resulting in high-computational cost. Conversely, RF is regarded as one of the most ideal tools in the application of deducing mathematical functions as well as quantitative relationships between distortion components and image quality. It is worth mentioning that compared with other tools, RF and SVR both show outstanding precise mapping capability in IQA tasks [19-21, 23, 30]. More importantly, Pei and Chen [20] used the RF regression strategy to train six DOG models and the results showed that for any DOG model, better and more accurate quality prediction results could be achieved by using RF as a regression tool than SVR. In fact, one regression technique may work for a particular data set yet fails the other data sets. In order to avoid this problem, the prediction results of these two tools are compared in the experimental part of Section 4.4, which exactly proves that the RF-based approach greatly outperforms the SVR-based method in any dataset and has better generalisation. Based on the above analysis, we decide to adopt RF tool in our method. Theoretically, the benefits of using RF is that it has only two parameters and the internal parallel regression tree structure can greatly speed up the calculation. 3 Proposed approach In this section, the specific process of our proposed approach called FVC-G will be elaborated on. The framework is shown in Fig. 1. Firstly, four complementary features are extracted separately from each image, including saliency, gradient, contrast sensitivity-related, and chromatic features. Then, four similarity maps are calculated for each distorted version. Subsequently, the pooling strategy is used to extract the mean, standard deviation, and entropy of the similarity maps that will generate a 3D feature vector for each similarity map. That is to say, if a dataset contains M distorted images, each type similarity map will generate an (M × 3)-D feature vector. Multi-feature fusion merges the four pooled feature vectors of each distorted image to generate a 12D feature vector. Hence, the (M × 12)-D feature vectors representing the images in the whole dataset are obtained. Finally, the feature vectors and subjective quality scores of corresponding images are trained through the RF regression tool so that the extracted image features from the test set can be returned to a quality score. Fig. 1Open in figure viewerPowerPoint General framework of our approach 3.1 Saliency feature extraction By simulating the processing mechanism of HVS, saliency detection can intelligently find an image's local regions that human eyes are interested in. Since the research of IQA tasks needs to design HVS-related algorithms to replace time-consuming and labour-intensive subjective evaluation methods, a large number of metrics have adopted the VS feature to achieve the desired effect. The more persuasive reason for extracting VS feature in IQA tasks is that the visual system is assumed to be more sensitive to the distortion of the regions of interest (ROI) than the other areas. Hence, the common practice of many IQA metrics in pooling strategy is to select the local saliency maps as a weighting function to emphasise the importance of ROI in global image. In VSI, eight mainstream VS models are compared on the TID2008 [41] database. The results show that the SPSD saliency detection model is significantly superior to others. So, in our method, we also adopt the SPSD model to extract salient information from the image. In fact, we have tried to combine the SPSD with another well-built VS model SO-RB in our method in order to obtain a more powerful feature vector for each image. However, as shown in the results shown in the subsequent experimental, this fused method seems to fail to achieve ideal performance. The changes in image quality that the human eyes can perceive are reflected in the VS map. An example is shown in Fig. 2, which reveals the relationship between the distorted images and their corresponding changes in the VS maps. The images used in Fig. 2 are from the TID2013 [42] database and Fig. 2a is one of its reference images, while Figs. 2b and c are two distorted images of Fig. 2a. Figs. 2b and c are images with the same distortion level but different distortion types. Figs. 2d–f are VS maps corresponding to Figs. 2a–c, respectively, calculated by the SPSD model. Obviously, the subjectively perceived quality of Figs. 2b and c is gradually declining (the subjective scores are 4.2571 and 2.2353, respectively), which is accurately reflected in their VS maps, with Fig. 2f showing the largest variation, followed by Fig. 2e. The above phenomenon is fully consistent with HVS's visual perception of image quality. Fig. 2b contains additive Gaussian noise points, which can be detected in its corresponding VS map Fig. 2e. In addition, there are local distorted patches in the background region of Fig. 2c, which can greatly affect the visual perception of the image. Meanwhile, these distorted patches, which attract the attention of human eyes are also detected in the corresponding regions of its VS map. From the above analysis, it can be seen that the VS map obtained by the SPSD model can well reflect image changes of different distortion types. Hence, it is sensible to utilise the SPSD model to extract VS feature as one of the indicators for describing the variation of local image quality. Fig. 2Open in figure viewerPowerPoint Relationship between distorted images and their corresponding changes in the VS maps (a) Reference image, (b) Distortedimage 1, (c) Distorted image 2,(d) VS map of Fig. 2a,(e) VS map of Fig. 2b,(f) VS map of Fig. 2c When the VS maps of the reference image and its distorted version are obtained, denoted here by and , respectively, the method for ca