Stereoscopic image quality assessment by analysing visual hierarchical structures and binocular effects

IET Image ProcessingVolume 13, Issue 10 p. 1608-1615 Research ArticleFree Access Stereoscopic image quality assessment by analysing visual hierarchical structures and binocular effects Yong Ding, Yong Ding College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorYang Zhao, Yang Zhao College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorXiaodong Chen, Xiaodong Chen The 14th Research Institute of China Electronic Technology Group Corporation, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaolei Zhu, Corresponding Author Xiaolei Zhu zhuxl@vlsi.zju.edu.cn College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorKrylov Andrey, Krylov Andrey Laboratory of Mathematical Methods for Image Processing, Lomonosov Moscow State University, Moscow, RussiaSearch for more papers by this author Yong Ding, Yong Ding College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorYang Zhao, Yang Zhao College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorXiaodong Chen, Xiaodong Chen The 14th Research Institute of China Electronic Technology Group Corporation, Nanjing, People's Republic of ChinaSearch for more papers by this authorXiaolei Zhu, Corresponding Author Xiaolei Zhu zhuxl@vlsi.zju.edu.cn College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, People's Republic of ChinaSearch for more papers by this authorKrylov Andrey, Krylov Andrey Laboratory of Mathematical Methods for Image Processing, Lomonosov Moscow State University, Moscow, RussiaSearch for more papers by this author First published: 18 July 2019 https://doi.org/10.1049/iet-ipr.2018.5605Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In the design of three-dimensional image processing systems, stereoscopic image quality assessment (SIQA) plays an indispensable role as a performance evaluator and a supervisor, yet the study upon which remains immature due to the complexity of the human visual system (HVS). In this study, a novel SIQA method is proposed by extracting quality-aware image features according to the properties of the hierarchical structure in the HVS. Especially, the interests of the primary and secondary visual cortex are taken into consideration, so that the image quality representation is constructed in a way both accurate and efficient. Moreover, influences caused by binocular effects including binocular rivalry and binocular visual discomfort are accounted to further improve the performance of the proposed method. The superiority of the proposed method is validated through experiments on public databases in comparison to state-of-the-art works in terms of accuracy and robustness. 1 Introduction Stereoscopic [three-dimensional (3D)] scene acquisition, processing, and display techniques have been rapidly developing in recent years [1]. Serving as an evaluator and a monitor for the quality of 3D scenes, stereoscopic image quality assessment (SIQA) is becoming an important research field and a necessary part of the modern 3D image and video processing system. Since humans are the final receivers of visual signals, their subjective evaluation provides the golden standard for the evaluation of image quality [2]. However, subjective SIQA is not a satisfactory option for real-world applications due to the high consumption in time and labour [3]. Therefore, designing objective methods that pursue high correlation with subjective quality assessment results is the mainstream of the SIQA study [4]. Despite the importance SIQA holds for modern image processing technologies, the solution to it remains mysterious because of the complexity of the human visual system (HVS), especially the sophistication of the human binocular vision generating process [5-7]. The obstacles in the design of SIQA methods are from at least two aspects. On the one hand, constructing a mathematical model for the visual signal processing flow in visual cortex is difficult, since the current knowledge about HVS is still shallow. On the other hand, a variety of binocular effects influencing stereoscopic image quality create further difficulties for SIQA. Binocular rivalry, a major binocular effect explaining that the quality of a stereoscopic image is a complicated synthesis rather than a simple averaging of its monocular views, is considered in many previous works [8-11]. However, other important binocular effects such as binocular visual discomfort are rarely seen considered in SIQA research. Based on the above observations, a SIQA method is presented in this study, which is innovative in the following aspects. On the one hand, a feature extraction process is built to simulate visual quality perception in the HVS, which is simple to implement and effective in practice. Specifically, by analysing human visual cortex areas related to quality perception, only two levels of image features are captured for quality representation, both of which are obtained in a multi-scale manner to describe image quality more comprehensively. The extracted low- and high-level features are responsible for representing the image properties that are attractive to the primary and secondary visual cortex (V1 and V2), the two major visual cortex areas related to visual quality perception, respectively [12]. On the other hand, to better score quality for stereoscopic images, binocular effects including binocular rivalry and visual discomfort are taken into account, the usage of the latter is rarely seen in existing works [13]. Moreover, a parallel framework is organised to implement the proposed method, as shown in Fig. 1. First, three operations are conducted: extracting features of both low and high levels, computing pixel-wise weights to account for binocular rivalry, and estimating binocular disparity. By describing image quality using two levels of features, the complicated modelling of HVS processing is avoided. Also, since all the three operations require only the pixel-level information of the input images and not dependent upon any pre-computed intermediate data, they can be implemented in a parallel organisation. Second, with the obtained weights are the disparities, the monocular low- and high-level feature maps are fused into multi-scale cyclopean maps so that the impact of binocular rivalry is taken into consideration. Then, the differences in low- and high-level cyclopean feature maps between the reference and tested images are regarded as two quality indicators. Meanwhile, a more comprehensive consideration of binocular effects is achieved by approximating the visual discomfort degree using the binocular disparity map and taking it as another quality indicator. Finally, all the quality indicators are synthesised to the stereoscopic image quality score Q by a trained regression function. This parallel organisation keeps the overall computational complexity moderate, and some required intermediate data are utilised for multiple purposes to further constrain the complexity. For instances, a local energy map derived by the log Gabor filter bank serves as both a quality-aware feature and weight for cyclopean image construction, disparity map helps with both cyclopean image construction and visual discomfort estimation. Fig. 1Open in figure viewerPowerPoint Framework of the proposed SIQA method The rest of this paper is organised as follows. Section 2 introduces and analyses some knowledge about the HVS which is related to as well as provides a theoretical basis for the proposed method. Section 3 describes the proposed method in detail. Section 4 presents and discusses experimental results. Conclusions are given in Section 5. 2 Related works The properties of the HVS are always of significant importance when designing visual quality assessment methods since the HVS is the ultimate receiver of these signals and offers the evaluation standard for objective methods. Therefore, some related knowledge about HVS is introduced in this section, serving as a theoretical basis for the proposed SIQA method in this study. Specifically, the introduction focuses on two aspects. On the one hand, the inner structure of the HVS is analysed, with certain parts related to image quality perception highlighted. On the other hand, two major binocular effects, binocular rivalry and visual discomfort, are introduced, since they have a huge impact upon stereoscopic image quality perception. 2.1 Hierarchical structure of HVS As is well known, human vision begins in eyes, where luminance and chrominance information from the outer world is perceived by cone and rod cells in the retina [14]. For the further process, these simple visual perceptions are transmitted to the visual cortex, the areas in human cortex related to vision, through lateral geniculate nucleus (LGN) [15]. It is generally considered that retina, LGN, and visual cortex together consist of the HVS. Visual cortex operates highly complex mechanisms. Its complexity allows for the completion of the sophisticated visual objects, i.e. visual quality assessment, object recognition, motion tracking etc., but creates difficulties for studying its inner structure. Fortunately, extensive truths about visual cortex have been exploited in the last few decades. One of the most important findings of the HVS is its hierarchical structure, denoting that visual signals are processed layer by layer. Each layer receives signals from its previous layer, and always tends to be capable of analysing signals in higher abstraction level and completing more complex tasks compared to the latter. Constructed with such basic structures, visual cortex can be divided into three parts, in general, occipital cortex, ventral pathway, and dorsal pathway. Occipital cortex develops an early vision, while ventral and dorsal pathway, receiving a signal from occipital cortex and working in parallel, are mainly responsible for object recognition and motion detection, respectively [12]. The perception of visual signal quality is a simple task compared to higher-level ones that ventral and dorsal pathways are responsible for; therefore, studying the characteristics of early vision is the main focus of image quality assessment (IQA). Occipital cortex includes V1, V2, the following areas V3, V4, and middle temporal (MT) [16]. The signal transmitting flow within the HVS is shown in Fig. 2 in a simplified manner. As pointed out by several previous works [17, 18], V1 and V2 are mostly important for the design of IQA methods among them, not only owing to their huge sizes, but also because the quality-related properties of V3, V4 and MT do not have fundamental differences compared to V1 and V2 [19]. Thus, the proposed method is designed largely relying on the interests of V1 and V2. Fig. 2Open in figure viewerPowerPoint General signal transmitting flow of HVS 2.2 Binocular rivalry and visual discomfort It has been discovered since very early stages of SIQA study that the quality of a stereoscopic image is not always equal to the average quality of its two monocular views, especially when the image suffers from asymmetrical distortion, i.e. the distortion types or degrees of the two views are not identical. This phenomenon is commonly explained by the 'binocular rivalry' concept, denoting that when the two monocular views mismatch each other, there will be a rivalry for attention between two eyes, and the side of view carrying more visual information would be in dominance [20]. As a result of binocular rivalry, when a monocular view of a stereoscopic image is distorted by JPEG compression and the other view is undistorted, the quality of the stereoscopic image tends to be closer to the distorted view, because the compression introduces blocks that are with a large amount of information, as the right view of Fig. 3a; however, if the distorted view suffers from Gaussian blur (GB), which reduces visual information, the overall quality would be closer to the undistorted view, as the left view of Fig. 3b. The illustrated stereoscopic images are from LIVE SIQA database Phase II [8]. Fig. 3Open in figure viewerPowerPoint Illustration of asymmetrically distorted stereoscopic images The existence of binocular rivalry proves that the perceptive quality of a stereoscopic image is not a simple summation of the quality of its monocular views; instead, side effects are brought about during the generation of binocular vision. These side effects are referred to as 'binocular effects'. By quantitatively computing the influences caused by binocular rivalry, evident performance improvement has indeed been gained by recent SIQA works [21, 22]. Taking binocular rivalry into account is widely considered necessary and inevitable for modern SIQA methods. However, other important binocular effects are rarely made use of. Binocular visual discomfort is a typical example of this ignorance. In artificial 3D scenarios, the eyes of viewers are fixed on a certain distance, unlike in the reality where the focal length of eyes changes with the locations of viewed objects. A constant focal length gives viewers symptoms including eye strain, reduced visual sensitivity etc., which are referred to as visual discomfort in general [23]. Serious visual discomfort inevitably leads to unsatisfactory visual experience and affects subjective perception upon image quality. The impact of visual discomfort is mentioned in several previous works in SIQA [4, 6, 17], but very rarely taken into quantitative computation [13]. In order to construct a comprehensive representation of stereoscopic image quality, influences by binocular rivalry and visual discomfort are both taken into consideration in this study. 3 Proposed SIQA method The process of the proposed method can be roughly divided into three stages working in a parallel flow. First, a cyclopean view is constructed by the monocular views of a stereo pair, which is a classical operation to account for binocular rivalry, but implemented in a multi-scale manner so that distortions on various scales can be captured. Then, spatial frequency and local texture features are extracted and quantified with the help of the cyclopean view to building low- and high-level image quality descriptions, respectively. In parallel to the two-level feature quantification, the degree of visual discomfort is estimated to serve as another quality indicator. With all the stages finished, the generated quality indices are synthesised to the quality score of the stereoscopic image by machine learning approach. 3.1 Multi-scale cyclopean view construction As is aforementioned, the existence of binocular rivalry causes the phenomenon that the monocular view conveying more visual information becomes dominant when visual signals from two eyes mismatch each other [8, 10]. The amount of information can be approximated by the magnitudes of spatial frequency responses, which Gabor filter is widely employed to compute [11]. In practice, the Gabor filter is usually implemented by building a multi-scale multi-orientation filter bank, to capture image features from more comprehensive aspects [8, 24]. In our work, one of its modifications, log Gabor filter, is adopted, because it preserves high frequency information, and better models the simple cells in V1 with the log axis. The response of the log Gabor filter in polar form is (1)where f0 and θ0 are the central frequency and orientation of the filter, σf and σθ denote its scale and angular bandwidth, respectively. A log Gabor filter is defined with a specified scale and angular. In this work, a bank composed of 20 log Gabor filters, with four scales and four orientations, is built. On a scale s, the local energy map E is constructed by summing up the modulus of the responses from four orientations (ori = 0°, 45°, 90°, 135°) (2)where Re and Im represent the real and imaginary parts, respectively. With the estimated local energy, the binocular rivalry is considered in a pixel-wise manner, assisted by binocular disparity calculated using a structural-similarity based disparity estimation scheme [8]. Elements in a disparity map D denote the position shift of corresponding pixels between the left and right views. In many previous works, two disparity maps are separately derived with the tested image pair and its reference; however, binocular disparity is estimated using only the reference in this study, because distortions would decrease the precision of the estimate. Denoting the feature maps of left and right views as FL and FR, the consideration on binocular rivalry is processed by determining weights for each image pixel. The cyclopean feature map C is computed as (3)where dx,y is the element of D representing pixel position shift, i.e. dx,y = D(x, y). WL and WR are weights assigned to the two pixels of a matched pixel pair that are measured by local energy but normalised so that the sum of the weights equals 1, i.e. elements in the left weight map is (4)where subscripts L and R denote the left and right views. WR is derived from ER in a similar way. 3.2 Spatial frequency feature extraction and quantification In order to fulfil the quality assessing target for stereoscopic images, quality-aware image features, i.e. features that are related to image quality perception, are extracted and quantified. Since V1 and V2 are mainly responsible for subjective visual signal quality sensation, in this study, it is attempted to capture image quality-aware features that are attractive to the two areas according to their preferences. More specifically, V1, the first processor in visual cortex and occupying the largest area of visual cortex, is interested in low-level features such as edges and spatial frequencies, while V2 focuses on features of both low-level and relatively higher level such as shapes and textures [12]. Local energy maps derived from the pre-built log Gabor filter banks are again utilised as the low-level feature maps since they reflect the spatial frequencies that V1 takes interests in. In other words, in the proposed SIQA method, log Gabor responses are employed both as binocular rivalry estimators and quality-aware features. For a 2D image taken from the LIVE 2D IQA database [25], the magnitudes of its log Gabor responses through the constructed four-scale four-orientation filter bank are shown in Fig. 4. As can be observed, filters of different scales or orientations outcome different results that give compensative information for each other, demonstrating the importance to develop a multi-scale multi-orientation filter bank. Fig. 4Open in figure viewerPowerPoint Log Gabor responses of a 2D image On every single scale, the log Gabor response from the four orientations is synthesised to one local energy map using the same strategy as in the binocular rivalry estimation process. Therefore, for scale s, the monocular energy maps EL and ER are regarded as monocular feature maps (FL and FR) and fused to cyclopean maps CE. The difference between cyclopean local energy feature maps of a stereoscopic image and its reference is used as the first quality indicator, measured by chi-square distance (5)where superscripts t and r denote the tested and reference image, subscripts i and ave denote the ith item and the average, respectively. 3.3 Local texture feature extraction and quantification Local textures captured using a pixel-wise method [26] are employed as the higher-level features that are appealing to V2. For an image pixel p, the first-order derivatives along horizontal and vertical directions are determined by its relative value with its neighbouring pixel to the top pv and to the right ph. Denoting the grey-scale level of p, pv, and ph as g, gv and gh, respectively, the derivatives are mathematically determined as (6) (7)The local texture representation of p is defined as (8)where pi is the ith pixel in the eight-neighbour of p, and function sgn judges whether its input is positive or negative (9)Obviously, the texture of each pixel is an eight-bit binary code. With all pixels in an image coded in such a manner, the encoded map can be represented as a grey-scale image, as illustrated in Fig. 5. Fig. 5Open in figure viewerPowerPoint Local texture features of a 2D image The eight-bit binary codes representing texture information are of enumerating type, thus, it is not suitable to directly employ them to construct monocular feature maps. To overcome this difficulty, the luminance of left and right views (LL and LR) is utilised as the monocular feature (FL and FR) and synthesised to cyclopean luminance maps. Then, cyclopean texture maps CT are constructed by capturing local textures from the cyclopean luminance maps, resulting in a 256-bit code derived for each image pixel. Again because the texture codes are of enumerated type, the difference of CT from the test and reference is quantified by their histograms with histogram intersection, as another quality indicator (10)where function H(x) computes the histogram of x. Since the four-scale weight map is used for cyclopean construction, CT is also with four scales. 3.4 Binocular visual discomfort estimation Since the degree of visual discomfort is closely connected to the experienced depth of viewers [27, 28], and because the depth information can be reflected by disparity maps, statistical features from the disparity map derived from reference stereoscopic images are quantified to estimate binocular visual discomfort. Concretely, three indices, spatial depth level, depth complexity, and experienced disparity range are employed as indices for visual discomfort, which are mathematically represented as the average, variance, and range (ratio of maximum to minimum) of the disparity map. Moreover, we make extra efforts for further performance improvement: (i) the viewing experience is resistant to the outliers in the disparity maps, which appears in the maps as elements with extremely high or low values. Therefore, α1 (0 < α1 < 1) of the data elements are regarded as outliers and abandoned, including half extremely large ones and half the opposite, and only(1 − α1) of data are retained for further processing. (ii) Since the image regions with higher saliency have a larger impact on the sensation of discomfort and visual quality, we again abandon certain ratio (α2, 0 < α2 < 1) of data that are with relatively insufficient saliency level. Visual saliency is computed using a graph-based scheme [29]. Parameters α1 and α2 are empirically set as 3 and 20%, respectively. The remaining data are reformed as a vector d. Supposing a monocular view contains N pixels, the disparity map D contains N elements and d contains n elements (n = (1 – α1)·(1 – α2)·N). The estimated visual discomfort is regarded as the third group of quality indicators, represented by three indices as (11)where di is the ith element of d; dave, dmax and dmin denote the average, maximum and minimum of d, respectively. Finally, the three sets of quality indices, q1,s, q2,s, and q3 (s = 1–5), are mapped to the final stereoscopic image quality score with a regression function trained using support vector machine (SVM). 4 Experimental results and discussions 4.1 Experimental setups Experiments are conducted on LIVE public SIQA databases Phase I [2] and Phase II [8], both containing stereoscopic images distorted by JPEG 2000 compression (JP2K), JPEG compression, white noise (WN), GB and Rayleigh channel fast-fading (FF). Images in Phase I are only symmetrically distorted, which means that the two monocular views of any stereo pair in the database are distorted by the same type and degree; while Phase II contains both symmetrically and asymmetrically distorted images. The image quantities of the distortion types in the databases are concluded in Table 1. Table 1. Image number of different distortion types in LIVE SIQA databases Phase I and Phase II JP2K JPEG WN GB FF Total LIVE I 80 80 80 45 80 365 LIVE II 72 72 72 72 72 360 Each image in either database is labelled with a subjective quality rating in the form of difference mean opinion score (DMOS). The performance of the SIQA method is measured by the coherency between the predicted objective quality scores and the subjective ratings. In this study, the Pearson linear correlation coefficient (PLCC), Spearman rank-order correlation coefficient (SRCC) and root mean squared error (RMSE) are adopted to measure this coherency. Higher PLCC and SRCC and lower RMSE indicate better performance. 4.2 Overall experimental results on public databases Existing SIQA methods proposed respectively by Benoit et al. [30], You et al. [31], Bensalma and Larabi [32], Shao et al. [9], Chen et al. [8], Shao et al. [10], Lin et al. [11], Geng et al. [3] are referred for comparison. Since the quality score mapping procedure of the proposed method is training based, a commonly adopted five-fold cross-validation principle is obeyed [9]. Following this principle, when a dataset is used for testing the proposed method, it is randomly divided into five subsets with equivalent sizes, four of them are employed for training and the rest one is for testing. Since the cross-validation includes a random division stage, each time of the training-testing round might result differently. Therefore, all the results shown in this study of PLCC, SRCC, and RMSE are the average values of 1000 times of cross-validation. The performance comparison of the proposed method and the existing ones on LIVE SIQA databases Phase I and Phase II is shown in Table 2, with the best result of each row shown in bold. Evidently, the proposed method outperforms the others on every metric in both databases. It can also be observed that the performance of SIQA methods improves significantly along time, especially, the influence caused by binocular effects for SIQA is addressed. Earlier methods such as [30, 31] fail to well quantify the impact of binocular effects, resulting in a huge performance gap between experimental results of Phase I and Phase II. Comparatively speaking, this gap for more recent methods is better controlled. As for the proposed method, its performance on both databases shows that the quality-aware feature extraction scheme under the guidance of visual hierarchical structures and binocular effects is convincing, and the results on Phase II further prove the consideration on binocular effects is practical. Table 2. Overall performance comparison on LIVE SIQA databases Phase I and Phase II Phase I Phase II PLCC SRCC RMSE PLCC SRCC RMSE Benoit'08 0.8899 0.8901 7.4786 0.7450 0.7280 7.4900 You'09 0.9303 0.9247 6.0161 0.7970 0.7840 6.7720 Bensalma'13 0.8874 0.8747 7.5585 0.7699 0.7513 7.2035 Shao'13 0.9245 0.9217 6.2522 0.7585 0.7451 7.3554 Chen'13 0.9167 0.9157 6.5503 0.9010 0.8930 4.9870 Shao'15 0.9350 0.9251 5.8155 0.8628 0.8494 5.7058 Lin'17 0.9366 0.9314 5.7440 0.9113 0.8935 4.6477 Geng'17 0.9430 0.9320 5.5140 0.9210 0.9190 5.4000 Proposed 0.9603 0.9498 4.4533 0.9412 0.9356 3.7156 Since the SVM-based mapping results differently each time, a T-test with a significant level at 95% is conducted for all metrics in Table 2 to examine its stability. The results are shown in Table 3, where 1 denotes a significant performance divergence between the proposed method and a comparison and 0 denotes an insignificant difference. As is illustrated, the superiority of the proposed method against others is evident for most metrics, demonstrating its effectiveness. Table 3. Overall performance comparison on LIVE SIQA databases Phase I and Phase II Phase I Phase II PLCC SRCC RMSE PLCC SRCC RMSE Benoit'08 1 1 1 1 1 1 You'09 1 1 1 1 1 1 Bensalma'13 1 1 1 1 1 1 Shao'13 1 1 1 1 1 1 Chen'13 1 1 1 1 1 1 Shao'15 1 1 1 1 1 1 Lin'17 1 0 1 0 1 1 Geng'17 1 0 1 0 0 1 Moreover, in order to prove that the estimated visual discomfort degree is helpful in the quantification of stereoscopic image quality, experiments that use only the low- and high-level quality indices for quality score synthesis are conducted, and the results are illustrated in Table 4, denoted as case A, compared with the results in Table 2 denoted as case B. According to Table 4, all the performance indicators suggest that introducing visual discomfort estimation to the SIQA framework is effective. Table 4. Performance comparison between the cases with and without visual discomfort estimation Phase I Phase II PLCC SRCC RMSE PLCC SRCC RMSE A 0.9527 0.9404 4.9104 0.9215 0.9186 4.3002 B 0.9603 0.9498 4.4533 0.9412 0.9356 3.7156 4.3 Cross-database test It is known that machine learning-based algorithms often suffer from an over-fitting problem: goo