# Blind Stereo Image Quality Evaluation Based on Convolutional Network and Saliency Weighting

Authors: Wujie Zhou
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

With the rapid development of stereo image applications, there is an increasing demand to develop a versatile tool to evaluate the perceived quality of stereo images. Therefore, in this study, a blind stereo image quality evaluation (SIQE) algorithm based on convolutional network and saliency weighting is proposed. The main network framework used by the algorithm is the quality map generation network, which is used to train the distortion image dataset and quality map label to obtain an optimal network framework. Finally, the left view, right view, and cyclopean view of the stereo image are used as inputs to the network frame, respectively, and then weighted fusion for the final stereo image quality score. The experimental results reveal that the proposed SIQE algorithm can improve the accuracy of the image quality prediction and prediction score to a certain extent and has good generalization ability.


---

## Body

## 1. Introduction

With the rapid development of stereo image applications, many related stereo image technologies and services have been introduced in our daily lives as well as in many professional fields [1–9]. A variety of distortions can occur during the collection, transmission, processing, and displaying of stereo images [10–19]. Therefore, it is of immense practical significance to establish a high-performance stereo image quality evaluation method. Stereo image quality evaluation (SIQE) is classified into objective and subjective evaluation. Subjective evaluation is the subjective evaluation of images directly by humans. Because the human visual system is the ultimate recipient of images, subjective evaluation is very persuasive. However, in practical applications, subjective evaluation becomes extremely time-consuming and laborious and is difficult to apply to real-time systems. Therefore, objective evaluation plays a dominant role in SIQE. There are three main types of objective stereo image quality evaluation methods: full-reference (FR) evaluation [4–8], reduced-reference (RR) evaluation [9], and no-reference (NR) evaluation [10–22]. The FR evaluation method compares the undistorted original image with the distorted image to obtain the difference between them. The reduced-reference evaluation method uses the partially undistorted original image information. The NR evaluation method does not use the undistorted original image at all. Because the original undistorted image is difficult to obtain in practical applications, the NR evaluation method has a higher research value.Many SIQA methods are designed based on the typical 2D image quality evaluation methods [1–3]. The typical full-reference SIQE method was studied, which was proposed by Chen et al. [4], mainly uses a stereo image pair, disparity image, and Gabor filter response synthesis image. The central eye image uses an FR 2D image quality assessment method to predict the 3D quality score. Zhou et al. [11] proposed a new NR SIQE based on binocular self-similarity and deep neural networks. In [12], the stereo image block is first input to the convolutional neural network and then pooling. The final image quality score is obtained through the multilayer perceptron, where the initial parameters of the convolutional neural network are trained by a large number of natural images. Jiang et al. [13] performed SIQE by learning color visual characteristics based on nonnegative matrix factorization and considering binocular interaction. The study [14] proposes an NR stereo image quality assessment based on the combination of wavelet decomposition and statistical models. The SIQE method proposed in [15, 16] and the method proposed in [15] are based on the evaluation method of binocular vision mechanism. The method proposed in [16] is based on the gradient dictionary color visual feature learning evaluation method. Wu et al. [17] proposed an evaluation method based on depth edge information and color signals, which also uses segmented self-encoders. Liu et al. [18] proposed an SIQE method based on classification and prediction. Jiang et al. proposed an SIQE algorithm for processing multiple distortions [19], which characterizes the local receptive field characteristics of the visual cortex by learning monocular and binocular local visual primitives for various distorted stereo images quality assessment and single distortion stereo images. Bensalma and Larabi [20] proposed perceptual quality metric for stereo images based on the binocular energy. Zhou et al. [21] proposed blind SIQE metric based on binocular combination and extreme learning machine.Although the above methods have achieved certain effects on the stereo image evaluation problem, these evaluation methods do not consider versatility or image saliency weighting. Moreover, few studies consider the left and right views weight assignment problems of stereo images. Therefore, in this study, an NR SIQE method based on convolutional networks and saliency weighting is proposed. The main contributions of this work are as follows.First, the training dataset used in the proposed method is a self-made distortion image dataset, and the quality map obtained by the high-performance FR image quality evaluation method is used to render corresponding labels.Second, the main network framework used by the algorithm is the quality map generation network, which is used to train the distortion image dataset and quality map label to obtain an optimal network framework.Finally, the left view, the right view, and the cyclopean view of the stereo image are, respectively, used as the input of the quality map generation optimal network framework, and the corresponding quality maps are predicted. Weighted fusion is sequentially performed to obtain the final stereo image quality score.


## 2. Model Methods

Figure1 illustrates the overall frame structure of the proposed method. The inputs to the network frame are a left-view distortion image, right-view distortion image, and cyclopean view of the left and right views. The output is a predicted value of the distortion stereo image quality score. As can be seen from the figure, the overall framework can be divided into three submodules, the quality map generation network, the saliency weighting module, and the weighted summation module of the left, right, and cyclopean views. The specifications of each module are described in detail in the following sections.Figure 1
Overall structure diagram.


### 2.1. Quality Map Generation Network

In the proposed method, the quality map generation network is a main component of the overall framework. The requirement for the quality map generative network is outputting a quality map of the same size with the input image. We construct an improved framework of U-Net, an extension of fully convolutional networks, as a base of quality map generative network because U-Net integrates the hierarchical representations in subsampling layers with the corresponding features in upsampling layers. Hence, these layers increase the resolution of the output. In order to localize, high-resolution features from the contracting path are combined with the upsampled output. A successive convolution layer can then learn to assemble a more precise output based on this information.A major component of the quality map generation network is the convolutional layer. In the encoder part, the basic structure consists of a two-layer convolution plus a layer of pooling as a small module. The number of convolution kernels in each convolutional layer is depicted in Figure2. The size of the input distortion image is w × h × 3, the size of the output quality map is w × h, and the size of the convolution kernel is 3 × 3. The activation function uses a linear correction unit (ReLU) function. Let wil denote the parameters of lth filter kernel in the ith convolution layer and bil denote its corresponding bias. Then, the lth feature map produced in the ith convolution layer is represented by(1)yil=swilyil−1+bil,where hi−1 denote i−1th feature maps outputted from previous convolution layer and h0 corresponds to the input image. s⋅ is the ReLU function.Figure 2
Quality map generation network.The supervisory label used in the network is based on the image structure similarity (SSIM) quality map, and the effect of locally obtaining the SSIM quality map is better than that of globally obtaining it.In this work, we first select 100 source images from a dataset [23] for the training set. The source images have different scenes (all reference images of the dataset are shown in Figure 2, resized to a fixed-size of 528 × 400). Four commonly observed distortion types, namely, JPEG 2000 (JP2K) compression, JPEG compression, Gaussian blur (GB), and white noise (WN) are used to generate the distorted images. Finally, the SSIM metric is employed to generate the ground-truth objective quality/similarity maps as training labels. In the proposed method, the SSIM quality map used is the local mapping matrix. Let x and y be two image signals. The original and distorted image signals are obtained, thereby obtaining a quality map based on the similarity, and the generation formula of the SSIM quality map is defined as(2)SSIMx,y=2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2,where μx is the mean of image x, μy is the mean of image y, σx is the standard deviation of image x, σy is the standard deviation of image y, and σxy is the covariance of x and y and C1 and C2 denote small positive constant for increasing stability when the denominator approaches zero. In this paper, we set C1 = C2 = 0.085 for our experiments.


### 2.2. Weighted Fusion Module

The weighted fusion module is one of the important components of the proposed method, and its main structure is depicted in Figure1.In this study, the fusion of the quality score of the left and right views is based on the widely used Bayesian theory [24], in which the binocular quality score can be obtained by(3)PϑVl,V⌢r=PVl,V⌢rϑ⋅PϑPVl,V⌢≈PVlPVl,V⌢r⋅PϑVl+PV⌢rPIl,V⌢rPϑV⌢r.In (3), Vl and V⌢r denote the left retinal view and the disparity/depth-compensated right retinal view, respectively; ϑ denotes the quality; PVl/PVl,V⌢r and PV⌢r/PVl,V⌢r denote feature distributions that are utilized to balance the roles of binocular visual mechanism in determining the overall visual quality; and the likelihoods PϑVl and PϑV⌢r denote quality scores of the left and right views.The left and right views weights of the distorted stereo image are referred to the left and right views weight assignment methods of the paper [24], and the formula is as follows:(4)wl=gl2gl2+gr2,(5)wr=gr2gl2+gr2,where gl and gr are the significant level estimates of the left and right views, respectively, and equations (4) and (5), respectively, represent the weights of the left and right views, thereby obtaining a weighted quality score of the left and right views quality scores:(6)Qlr=wlQl+wrQr,where Ql=∑Smap,l×Qmap,l/∑Smap,l,Qr=∑Smap,r×Qmap,r/∑Smap,r. Smap,l and Smap,r denote the saliency maps of the left and right views (we use the method of reference [25] to derive saliency maps from distorted views), respectively. Qmap,l and Qmap,r denote the predict quality maps of the left and right views, respectively.In addition to the weights used in the combination of the left and right views quality scores, there is a set of weights that left and right views weighted scoreQlr and cyclopean view quality score Qm=∑Smap,m×Qmap,m/∑Smap,m. Smap,m denotes the saliency maps of the cyclopean views. Qmap,m denotes the predict quality maps of the cyclopean views. For the distorted stereo images, the over quality score can be obtained by(7)Q=αQlr+1−αQm,where α is the weight. The selection of weight α is discussed in the experimental section.


## 2.1. Quality Map Generation Network

In the proposed method, the quality map generation network is a main component of the overall framework. The requirement for the quality map generative network is outputting a quality map of the same size with the input image. We construct an improved framework of U-Net, an extension of fully convolutional networks, as a base of quality map generative network because U-Net integrates the hierarchical representations in subsampling layers with the corresponding features in upsampling layers. Hence, these layers increase the resolution of the output. In order to localize, high-resolution features from the contracting path are combined with the upsampled output. A successive convolution layer can then learn to assemble a more precise output based on this information.A major component of the quality map generation network is the convolutional layer. In the encoder part, the basic structure consists of a two-layer convolution plus a layer of pooling as a small module. The number of convolution kernels in each convolutional layer is depicted in Figure2. The size of the input distortion image is w × h × 3, the size of the output quality map is w × h, and the size of the convolution kernel is 3 × 3. The activation function uses a linear correction unit (ReLU) function. Let wil denote the parameters of lth filter kernel in the ith convolution layer and bil denote its corresponding bias. Then, the lth feature map produced in the ith convolution layer is represented by(1)yil=swilyil−1+bil,where hi−1 denote i−1th feature maps outputted from previous convolution layer and h0 corresponds to the input image. s⋅ is the ReLU function.Figure 2
Quality map generation network.The supervisory label used in the network is based on the image structure similarity (SSIM) quality map, and the effect of locally obtaining the SSIM quality map is better than that of globally obtaining it.In this work, we first select 100 source images from a dataset [23] for the training set. The source images have different scenes (all reference images of the dataset are shown in Figure 2, resized to a fixed-size of 528 × 400). Four commonly observed distortion types, namely, JPEG 2000 (JP2K) compression, JPEG compression, Gaussian blur (GB), and white noise (WN) are used to generate the distorted images. Finally, the SSIM metric is employed to generate the ground-truth objective quality/similarity maps as training labels. In the proposed method, the SSIM quality map used is the local mapping matrix. Let x and y be two image signals. The original and distorted image signals are obtained, thereby obtaining a quality map based on the similarity, and the generation formula of the SSIM quality map is defined as(2)SSIMx,y=2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2,where μx is the mean of image x, μy is the mean of image y, σx is the standard deviation of image x, σy is the standard deviation of image y, and σxy is the covariance of x and y and C1 and C2 denote small positive constant for increasing stability when the denominator approaches zero. In this paper, we set C1 = C2 = 0.085 for our experiments.


## 2.2. Weighted Fusion Module

The weighted fusion module is one of the important components of the proposed method, and its main structure is depicted in Figure1.In this study, the fusion of the quality score of the left and right views is based on the widely used Bayesian theory [24], in which the binocular quality score can be obtained by(3)PϑVl,V⌢r=PVl,V⌢rϑ⋅PϑPVl,V⌢≈PVlPVl,V⌢r⋅PϑVl+PV⌢rPIl,V⌢rPϑV⌢r.In (3), Vl and V⌢r denote the left retinal view and the disparity/depth-compensated right retinal view, respectively; ϑ denotes the quality; PVl/PVl,V⌢r and PV⌢r/PVl,V⌢r denote feature distributions that are utilized to balance the roles of binocular visual mechanism in determining the overall visual quality; and the likelihoods PϑVl and PϑV⌢r denote quality scores of the left and right views.The left and right views weights of the distorted stereo image are referred to the left and right views weight assignment methods of the paper [24], and the formula is as follows:(4)wl=gl2gl2+gr2,(5)wr=gr2gl2+gr2,where gl and gr are the significant level estimates of the left and right views, respectively, and equations (4) and (5), respectively, represent the weights of the left and right views, thereby obtaining a weighted quality score of the left and right views quality scores:(6)Qlr=wlQl+wrQr,where Ql=∑Smap,l×Qmap,l/∑Smap,l,Qr=∑Smap,r×Qmap,r/∑Smap,r. Smap,l and Smap,r denote the saliency maps of the left and right views (we use the method of reference [25] to derive saliency maps from distorted views), respectively. Qmap,l and Qmap,r denote the predict quality maps of the left and right views, respectively.In addition to the weights used in the combination of the left and right views quality scores, there is a set of weights that left and right views weighted scoreQlr and cyclopean view quality score Qm=∑Smap,m×Qmap,m/∑Smap,m. Smap,m denotes the saliency maps of the cyclopean views. Qmap,m denotes the predict quality maps of the cyclopean views. For the distorted stereo images, the over quality score can be obtained by(7)Q=αQlr+1−αQm,where α is the weight. The selection of weight α is discussed in the experimental section.


## 3. Experimental Results and Analysis

### 3.1. Experimental Database

In the experiment, the performance of the proposed method was verified using two published three-dimensional image quality evaluation databases: LIVE 3D Phase I [26] and LIVE 3D Phase II [27]. Basic information of the two databases is provided below.LIVE 3D Phase I: This database includes 20 pairs of original undistorted stereo images and 365 pairs of distorted stereo images. The size of each view is 640 × 360. All distorted images contain five different levels of distortion, including JPEG distortion, JPEG 2000 distortion, additive white Gaussian noise (WN) distortion, fast decay channel (FF) distortion, and Gaussian blur (GB) distortion. In addition, each pair of distorted images has a differential mean opinion score (DMOS), which is a human subjective image quality score obtained through a large number of experiments.LIVE 3D Phase II (LIVE 3D Phase II-Symmetric and LIVE 3D Phase II-Asymmetric) database includes 8 pairs of original undistorted stereo images and 360 pairs of distorted stereo images. The size of each view is 640 × 360. Similar to LIVE 3D Phase I, all the distorted images contain 5 different levels of distortion. For each type of distortion, three sets of symmetrically distorted stereo image pairs and six sets of asymmetrically distorted stereo image pairs are generated for each pair of original stereo images. “Asymmetric” means that the left and right views of the stereo image have different types or different levels of distortion. Each pair of stereo images has a DMOS value.


### 3.2. Experimental Training Step

The experiment was implemented on a 64 bit memory computer using an NVIDIA GTX 1080 TI. The deep network framework was implemented using the Keras depth framework with TensorFlow as the backend. When training deep neural networks, the optimizer used is Adam, which, also known as “adaptive momentum estimation,” is a parameter updating method that is used to calculate the adaptive learning rate for each parameter. The learning rate of some parameter updating methods is adjusted in the global scope, and the adjustment of all network parameters is equivalent, which makes the adjustment of the learning rate difficult, and requires very good initial network parameters, compared to these parameter updating methods. Adam has a bigger advantage. We use the Adam [28] optimization method with a learning rate of 1 × 10−4 with a decay of 0.5 every 50 epochs and a weight decay of 1 × 10−5 for regularization to optimize the network. In the iterative process, the batch size used is 16, and the loss function uses the mean square error (MSE).


### 3.3. Analysis of Experimental Results

The three classical performance evaluation indexes are used to evaluate the performance of the proposed SIQE methods, namely, Pearson linear correlation coefficient (PLCC), Spearman correlation coefficient (SROCC), and root mean square error (RMSE). The PLCC index can be used to evaluate the prediction accuracy of the NR image quality evaluation algorithm. The higher the correlation between the objective prediction score of the image and the subjective score is, the closer the PLCC correlation coefficient is to 1, indicating better performance of the image quality evaluation algorithm.Since the planar database we created contains only four types of distortion, namely, JPEG, JPEG 2000, WN, and GB, only four types of distortions are tested and analyzed in the experiment. The data shown in Table1 are the data for all four distortions.Table 1
Weights of the left and right views quality scores and the cyclopean view quality scores assigned to the PLCC, SROCC, and RMSE performance coefficients.
Databases
Indicator
α = 0.1
α = 0.2
Α = 0.3
α = 0.4
α = 0.5
α = 0.6
α = 0.7
α = 0.8
α = 0.9
LIVE 3D Phase I
PLCC
0.913
0.913
0.913
0.912
0.910
0.908
0.907
0.904
0.902
SROCC
0.885
0.887
0.889
0.890
0.889
0.888
0.887
0.885
0.883
RMSE
6.347
6.337
6.354
6.389
6.438
6.496
6.560
6.629
6.700
LIVE 3D Phase II
PLCC
0.858
0.863
0.861
0.856
0.849
0.841
0.833
0.824
0.816
SROCC
0.849
0.863
0.849
0.842
0.833
0.825
0.818
0.812
0.802
RMSE
5.739
5.651
5.690
5.787
5.910
6.045
6.186
6.329
6.470The results of the weight distribution experiment when the left and right views quality scores are combined with the cyclopean view quality score are shown in Table1. From the experiment, it can be concluded that, in the stereo distortion database, the best performance is obtained under this condition, that is, the value is α = 0.2.The overall test results are shown in Table2. Here, the final result obtained with a value of 0.2 was used. It can be seen from the relevant performance index that the image quality evaluation method proposed in this paper has a good generalization ability.Table 2
Results of the proposed method in terms of the performance coefficients of PLCC, SROCC, and RMSE.
Databases
Indicator
JPEG
JPEG 2000
WN
GB
All
LIVE 3D Phase I
PLCC
0.621
0.908
0.941
0.915
0.913
SROCC
0.592
0.881
0.931
0.857
0.887
RMSE
5.124
5.439
5.632
5.845
6.337
LIVE 3D Phase II
PLCC
0.827
0.727
0.973
0.960
0.863
SROCC
0.790
0.715
0.962
0.852
0.863
RMSE
4.120
6.742
2.484
3.882
5.651In Tables3 and 4, two quality evaluation performance indicators PLCC and SROCC are used to compare the SIQE method proposed in this paper with the methods proposed by the predecessors, including three 2D full-reference image quality evaluation methods (SSIM [1], FSIM [2], and GMSD [3]) and three SIQE methods (Chen [4], Bensalma and Larabi [20] and Zhou [21]). It can be seen from Table 3 that the stereo image evaluation method proposed in this paper is better than the other NR image evaluation methods. As can be seen from Table 4, the method proposed in this paper is more advantageous for evaluating asymmetric stereo images. Since the proposed method does not require original undistorted images and human subjective scores, its evaluation performance may be slightly lower than that of the FR evaluation method, but it achieves the desired effect of no-reference SIQE.Table 3
Comparison of the results of the proposed method and several other methods in terms of the PLCC coefficient of performance.
Databases
Distortion
FR
Blind
SSIM
FSIM
GMSD
Chen
Bensalma
Shao
Proposed
LIVE 3D Phase I
JP2K
0.868
0.937
0.928
0.855
0.848
0.872
0.908
JPEG
0.496
0.601
0.652
0.476
0.376
0.597
0.622
WN
0.938
0.931
0.947
0.9533
0.914
0.916
0.941
Gblur
0.912
0.933
0.938
0.939
0.916
0.923
0.915
ALL
0.899
0.936
0.943
0.929
0.895
0.899
0.913
LIVE 3D Phase II-symmetric
JP2K
0.8162
0.8183
0.875
0.670
0.690
0.903
0.898
JPEG
0.6770
0.8456
0.844
0.601
0.551
0.873
0.890
WN
0.9749
0.9630
0.961
0.946
0.936
0.917
0.982
GBLUR
0.8325
0.8638
0.928
0.918
0.953
0.977
0.953
ALL
0.7326
0.8301
0.925
0.814
0.823
0.912
0.918
LIVE 3D Phase II-asymmetric
JP2K
0.676
0.785
0.868
0.722
0.619
0.789
0.876
JPEG
0.685
0.796
0.869
0.564
0.631
0.705
0.699
WN
0.823
0.941
0.916
0.945
0.933
0.924
0.965
GBLUR
0.840
0.888
0.741
0.692
0.862
0.855
0.951
ALL
0.750
0.678
0.653
0.634
0.743
0.565
0.766Table 4
Comparison of the results of the proposed method and several other methods in terms of the SROCC coefficient of performance.
Databases
Distortion
FR
Blind
SSIM
FSIM
GMAD
Chen
Bensalm
Shao
Proposed
LIVE 3D Phase I
JP2K
0.867
0.901
0.905
0.871
0.817
0.900
0.881
JPEG
0.456
0.563
0.610
0.435
0.328
0.607
0.594
WN
0.938
0.930
0.947
0.939
0.906
0.927
0.931
Gblur
0.899
0.925
0.937
0.921
0.918
0.924
0.858
ALL
0.882
0.913
0.922
0.882
0.840
0.894
0.887
LIVE 3D Phase II-Symmetric
JP2K
0.726
0.824
0.867
0.662
0.608
0.904
0.872
JPEG
0.718
0.841
0.838
0.630
0.548
0.910
0.884
WN
0.945
0.937
0.927
0.907
0.924
0.937
0.942
GBLUR
0.770
0.850
0.836
0.845
0.846
0.911
0.652
ALL
0.700
0.909
0.910
0.837
0.805
0.897
0.893
LIVE 3D Phase II-Asymmetric
JP2K
0.724
0.805
0.854
0.722
0.619
0.789
0.872
JPEG
0.714
0.805
0.876
0.636
0.678
0.696
0.686
WN
0.882
0.952
0.937
0.929
0.941
0.924
0.955
GBLUR
0.807
0.850
0.888
0.691
0.840
0.803
0.838
ALL
0.719
0.661
0.642
0.611
0.697
0.524
0.711A good image quality evaluation method not only yields high performance, but also provides good computational efficiency. Predictive speed is also an important component of image quality evaluation performance indicators, because we need to consider the practicality of the algorithm. Here, we present the calculation time required for a pair of stereo images. In the LIVE 3D Phase I database, the prediction time for a pair of stereo images by using the NVIDIA 1080Ti GPU is approximately 0.034 s (more than 25 frames per second), while testing the pair in the LIVE 3D Phase II database, the prediction time for the stereo image is 0.039 s (more than 25 frames per second). In terms of speed, the method proposed in this paper can achieve real-time prediction (the real-time system run at the speed of 25 frames per second).


## 3.1. Experimental Database

In the experiment, the performance of the proposed method was verified using two published three-dimensional image quality evaluation databases: LIVE 3D Phase I [26] and LIVE 3D Phase II [27]. Basic information of the two databases is provided below.LIVE 3D Phase I: This database includes 20 pairs of original undistorted stereo images and 365 pairs of distorted stereo images. The size of each view is 640 × 360. All distorted images contain five different levels of distortion, including JPEG distortion, JPEG 2000 distortion, additive white Gaussian noise (WN) distortion, fast decay channel (FF) distortion, and Gaussian blur (GB) distortion. In addition, each pair of distorted images has a differential mean opinion score (DMOS), which is a human subjective image quality score obtained through a large number of experiments.LIVE 3D Phase II (LIVE 3D Phase II-Symmetric and LIVE 3D Phase II-Asymmetric) database includes 8 pairs of original undistorted stereo images and 360 pairs of distorted stereo images. The size of each view is 640 × 360. Similar to LIVE 3D Phase I, all the distorted images contain 5 different levels of distortion. For each type of distortion, three sets of symmetrically distorted stereo image pairs and six sets of asymmetrically distorted stereo image pairs are generated for each pair of original stereo images. “Asymmetric” means that the left and right views of the stereo image have different types or different levels of distortion. Each pair of stereo images has a DMOS value.


## 3.2. Experimental Training Step

The experiment was implemented on a 64 bit memory computer using an NVIDIA GTX 1080 TI. The deep network framework was implemented using the Keras depth framework with TensorFlow as the backend. When training deep neural networks, the optimizer used is Adam, which, also known as “adaptive momentum estimation,” is a parameter updating method that is used to calculate the adaptive learning rate for each parameter. The learning rate of some parameter updating methods is adjusted in the global scope, and the adjustment of all network parameters is equivalent, which makes the adjustment of the learning rate difficult, and requires very good initial network parameters, compared to these parameter updating methods. Adam has a bigger advantage. We use the Adam [28] optimization method with a learning rate of 1 × 10−4 with a decay of 0.5 every 50 epochs and a weight decay of 1 × 10−5 for regularization to optimize the network. In the iterative process, the batch size used is 16, and the loss function uses the mean square error (MSE).


## 3.3. Analysis of Experimental Results

The three classical performance evaluation indexes are used to evaluate the performance of the proposed SIQE methods, namely, Pearson linear correlation coefficient (PLCC), Spearman correlation coefficient (SROCC), and root mean square error (RMSE). The PLCC index can be used to evaluate the prediction accuracy of the NR image quality evaluation algorithm. The higher the correlation between the objective prediction score of the image and the subjective score is, the closer the PLCC correlation coefficient is to 1, indicating better performance of the image quality evaluation algorithm.Since the planar database we created contains only four types of distortion, namely, JPEG, JPEG 2000, WN, and GB, only four types of distortions are tested and analyzed in the experiment. The data shown in Table1 are the data for all four distortions.Table 1
Weights of the left and right views quality scores and the cyclopean view quality scores assigned to the PLCC, SROCC, and RMSE performance coefficients.
Databases
Indicator
α = 0.1
α = 0.2
Α = 0.3
α = 0.4
α = 0.5
α = 0.6
α = 0.7
α = 0.8
α = 0.9
LIVE 3D Phase I
PLCC
0.913
0.913
0.913
0.912
0.910
0.908
0.907
0.904
0.902
SROCC
0.885
0.887
0.889
0.890
0.889
0.888
0.887
0.885
0.883
RMSE
6.347
6.337
6.354
6.389
6.438
6.496
6.560
6.629
6.700
LIVE 3D Phase II
PLCC
0.858
0.863
0.861
0.856
0.849
0.841
0.833
0.824
0.816
SROCC
0.849
0.863
0.849
0.842
0.833
0.825
0.818
0.812
0.802
RMSE
5.739
5.651
5.690
5.787
5.910
6.045
6.186
6.329
6.470The results of the weight distribution experiment when the left and right views quality scores are combined with the cyclopean view quality score are shown in Table1. From the experiment, it can be concluded that, in the stereo distortion database, the best performance is obtained under this condition, that is, the value is α = 0.2.The overall test results are shown in Table2. Here, the final result obtained with a value of 0.2 was used. It can be seen from the relevant performance index that the image quality evaluation method proposed in this paper has a good generalization ability.Table 2
Results of the proposed method in terms of the performance coefficients of PLCC, SROCC, and RMSE.
Databases
Indicator
JPEG
JPEG 2000
WN
GB
All
LIVE 3D Phase I
PLCC
0.621
0.908
0.941
0.915
0.913
SROCC
0.592
0.881
0.931
0.857
0.887
RMSE
5.124
5.439
5.632
5.845
6.337
LIVE 3D Phase II
PLCC
0.827
0.727
0.973
0.960
0.863
SROCC
0.790
0.715
0.962
0.852
0.863
RMSE
4.120
6.742
2.484
3.882
5.651In Tables3 and 4, two quality evaluation performance indicators PLCC and SROCC are used to compare the SIQE method proposed in this paper with the methods proposed by the predecessors, including three 2D full-reference image quality evaluation methods (SSIM [1], FSIM [2], and GMSD [3]) and three SIQE methods (Chen [4], Bensalma and Larabi [20] and Zhou [21]). It can be seen from Table 3 that the stereo image evaluation method proposed in this paper is better than the other NR image evaluation methods. As can be seen from Table 4, the method proposed in this paper is more advantageous for evaluating asymmetric stereo images. Since the proposed method does not require original undistorted images and human subjective scores, its evaluation performance may be slightly lower than that of the FR evaluation method, but it achieves the desired effect of no-reference SIQE.Table 3
Comparison of the results of the proposed method and several other methods in terms of the PLCC coefficient of performance.
Databases
Distortion
FR
Blind
SSIM
FSIM
GMSD
Chen
Bensalma
Shao
Proposed
LIVE 3D Phase I
JP2K
0.868
0.937
0.928
0.855
0.848
0.872
0.908
JPEG
0.496
0.601
0.652
0.476
0.376
0.597
0.622
WN
0.938
0.931
0.947
0.9533
0.914
0.916
0.941
Gblur
0.912
0.933
0.938
0.939
0.916
0.923
0.915
ALL
0.899
0.936
0.943
0.929
0.895
0.899
0.913
LIVE 3D Phase II-symmetric
JP2K
0.8162
0.8183
0.875
0.670
0.690
0.903
0.898
JPEG
0.6770
0.8456
0.844
0.601
0.551
0.873
0.890
WN
0.9749
0.9630
0.961
0.946
0.936
0.917
0.982
GBLUR
0.8325
0.8638
0.928
0.918
0.953
0.977
0.953
ALL
0.7326
0.8301
0.925
0.814
0.823
0.912
0.918
LIVE 3D Phase II-asymmetric
JP2K
0.676
0.785
0.868
0.722
0.619
0.789
0.876
JPEG
0.685
0.796
0.869
0.564
0.631
0.705
0.699
WN
0.823
0.941
0.916
0.945
0.933
0.924
0.965
GBLUR
0.840
0.888
0.741
0.692
0.862
0.855
0.951
ALL
0.750
0.678
0.653
0.634
0.743
0.565
0.766Table 4
Comparison of the results of the proposed method and several other methods in terms of the SROCC coefficient of performance.
Databases
Distortion
FR
Blind
SSIM
FSIM
GMAD
Chen
Bensalm
Shao
Proposed
LIVE 3D Phase I
JP2K
0.867
0.901
0.905
0.871
0.817
0.900
0.881
JPEG
0.456
0.563
0.610
0.435
0.328
0.607
0.594
WN
0.938
0.930
0.947
0.939
0.906
0.927
0.931
Gblur
0.899
0.925
0.937
0.921
0.918
0.924
0.858
ALL
0.882
0.913
0.922
0.882
0.840
0.894
0.887
LIVE 3D Phase II-Symmetric
JP2K
0.726
0.824
0.867
0.662
0.608
0.904
0.872
JPEG
0.718
0.841
0.838
0.630
0.548
0.910
0.884
WN
0.945
0.937
0.927
0.907
0.924
0.937
0.942
GBLUR
0.770
0.850
0.836
0.845
0.846
0.911
0.652
ALL
0.700
0.909
0.910
0.837
0.805
0.897
0.893
LIVE 3D Phase II-Asymmetric
JP2K
0.724
0.805
0.854
0.722
0.619
0.789
0.872
JPEG
0.714
0.805
0.876
0.636
0.678
0.696
0.686
WN
0.882
0.952
0.937
0.929
0.941
0.924
0.955
GBLUR
0.807
0.850
0.888
0.691
0.840
0.803
0.838
ALL
0.719
0.661
0.642
0.611
0.697
0.524
0.711A good image quality evaluation method not only yields high performance, but also provides good computational efficiency. Predictive speed is also an important component of image quality evaluation performance indicators, because we need to consider the practicality of the algorithm. Here, we present the calculation time required for a pair of stereo images. In the LIVE 3D Phase I database, the prediction time for a pair of stereo images by using the NVIDIA 1080Ti GPU is approximately 0.034 s (more than 25 frames per second), while testing the pair in the LIVE 3D Phase II database, the prediction time for the stereo image is 0.039 s (more than 25 frames per second). In terms of speed, the method proposed in this paper can achieve real-time prediction (the real-time system run at the speed of 25 frames per second).


## 4. Conclusions

In this paper, we propose an effective SIQE method, which is different from the existing image evaluation method based on deep learning. First, the proposed method training dataset is a self-made distortion image dataset and the corresponding quality maps obtained by the high-performance FR image quality evaluation method as labels. Second, the main network framework used by the algorithm is the quality map generation network, which is used to train the distortion image dataset and quality map label to obtain an optimal network framework. The physiological function of the neurons ensured that the predicted results are highly consistent with the original quality map. Finally, the left view, the right view, and the cyclopean view of the stereo image are, respectively, input into the quality map generation network framework, and the corresponding quality maps are predicted, the three quality prediction scores are obtained using saliency weighting, and the three quality values are obtained. Weighted fusion is sequentially performed to obtain the final stereo image quality score. Experiment was performed on two 3D LIVE databases to verify the effectiveness of the proposed algorithm. The experimental results show that the algorithm can improve the accuracy of image quality prediction and prediction scores and has a good generalization ability.



---

*Source: 1384921-2019-09-09.xml*