# Image Quality Predictor with Highly Efficient Fully Convolutional Neural Network

Authors: Cao Yu-Dong; Liao Xin-Lin; Liu Hai-Yan
Journal: Advances in Multimedia
 (2022)

---

## Abstract

A highly efficient deep fully convolutional neural network (DFCN) for image quality assessment (IQA) is designed in this paper. The DFCN consists of two branches, one scoring local patches and the other estimating the weights of local patches to enhance quality prediction. Then, the DFCN outputs quality score of the whole image with aggregate weighted average pooling. There are no fully connected layers in the DFCN, resulting in far fewer parameters. In addition, the network model utilizes multiscale images as inputs to enrich the extracted distortion information. Furthermore, the parameters of the model are optimized in two steps to reduce the requirement for computing power and the risk of overfitting. The parameters of the shared layers and the quality module are optimized firstly, and then, the parameters of the weight module are optimized with the designed loss function. The extensive experimental results show that the proposed DFCN outperforms other competing IQA methods and has strong generalization ability.


---

## Body

## 1. Introduction

The quality of digital images is degraded by noise or other factors during acquisition, compression, storage, or transmission. Only after correctly evaluating the quality of image can the image postprocessing be executed effectively. In addition, the performance comparison on digital image processing algorithms requires IQA metrics.The IQA methods are generally classified into two categories: subjective quality evaluation and objective quality evaluation [1]. Subjective evaluation is performed by the observer. The quality of subjective evaluation is usually tiered in five discrete levels: excellent, good, fair, poor, or bad. Objective evaluation is usually executed by a mathematical model that yields the numerical description of the image quality. The objective IQA methods are commonly divided into three types according to the availability of reference images: full-reference image quality assessment (FR-IQA), no-reference image quality assessment (NR-IQA), and reduced-reference image quality assessment (RR-IQA). FR-IQA evaluates the quality of the distorted image using a source reference image, whereas NR-IQA evaluates the quality of the distorted image without a source reference image (NR-IQA is also called blind image quality assessment, B-IQA). In contrast, RR-IQA evaluates the image quality using limited information from a reference image. IQA research started with FR-IQA, which is usually focused on evaluating the difference between the distorted image and the reference image based on the study of human vision. NR-IQA is not easily utilized due to the lack of the reference images. However, NR-IQA is of importance because it can be used in some real-time applications. NR-IQA has grown into a vigorous research topic in the past ten years.In the past decade, deep learning technology has developed rapidly in the automation and artificial intelligence fields. Deep learning usually contains three different types of neural network structures: convolutional neural networks (ConvNets), recurrent neural networks (RNNs), and generative adversarial networks (GANs). Deep neural network models incorporate image feature extraction and classification or regression into the unified optimizing framework to implement a real end-to-end training process. In image recognition, deep learning has performed better than traditional algorithms. Specifically, ConvNet-based deep learning has been widely applied to IQA metrics. Deep learning-based IQA methods usually require strong computing power and a large amount of training data. However, IQA datasets are usually small. Some deep learning-based IQA methods enhanced data by segmenting images into patches, but the label noise was produced.Motivated by Kang and Wang [2] and Ma et al. [3], we proposed a simple and highly efficient deep fully convolutional neural network model; one branch predicts primary scores of local patches, and the other branch enhances the predicted quality by estimating the weights for the local patches. Converting fully connected layers into convolutional layers at the end of the network enables our IQA method to generate conveniently quality scores and weights for the local patches with matrix format.This study and its features are summarized as follows. First, multiscale images are used as inputs to acquire more detailed distorted information. Second, the quality of the entire image is predicted by weighted scoring local patches of the image. Third, the weights of local patches (derived from the learning stage) have adaptive characteristics. Fourth, substituting convolutional layers for fully connected layers in the network results in fewer parameters, which mitigates overfitting. Fifth, the parameter optimization is executed through two sequential stages using the designed different loss functions, which reduces the requirements of computing power and facilitates training.Besides, we also included a review of IQA-related works before we proposed the network model in detail with experimental results presented. Our conclusions are presented at the end of this paper.


## 2. Related Works

The conventional IQA approaches try to design elaborate feature descriptors empirically which can efficiently depict the image degradation [4]. The structural similarity index (SSIM) [5] is such a classic FR-IQA algorithm that depends heavily on hand-crafted features. The SSIM imitates the human visual system’s evaluation of image quality by calculating the similarity measure of two images in terms of luminance, structure, and contrast features. Since the origin of the SSIM, Lin Zhang et al. [6] have made subsequent improvements to its performance.Early NR-IQA methods first extracted the statistical features of a distorted image and then predicted the quality score of the image using regression. Kumar and Singh Bawa [7] calculated the regional mutual information in the spatial domain based on the loss information of the distortion and then predicted the quality of the entire image. In addition, Oszust [8] converted the RGB image into the YCbCr color space, extracted the local features of key points, and then used the kernel-based support vector regression to output the image quality score. Bampis et al. [9] extracted detail loss measure features in the spatial domain and then used support vector regression to predict the quality of an image. Mittal et al. [10] presented the natural image quality evaluator (NIQE), which utilized the distance between two multivariate Gaussian models to quantify the score of a distorted image after extracting the quality-aware features. In 2012, Ye et al. [11] proposed COdebook Representation for No-reference Image Assessment (CORNIA), which is one of the first learning-based IQA methods. In CORNIA, spatially normalized patches are clustered using k-means based on which soft assignment encoding and maximal pooling are executed for codebook representation. CORNIA features have also been applied to dipIQ [12]. The deep artificial neural network has subsequently arisen in some IQA studies as the mainstream approach from conventional methods.The deep neural network predicts image quality through end-to-end optimization. For example, Kang et al. [13] executed a deep neural network using one convolutional layer and two fully connected layers as an end-to-end version of CORNIA. Lin and Wang [14] designed a GAN structure that generates simulated reference images for IQA. However, the training process of the GAN model is very complicated and prone to failure. Bosse et al. [15] proposed a deep neural network-based IQA model (WaDIQaM) that split the distorted images into many true patches as inputs, which led to label noise. WaDIQaM regresses the quality scores of the patches using a group of fully connected layers, each of which was weighted by the other fully connected layers to predict the final quality of a distorted image.In recent years, the transformer has been executed firstly in natural language processing, and it also attracts research interests in the computer vision field. You and Korhonen [16] applied the transformer to image quality assessment. The transformer encoder was used on the top of a feature map extracted by convolution neural networks. The transformer has achieved success in some computer vision tasks, but it has not exceeded convolutional neural networks for IQA.We designed the DFCN-IQA model without fully connected layers, which used multiscale images as inputs. To the best of our knowledge, the deep fully convolutional neural network has not yet been applied to the IQA tasks.


## 3. DFCN for NR-IQA

The developed DFCN-IQA approach employs a deep fully convolutional neural network with multiscale images as inputs and the adaptive weights for enhancing quality prediction. Multiscale images have much more distorted feature information. An image pyramid is used to represent multiscale images.


### 3.1. Image Pyramid for Multiscale Input

Lindeberg [17] has theoretically proved that the Gaussian function is the only possible scale-space kernel and that the scale space generated by the Gaussian kernel is closely related to visual cognition. Accordingly, the scale space of an image is expressed as(1)Lx,y,σ=Ix,y⊗Gx,y,σ,where I represents the initial input image, x and y are the coordinates of the pixels, and the symbol ⊗ represents the convolution operation. Gaussian kernel is defined as(2)Gx,y,σ=Ae−x2+y2/2σ2,where A is a constant coefficient and σ is the standard deviation, which controls the shape of the function. In practice, Gaussian convolution is executed with a sliding window, the boundary of which is usually limited to 3σ.Our image pyramid is composed of four scale images; the first scale image is the original image. After the original image is convoluted with a Gaussian function, it is downsampled sequentially by a factor of two to produce three other coarser scale images.As shown in Figure1, both the normal clean image and the distorted noisy image are decomposed into 2 scale images. The 5 × 5 patches at the same relative coordinates in all four images are displayed exaggeratedly. There is a similarity between the corresponding patches across the scales in the “clean” image; however, there are significant differences between the corresponding patches across the scales in the “noisy” image. Thus, the distortion changes for different scales in the noisy image. Multiscale distorted images contain rich feature information. We use 4 scale images as inputs in the IQA model, which can reach a good performance in the experiments. It can be seen that there is little difference between the two far coarser scales from Figure 1.Figure 1
The 5 × 5 patches at the same relative coordinates in all four images (from [18]).
(a)(b)


### 3.2. Network Architecture

The structure of the designed IQA network model consists of the quality module g1, the weight module g2, and shared convolutional layers as shown in Figure2. In the “conv-1” layer, the input original image is convolved with 32 filters, and each filter generates a feature map. Then, pooling is executed on each feature map to reduce the filter responses to a lower dimension. The second scale image is sent to the “conv-2” layer. The second feature map generated from the “conv-2” layer and the first feature map are sent to the “conv-3” layer together. Likewise, the third scale and fourth scale images are sent to the “conv-4” and “conv-6” layers, respectively. The “conv-7” layer produces the seventh feature map. Finally, the input image is mapped as 128-dimensional feature maps. The feature learning process in the shared layers is completed. The last aggregate average pooling operation yields the quality score of the overall image.Figure 2
Network structure of the IQA model.On top of the shared layers, the quality moduleg1, which implements subtask I, generates the primary quality scores for the local patches. The normal region of interest (ROI) in an image can be detected by the visual salience method [19], but the saliency region and the distortion region may not be the same location. Thus, the weight module g2, which implements subtask II, learns the weights for the local patches. Finally, the IQA model enhances the quality prediction through the weighted scoring method. Compared to a neural network containing fully connected layers with same depth, a fully convolutional neural network reduces the number of parameters because of the inductive bias of local connection and parameter sharing, which simplifies the computational complexity to a linear order.The configuration of the proposed model is shown in Table1. The stride of the convolution is fixed at 1 to extract the refined feature information of the distorted image. The size of the convolutional kernel, which determines the receptive field of the filter, is 3 × 3. Such a small convolutional kernel offers two main advantages. First, it can capture more refined changes in the image. Second, it can increase the depth of the network for the same receptive field. The spatial padding of the convolutional layer is structured such that the spatial resolution is preserved after the convolution operation (i.e., the padding is one pixel around the border for the 3 × 3 filter). The spatial pooling operation is performed after the convolutional layers have been processed. The spatial size was reduced by a factor of four via each maxpooling over a 2 × 2 window, with a stride of 2.Table 1
Configurations of the proposed deep ConvNet model.
ModuleLayer nameReceptive field sizeChannel dim.StridePadShared layerconv-13 × 33211pool2 × 220conv-23 × 33211conv-33 × 36411pool2 × 220conv-43 × 36411conv-53 × 312811pool2 × 220conv-63 × 36411conv-73 × 312811g1conv-83 × 312811conv-93 × 3111g2conv-103 × 312811conv-113 × 3111


### 3.3. Optimization of the Model

More parameters and less training data can easily lead to overfitting. A distinct feature of the proposed network model is that there are no fully connected layers, which results in far fewer parameters; however, the number of parameters is still much larger than traditional IQA methods. Thus, optimization of the parameters is executed in two steps with the different designed loss functions. Such an optimization method can reduce the requirements for computing power and the risk of overfitting.First, we optimize the shared layers and the quality moduleg1. The primary quality scores matrix y can be defined as(3)y=g1fI,w0,w1,where the parameters of the shared layers are denoted by w0, the parameters of module g1 are denoted by w1, I represents the input image (which is sent as input to the shared layers), and f (·,·), which is the input to module g1, represents the output of shared layers. Module g2 outputs the weight matrix a, which is defined as(4)a=g2fI,w0,w2,where the parameters of module g2 are denoted by w2 and f (·,·) is the input of module g2.The quality score of the overall image is generated through an aggregate weighted average pooling, and it is computed via(5)s=∑i,jsij=a×.y=∑i,jaij⋅yij,where the symbol ×. denotes Hadamard product between identically shaped primary quality score matrix and weight matrix and the symbol ⌈⋅⌉ means to find the sum of all matrix elements. sij represents the ultimate quality score of a local patch, aij is an element of a, and yij is an element of primary quality scores matrix y. The maximal values of subscripts i and j are equal to the size of the final feature maps, which is the same as the size of the fourth scale image.In the first step, the parameters of the shared layers and the quality moduleg1 are optimized, and the loss function is designed as(6)ℒ1=1M∑m=1Mg1fIm,w0,w1−lm1,where ⋅1 represents the norm of matrix and lm represents label scores of local patches in an image with a matrix format. We supposed that the local patches are evenly assigned quality labels from the whole annotated images. Im represents the mth image. Equation (6) describes the average error between the predicted quality score and ground truth score. To mitigate overfitting of the model, the regularization constraint item is introduced as(7)ℒ1=1M∑m=1Mg1fIm,w0,w1−lm1+λw1p,where ⋅p represents p norm. p takes 0, 1, or 2. We set p = 2. λ is a balance parameter between two items. The assumption that local quality is uniformly assigned over the distorted image causes a tremendous amount of label noise. So we design the other branch, weight module, which learns the weights for the local patches to enhance quality prediction.In the second step, the parameters of moduleg2 are optimized with a unified loss function after w0 and w1 are fixed, which is described as(8)ℒ2=1M∑m=1Mg2fIm,w0,w2×.g1fIm,w0,w1−lm,where lm represents the ground score of the distorted image (which satisfies lm = ⌈lm⌉). The symbols ⌈⋅⌉ and ×. have the same meanings as in equation (5). The symbol ⋅ represents absolute value. The norm regularization term constraint is introduced to mitigate overfitting, and then the unified loss function is rewritten as(9)ℒ2=1M∑m=1Mg2⋅,w2×.g1⋅,w1−ym+λw2p.Data enhancement, norm regularization, and dropout technology are employed during training to reduce the risk of overfitting. The training process is outlined as follows:Input: distorted image IOutput: predicted quality scoreStep 1. Given the original image, obtain four different scale images using an image pyramid methodStep 2. Input the four scale images into the network, and optimize the parameters of the shared layers and moduleg1 using equation (7)Step 3. Optimize the parameters of moduleg2 according to equation (9)Our network model can predict the quality score from coarse-to-fine grains with the dual-branch structure after training. Four scale images are generated with the image pyramid. The primary quality scores and the weights for the local patches are obtained through trained modulesg1 and g2, respectively. Then, the more accurate quality scores of the local patches are computed via the Hadamard product. Finally, the quality score of the whole image is computed by summing all the elements of this matrix.


## 3.1. Image Pyramid for Multiscale Input

Lindeberg [17] has theoretically proved that the Gaussian function is the only possible scale-space kernel and that the scale space generated by the Gaussian kernel is closely related to visual cognition. Accordingly, the scale space of an image is expressed as(1)Lx,y,σ=Ix,y⊗Gx,y,σ,where I represents the initial input image, x and y are the coordinates of the pixels, and the symbol ⊗ represents the convolution operation. Gaussian kernel is defined as(2)Gx,y,σ=Ae−x2+y2/2σ2,where A is a constant coefficient and σ is the standard deviation, which controls the shape of the function. In practice, Gaussian convolution is executed with a sliding window, the boundary of which is usually limited to 3σ.Our image pyramid is composed of four scale images; the first scale image is the original image. After the original image is convoluted with a Gaussian function, it is downsampled sequentially by a factor of two to produce three other coarser scale images.As shown in Figure1, both the normal clean image and the distorted noisy image are decomposed into 2 scale images. The 5 × 5 patches at the same relative coordinates in all four images are displayed exaggeratedly. There is a similarity between the corresponding patches across the scales in the “clean” image; however, there are significant differences between the corresponding patches across the scales in the “noisy” image. Thus, the distortion changes for different scales in the noisy image. Multiscale distorted images contain rich feature information. We use 4 scale images as inputs in the IQA model, which can reach a good performance in the experiments. It can be seen that there is little difference between the two far coarser scales from Figure 1.Figure 1
The 5 × 5 patches at the same relative coordinates in all four images (from [18]).
(a)(b)


## 3.2. Network Architecture

The structure of the designed IQA network model consists of the quality module g1, the weight module g2, and shared convolutional layers as shown in Figure2. In the “conv-1” layer, the input original image is convolved with 32 filters, and each filter generates a feature map. Then, pooling is executed on each feature map to reduce the filter responses to a lower dimension. The second scale image is sent to the “conv-2” layer. The second feature map generated from the “conv-2” layer and the first feature map are sent to the “conv-3” layer together. Likewise, the third scale and fourth scale images are sent to the “conv-4” and “conv-6” layers, respectively. The “conv-7” layer produces the seventh feature map. Finally, the input image is mapped as 128-dimensional feature maps. The feature learning process in the shared layers is completed. The last aggregate average pooling operation yields the quality score of the overall image.Figure 2
Network structure of the IQA model.On top of the shared layers, the quality moduleg1, which implements subtask I, generates the primary quality scores for the local patches. The normal region of interest (ROI) in an image can be detected by the visual salience method [19], but the saliency region and the distortion region may not be the same location. Thus, the weight module g2, which implements subtask II, learns the weights for the local patches. Finally, the IQA model enhances the quality prediction through the weighted scoring method. Compared to a neural network containing fully connected layers with same depth, a fully convolutional neural network reduces the number of parameters because of the inductive bias of local connection and parameter sharing, which simplifies the computational complexity to a linear order.The configuration of the proposed model is shown in Table1. The stride of the convolution is fixed at 1 to extract the refined feature information of the distorted image. The size of the convolutional kernel, which determines the receptive field of the filter, is 3 × 3. Such a small convolutional kernel offers two main advantages. First, it can capture more refined changes in the image. Second, it can increase the depth of the network for the same receptive field. The spatial padding of the convolutional layer is structured such that the spatial resolution is preserved after the convolution operation (i.e., the padding is one pixel around the border for the 3 × 3 filter). The spatial pooling operation is performed after the convolutional layers have been processed. The spatial size was reduced by a factor of four via each maxpooling over a 2 × 2 window, with a stride of 2.Table 1
Configurations of the proposed deep ConvNet model.
ModuleLayer nameReceptive field sizeChannel dim.StridePadShared layerconv-13 × 33211pool2 × 220conv-23 × 33211conv-33 × 36411pool2 × 220conv-43 × 36411conv-53 × 312811pool2 × 220conv-63 × 36411conv-73 × 312811g1conv-83 × 312811conv-93 × 3111g2conv-103 × 312811conv-113 × 3111


## 3.3. Optimization of the Model

More parameters and less training data can easily lead to overfitting. A distinct feature of the proposed network model is that there are no fully connected layers, which results in far fewer parameters; however, the number of parameters is still much larger than traditional IQA methods. Thus, optimization of the parameters is executed in two steps with the different designed loss functions. Such an optimization method can reduce the requirements for computing power and the risk of overfitting.First, we optimize the shared layers and the quality moduleg1. The primary quality scores matrix y can be defined as(3)y=g1fI,w0,w1,where the parameters of the shared layers are denoted by w0, the parameters of module g1 are denoted by w1, I represents the input image (which is sent as input to the shared layers), and f (·,·), which is the input to module g1, represents the output of shared layers. Module g2 outputs the weight matrix a, which is defined as(4)a=g2fI,w0,w2,where the parameters of module g2 are denoted by w2 and f (·,·) is the input of module g2.The quality score of the overall image is generated through an aggregate weighted average pooling, and it is computed via(5)s=∑i,jsij=a×.y=∑i,jaij⋅yij,where the symbol ×. denotes Hadamard product between identically shaped primary quality score matrix and weight matrix and the symbol ⌈⋅⌉ means to find the sum of all matrix elements. sij represents the ultimate quality score of a local patch, aij is an element of a, and yij is an element of primary quality scores matrix y. The maximal values of subscripts i and j are equal to the size of the final feature maps, which is the same as the size of the fourth scale image.In the first step, the parameters of the shared layers and the quality moduleg1 are optimized, and the loss function is designed as(6)ℒ1=1M∑m=1Mg1fIm,w0,w1−lm1,where ⋅1 represents the norm of matrix and lm represents label scores of local patches in an image with a matrix format. We supposed that the local patches are evenly assigned quality labels from the whole annotated images. Im represents the mth image. Equation (6) describes the average error between the predicted quality score and ground truth score. To mitigate overfitting of the model, the regularization constraint item is introduced as(7)ℒ1=1M∑m=1Mg1fIm,w0,w1−lm1+λw1p,where ⋅p represents p norm. p takes 0, 1, or 2. We set p = 2. λ is a balance parameter between two items. The assumption that local quality is uniformly assigned over the distorted image causes a tremendous amount of label noise. So we design the other branch, weight module, which learns the weights for the local patches to enhance quality prediction.In the second step, the parameters of moduleg2 are optimized with a unified loss function after w0 and w1 are fixed, which is described as(8)ℒ2=1M∑m=1Mg2fIm,w0,w2×.g1fIm,w0,w1−lm,where lm represents the ground score of the distorted image (which satisfies lm = ⌈lm⌉). The symbols ⌈⋅⌉ and ×. have the same meanings as in equation (5). The symbol ⋅ represents absolute value. The norm regularization term constraint is introduced to mitigate overfitting, and then the unified loss function is rewritten as(9)ℒ2=1M∑m=1Mg2⋅,w2×.g1⋅,w1−ym+λw2p.Data enhancement, norm regularization, and dropout technology are employed during training to reduce the risk of overfitting. The training process is outlined as follows:Input: distorted image IOutput: predicted quality scoreStep 1. Given the original image, obtain four different scale images using an image pyramid methodStep 2. Input the four scale images into the network, and optimize the parameters of the shared layers and moduleg1 using equation (7)Step 3. Optimize the parameters of moduleg2 according to equation (9)Our network model can predict the quality score from coarse-to-fine grains with the dual-branch structure after training. Four scale images are generated with the image pyramid. The primary quality scores and the weights for the local patches are obtained through trained modulesg1 and g2, respectively. Then, the more accurate quality scores of the local patches are computed via the Hadamard product. Finally, the quality score of the whole image is computed by summing all the elements of this matrix.


## 4. Experimental Results and Analysis

In this section, we first describe the IQA datasets and the evaluation metrics. We then conduct ablation experiments to identify the contributing factors in DFCN-IQA model. Finally, we compare the DFCN to classic and state-of-the-art IQA metrics. These results are computed through source code released by authors or come from existing papers.


### 4.1. Datasets and Their Enhancement Processing

LIVE [20] and TID2013 [21], adopted in our experiments, are popular IQA datasets. The KonIQ-10k dataset is not chosen because it is not used by most of the compared IQA algorithms. The main differences between the datasets are the numbers of reference images and distorted images, the types and levels of distortions, and the scoring standards. LIVE contains 29 source reference images, five distortion types, and 779 distorted images. The difference mean opinion score (DMOS) was used for subjective scoring in LIVE [22]. TID2013 dataset contains 25 source reference images, 24 distortions, and 3000 distorted images. The mean opinion score (MOS) was used for subjective scoring in TID2013. A reference image and its associated distorted images are shown in Figure 3.Figure 3
Examples of different types of distortions. (a) Reference image. (b) Spatially correlated noise. (c) Local block-wise distortions. (d) Comfort noise. (e) Quantization noise. (f) Chromatic aberrations. (g) Contrast change. (h) Change of color saturation.
(a)(b)(c)(d)(e)(f)(g)(h)It is challenging to train a deep learning model on LIVE and TID because the number of distorted images is scarce. To prevent overfitting, data enhancement is performed by rotating and mirroring distorted images. We then divide the training and test sets according to the reference images to ensure content independence regarding synthetic datasets LIVE and TID2013. We take 80% random images from each dataset to construct the training set. Then, we leave the remaining 20% for the testing. Different from the literature [15, 23], the training images were not normalized to reserve the distortion information such as original size, contrast, and luminance change.


### 4.2. Evaluation Criteria

The Spearman rank-order correlation coefficient (SROCC) and the Pearson linear correlation coefficient (PLCC) are standard evaluation criteria used by the Video Quality Experts Group (VQEG) [24]. The SROCC measures prediction monotonicity and the PLCC quantifies prediction accuracy. Both are correlation metric criteria, and their values close to 1 indicate good performance. The SROCC is defined as(10)PSROCC=1−6∑k=1Nrxk−ryk2NN2−1,where N refers to the size of the sample data, xk represents the subjective evaluating score, yk represents the predicted scores of the model, and rxk and ryk describe the rankings of xk and yk, respectively, in their own sample data. The PLCC is defined as(11)PPLCC=1N−1∑k=1Nxk−x¯δxyk−y¯δy,where x¯ and y¯ are the average values of the two sets of data and δx and δy are their corresponding standard deviations.


### 4.3. Ablation Study

A series of ablation experiments were conducted to identify the influences of the core factors. The DFCN model enhances the quality prediction of the local patches by estimating the weights. To verify the effect of the weight moduleg2, comparative experiments were conducted with multiple distortion types in TID2013: additive Gaussian noise (#1), spatially correlated noise (#3), quantization noise (#7), JPEG compression (#11), local block-wise distortions of different intensity (#15), change of color saturation (#18), and image color quantization with dither (#22). If the weight module g2 does not work, it means that all the elements in matrix a are 1 in equation (5). It can be seen from Figure 4 that the weight module improves the performance of the model.Figure 4
Influence of weight module on performance.The performance of the SROCC was compared between a single input and multiscale inputs. As shown in Figure5, the multiscale images as inputs enhance the quality prediction performance. Although the size of the distorted image is not fixed, we find that the four scale inputs can get excellent experimental performance.Figure 5
Comparison between single input and multiscale inputs.Gaussian convolution is indispensable in generating the image pyramid. We investigated how differentσ values affect the quality prediction performance, as illustrated in Figure 6. The value of σ was set to 1.65.Figure 6
Effect of differentσ (in the Gaussian function) on SROCC performance with the LIVE dataset.


### 4.4. Evaluation on Datasets

Our approach outperforms state-of-the-art NR-IQA methods, even some FR-IQA methods, and competes with other deep learning-based IQA methods.


#### 4.4.1. Performance Comparison on LIVE

We compare the proposed DFCN with 16 IQAs (8 traditional and 8 deep learning-based) on the LIVE dataset. Experimental results are displayed in Tables2 and 3, respectively. The best performance for each column is in bold. The distortion types include JPEG2000 compression distortion (JP2K), JPEG compression distortion (JPEG), white noise distortion (WN), and Gaussian blur distortion (BLUR). “ALL4” in the last column is a comprehensive value for the four distortions to the left.Table 2
Comparison of SROCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.940.970.910.95CORNIA0.910.950.950.900.92DIIVINE0.840.820.880.880.84SOM0.940.950.980.970.96BIQI0.780.880.920.830.91SNP-NIQE0.920.970.970.960.96BRISQUE0.910.910.950.940.93CNN0.950.970.970.960.96WaDIQaM0.940.960.970.940.95BIECON0.950.970.980.960.97DLIQA0.930.920.970.950.95MS-C0.970.970.980.980.96FSI0.900.960.920.960.88SGDNet0.960.960.970.980.98UNIQUE0.960.970.980.960.98IQT0.960.970.970.970.98DFCN0.950.970.990.990.98The top result is highlighted in boldface in each column.Table 3
Comparison of PLCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.980.980.920.96CORNIA0.910.950.950.900.92DIIVINE0.900.820.900.910.86SOM0.950.990.990.970.96BIQI0.780.930.930.830.92SNP-NIQE0.940.980.970.970.96BRISQUE0.930.950.950.930.94CNN0.950.980.980.950.96WaDIQaM0.960.940.940.940.96BIECON0.960.940.940.940.95DLIQA0.950.960.960.950.95MS-C0.960.960.960.970.96FSI0.910.930.930.970.89SGDNet0.960.960.960.970.97UNIQUE0.960.980.970.980.97IQT0.970.980.980.970.97DFCN0.950.990.990.980.97The 8 traditional competing methods are SSIM [5], CORNIA [11], BIQI (Blind Image Quality Index) [25], SOM (semantic obviousness metric) [26], DIIVINE (distortion identification-based image verity and integrity evaluation) [27], SNP-NIQE (Structure, Naturalness, and Perception quality-driven Natural Image Quality Evaluator) [28], BRISQUE (Blind/Referenceless Image Spatial QUality Evaluator) [29], and FSI (Free-energy principle and Sparse representation-based Index) [30].The 8 state-of-art deep learning-based methods are CNN (convolutional neural network for image quality assessment) [13], WaDIQaM-NR [15], SGDNet (Saliency-Guided Deep neural network) [31], DLIQA (Deep Learning-based Image Quality Assessment) [32], BIECON (Blind Image Evaluator based on a CONvolutional neural Network) [33], MS-C (Multiple Scale Concat) [34], UNIQUE (Unified No-reference Image Quality and Uncertainty Evaluator based on a deep neural network) [35], and IQT (Image Quality assessment with Transformers) [36].In the compared algorithms, CORNIA [11] is a classic learning-based method, and it was subsequently refined to SOM [26]. DLIQA [32] classified a distorted image into five levels and then estimated the quality score. DIIVINE [27] is an improved version of BIQI [25] with more advanced natural scene statistics. SNP-NIQE [28] was extended from NIQE, which extracted features of the structure, naturalness, and perception attributes. BRISQUE [29] extracted natural scene statistics features from locally normalized luminance coefficients. UNIQUE [35] used complex ResNet-34 as its backbone, and its parameters were initialized with the weights pretrained on ImageNet [37]. IQT [36] applied successfully a transformer architecture to a perceptual full-reference IQA task. Our DFCN-IQA demonstrates a state-of-the-art performance with LIVE compared to UNIQUE and IQT, despite a slight weak performance on JP2K. The excellent performance is mainly due to enhanced dual-task learning.As shown in Tables2 and 3, the overall performance of our approach is superior to that of the competing methods. In particular, it performed the best for the JPEG, WN, and GBLUR distortions. The network architecture of WaDIQaM-NR [15] is composed of 10 convolutional layers and five pooling layers. MS-C was proposed in 2020, and its basic modes are composed of parallel convolutional networks with fully connected layers. The experimental results demonstrate that our DFCN-IQA outperforms almost all other IQA methods for the four distortion types in LIVE. The comprehensive SROCC and PLCC values (“ALL4”) for the proposed DFCN-IQA method are superior to other mainstream methods on the LIVE.


#### 4.4.2. Performance Comparison on TID2013

The level and number of distortions in TID2013 far exceed those in LIVE. Accordingly, the IQA method was further tested on TID2013. The experimental results are shown in Figure7. The compared methods include CaHDC [4], SSIM [5], WaDIQaM-NR [15], SGDNet [31], BIQI [25], BIECON [33], HOSA (high order statistics aggregation) [38], MS-C [34], and FSI [30]. HOSA [38] utilizes improved CORNIA feature sets; its SROCC value is about 11% lower than that of DFCN-IQA. Here, CaHDC [4], WaDIQaM-NR [15], BIECON [33], MS-C [34], and our DFCN-IQA are deep learning-based IQA methods. The DFCN-IQA outperforms other IQA methods, including the four deep learning-based methods just mentioned, on TID2013. CaHDC [4] and SNP-NIQE [28] (published in 2020) are the latest IQA methods, which are an end-to-end blind image quality predictor with the cascaded deep neural network. CaHDC jointly optimized the multilevel feature extraction, hierarchical degradation concatenation, and quality score prediction; its SROCC performance is also slightly lower than that of our approach.Figure 7
Comparison of performances on TID2013.To further test the performance of our approach, the DFCN was compared to IL-NIQE [39], HOSA [38], SNP-NIQE [28], RankIQA [40], Pseudo [41], VRPON [42], MSDD [43], NSSI [44], and DIQA (deep image quality assessor) [45] for each individual distortion in TID2013.As illustrated in Table4, our DFCN is superior to the other methods. RankIQA [40] trains networks using ranking quality data and then trains deeper networks through transfer learning; its performance is 7% worse than that of our approach. DIQA [45] combined additionally two handcrafted features with a deep convolutional neural network model. In particular, VRPON [42], MSDD [43], NSSI [44], and DIQA [45] are the latest methods, but their performance is worse than that of our approach. In short, our approach exhibited the higher accuracy on most distortions of TID2013, and its quality prediction was highly consistent with human subjective evaluation scores.Table 4
Comparison of SROCC on 24 distortions of TID2013.
Models#1#2#3#4#5#6#7#8#9#10#11#12HOSA0.850.620.780.360.900.770.810.890.870.890.930.74RankIQA0.670.620.820.360.760.730.780.800.760.860.870.70IL-NIQE0.870.810.920.510.860.750.870.810.750.830.850.28SNP-NIQE0.880.730.650.740.870.800.860.860.610.880.880.29FSI0.710.720.700.720.770.700.260.950.830.860.900.36Pseudo0.920.860.530.750.920.460.490.860.420.910.870.79BIQI0.340.200.700.180.610.020.670.890.790.780.880.55VRPON0.830.730.900.560.880.910.880.920.830.890.920.71MSDD0.650.480.780.370.780.680.800.900.820.840.920.60NSSI0.920.860.930.830.940.890.830.920.840.900.920.68DIQA0.910.750.870.730.940.840.860.920.790.890.910.86DFCN0.960.960.980.910.960.960.950.930.880.890.930.93Models#13#14#15#16#17#18#19#20#21#22#23#24HOSA0.700.190.320.230.290.110.780.530.830.850.800.90RankIQA0.810.510.620.260.610.660.610.640.800.770.620.85IL-NIQE0.520.080.130.180.010.160.690.360.830.760.680.86SNP-NIQE0.600.020.030.100.260.110.740.210.840.790.640.53FSI0.630.440.560.620.570.260.640.530.360.760.750.88Pseudo0.490.010.230.110.180.380.860.070.600.680.730.79BIQI0.550.160.100.010.420.060.260.610.550.590.760.90VRPON0.800.600.520.360.470.690.840.540.830.800.790.86MSDD0.640.210.150.210.420.120.380.620.600.680.780.90NSSI0.680.180.650.090.760.450.890.420.760.860.890.91DIQA0.810.660.410.300.69−0.150.900.660.930.940.750.91DFCN0.820.330.760.760.900.930.960.780.960.930.820.93The proposed DFCN significantly outperforms the other methods in Table4. In particular, it exhibited a relatively high performance for masked noise (#4), mean shift noise (#16), and change of color saturation (#18). This indicates that our approach can effectively address visual linear distortion. The relatively high performances for impulse noise (#6), quantization noise (#7), and local block-wise distortions (#15) show that our approach can also manage visual nonlinear distortion. It has only a slight performance bias for four distortion types.To summarize, the validation results indicate that the predicted quality of our approach is consistent with human subjective evaluation. According to the current literature, there is no IQA index that is superior to others in terms of all the individual distortion types in TID2013, but our experimental results demonstrate that the proposed DFCN outperforms other IQA metrics on the two benchmark databases: LIVE and TID2013.


#### 4.4.3. Generalizability Evaluation

The most commonly encountered distortion types in TID2013 are shared with LIVE, which are JP2K, JPEG, WN, and BLUR. The SROCC and PLCC values of the cross-dataset evaluation with comparable results are displayed in Tables5 and 6. All models are trained on LIVE and tested on TID2013.Table 5
Cross-dataset evaluation on SROCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.910.870.890.820.88DIIVINE0.860.680.880.860.80BRISQUE0.910.890.890.890.88CORNIA0.910.920.800.930.89HOSA0.930.920.840.920.90dipIQ0.930.930.900.920.88WaDIQaM0.950.920.940.910.89MEON0.910.920.910.890.91DFCN0.960.930.960.940.92Table 6
Cross-dataset evaluations on PLCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.930.940.900.820.89DIIVINE0.900.700.880.860.79BRISQUE0.920.950.880.880.90CORNIA0.930.960.780.940.90HOSA0.950.950.840.920.92dipIQ0.950.970.910.930.89WaDIQaM0.950.920.940.910.89MEON0.960.960.940.900.91DFCN0.950.940.960.940.93The competing methods were selected to cover a diversity of design philosophies, containing five classic ones (CORNIA [11], IL-NIQE [39], DIIVINE [27], HOSA [38], and BRISQUE [29]) and three state-of-the-art deep learning-based ones (MEON [3], dipIQ [12], and WaDIQaM [15]). DFCN-IQA approach exhibits more powerful generalization ability compared to the deep neural network-based methods (i.e., MEON [3] and WaDIQaM-NR [15]). Unlike early deep neural networks, MEON [3] uses generalized divisive normalization (GDN) as the activation function, which is followed by four convolutional layers and two fully connected layers. The validation results indicate that the simple and highly efficient DFCN-IQA is superior to comparable methods, as it features excellent accuracy as well as strong generalization ability.Similar to other deep learning-based IQA methods, our approach has common features such as purely data-driven processes and end-to-end optimization. Moreover, we believe that the reasons for the superior performance of DFCN-IQA are as follows: first, the whole images, instead of the segments, are utilized as the inputs to reduce the label noise; second, the dual-branch learning framework enhances the quality score prediction regularized by the weights of the local patches; third, the staged optimization enables the network to decrease the risk of overfitting, resulting in a more robust model.


## 4.1. Datasets and Their Enhancement Processing

LIVE [20] and TID2013 [21], adopted in our experiments, are popular IQA datasets. The KonIQ-10k dataset is not chosen because it is not used by most of the compared IQA algorithms. The main differences between the datasets are the numbers of reference images and distorted images, the types and levels of distortions, and the scoring standards. LIVE contains 29 source reference images, five distortion types, and 779 distorted images. The difference mean opinion score (DMOS) was used for subjective scoring in LIVE [22]. TID2013 dataset contains 25 source reference images, 24 distortions, and 3000 distorted images. The mean opinion score (MOS) was used for subjective scoring in TID2013. A reference image and its associated distorted images are shown in Figure 3.Figure 3
Examples of different types of distortions. (a) Reference image. (b) Spatially correlated noise. (c) Local block-wise distortions. (d) Comfort noise. (e) Quantization noise. (f) Chromatic aberrations. (g) Contrast change. (h) Change of color saturation.
(a)(b)(c)(d)(e)(f)(g)(h)It is challenging to train a deep learning model on LIVE and TID because the number of distorted images is scarce. To prevent overfitting, data enhancement is performed by rotating and mirroring distorted images. We then divide the training and test sets according to the reference images to ensure content independence regarding synthetic datasets LIVE and TID2013. We take 80% random images from each dataset to construct the training set. Then, we leave the remaining 20% for the testing. Different from the literature [15, 23], the training images were not normalized to reserve the distortion information such as original size, contrast, and luminance change.


## 4.2. Evaluation Criteria

The Spearman rank-order correlation coefficient (SROCC) and the Pearson linear correlation coefficient (PLCC) are standard evaluation criteria used by the Video Quality Experts Group (VQEG) [24]. The SROCC measures prediction monotonicity and the PLCC quantifies prediction accuracy. Both are correlation metric criteria, and their values close to 1 indicate good performance. The SROCC is defined as(10)PSROCC=1−6∑k=1Nrxk−ryk2NN2−1,where N refers to the size of the sample data, xk represents the subjective evaluating score, yk represents the predicted scores of the model, and rxk and ryk describe the rankings of xk and yk, respectively, in their own sample data. The PLCC is defined as(11)PPLCC=1N−1∑k=1Nxk−x¯δxyk−y¯δy,where x¯ and y¯ are the average values of the two sets of data and δx and δy are their corresponding standard deviations.


## 4.3. Ablation Study

A series of ablation experiments were conducted to identify the influences of the core factors. The DFCN model enhances the quality prediction of the local patches by estimating the weights. To verify the effect of the weight moduleg2, comparative experiments were conducted with multiple distortion types in TID2013: additive Gaussian noise (#1), spatially correlated noise (#3), quantization noise (#7), JPEG compression (#11), local block-wise distortions of different intensity (#15), change of color saturation (#18), and image color quantization with dither (#22). If the weight module g2 does not work, it means that all the elements in matrix a are 1 in equation (5). It can be seen from Figure 4 that the weight module improves the performance of the model.Figure 4
Influence of weight module on performance.The performance of the SROCC was compared between a single input and multiscale inputs. As shown in Figure5, the multiscale images as inputs enhance the quality prediction performance. Although the size of the distorted image is not fixed, we find that the four scale inputs can get excellent experimental performance.Figure 5
Comparison between single input and multiscale inputs.Gaussian convolution is indispensable in generating the image pyramid. We investigated how differentσ values affect the quality prediction performance, as illustrated in Figure 6. The value of σ was set to 1.65.Figure 6
Effect of differentσ (in the Gaussian function) on SROCC performance with the LIVE dataset.


## 4.4. Evaluation on Datasets

Our approach outperforms state-of-the-art NR-IQA methods, even some FR-IQA methods, and competes with other deep learning-based IQA methods.


### 4.4.1. Performance Comparison on LIVE

We compare the proposed DFCN with 16 IQAs (8 traditional and 8 deep learning-based) on the LIVE dataset. Experimental results are displayed in Tables2 and 3, respectively. The best performance for each column is in bold. The distortion types include JPEG2000 compression distortion (JP2K), JPEG compression distortion (JPEG), white noise distortion (WN), and Gaussian blur distortion (BLUR). “ALL4” in the last column is a comprehensive value for the four distortions to the left.Table 2
Comparison of SROCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.940.970.910.95CORNIA0.910.950.950.900.92DIIVINE0.840.820.880.880.84SOM0.940.950.980.970.96BIQI0.780.880.920.830.91SNP-NIQE0.920.970.970.960.96BRISQUE0.910.910.950.940.93CNN0.950.970.970.960.96WaDIQaM0.940.960.970.940.95BIECON0.950.970.980.960.97DLIQA0.930.920.970.950.95MS-C0.970.970.980.980.96FSI0.900.960.920.960.88SGDNet0.960.960.970.980.98UNIQUE0.960.970.980.960.98IQT0.960.970.970.970.98DFCN0.950.970.990.990.98The top result is highlighted in boldface in each column.Table 3
Comparison of PLCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.980.980.920.96CORNIA0.910.950.950.900.92DIIVINE0.900.820.900.910.86SOM0.950.990.990.970.96BIQI0.780.930.930.830.92SNP-NIQE0.940.980.970.970.96BRISQUE0.930.950.950.930.94CNN0.950.980.980.950.96WaDIQaM0.960.940.940.940.96BIECON0.960.940.940.940.95DLIQA0.950.960.960.950.95MS-C0.960.960.960.970.96FSI0.910.930.930.970.89SGDNet0.960.960.960.970.97UNIQUE0.960.980.970.980.97IQT0.970.980.980.970.97DFCN0.950.990.990.980.97The 8 traditional competing methods are SSIM [5], CORNIA [11], BIQI (Blind Image Quality Index) [25], SOM (semantic obviousness metric) [26], DIIVINE (distortion identification-based image verity and integrity evaluation) [27], SNP-NIQE (Structure, Naturalness, and Perception quality-driven Natural Image Quality Evaluator) [28], BRISQUE (Blind/Referenceless Image Spatial QUality Evaluator) [29], and FSI (Free-energy principle and Sparse representation-based Index) [30].The 8 state-of-art deep learning-based methods are CNN (convolutional neural network for image quality assessment) [13], WaDIQaM-NR [15], SGDNet (Saliency-Guided Deep neural network) [31], DLIQA (Deep Learning-based Image Quality Assessment) [32], BIECON (Blind Image Evaluator based on a CONvolutional neural Network) [33], MS-C (Multiple Scale Concat) [34], UNIQUE (Unified No-reference Image Quality and Uncertainty Evaluator based on a deep neural network) [35], and IQT (Image Quality assessment with Transformers) [36].In the compared algorithms, CORNIA [11] is a classic learning-based method, and it was subsequently refined to SOM [26]. DLIQA [32] classified a distorted image into five levels and then estimated the quality score. DIIVINE [27] is an improved version of BIQI [25] with more advanced natural scene statistics. SNP-NIQE [28] was extended from NIQE, which extracted features of the structure, naturalness, and perception attributes. BRISQUE [29] extracted natural scene statistics features from locally normalized luminance coefficients. UNIQUE [35] used complex ResNet-34 as its backbone, and its parameters were initialized with the weights pretrained on ImageNet [37]. IQT [36] applied successfully a transformer architecture to a perceptual full-reference IQA task. Our DFCN-IQA demonstrates a state-of-the-art performance with LIVE compared to UNIQUE and IQT, despite a slight weak performance on JP2K. The excellent performance is mainly due to enhanced dual-task learning.As shown in Tables2 and 3, the overall performance of our approach is superior to that of the competing methods. In particular, it performed the best for the JPEG, WN, and GBLUR distortions. The network architecture of WaDIQaM-NR [15] is composed of 10 convolutional layers and five pooling layers. MS-C was proposed in 2020, and its basic modes are composed of parallel convolutional networks with fully connected layers. The experimental results demonstrate that our DFCN-IQA outperforms almost all other IQA methods for the four distortion types in LIVE. The comprehensive SROCC and PLCC values (“ALL4”) for the proposed DFCN-IQA method are superior to other mainstream methods on the LIVE.


### 4.4.2. Performance Comparison on TID2013

The level and number of distortions in TID2013 far exceed those in LIVE. Accordingly, the IQA method was further tested on TID2013. The experimental results are shown in Figure7. The compared methods include CaHDC [4], SSIM [5], WaDIQaM-NR [15], SGDNet [31], BIQI [25], BIECON [33], HOSA (high order statistics aggregation) [38], MS-C [34], and FSI [30]. HOSA [38] utilizes improved CORNIA feature sets; its SROCC value is about 11% lower than that of DFCN-IQA. Here, CaHDC [4], WaDIQaM-NR [15], BIECON [33], MS-C [34], and our DFCN-IQA are deep learning-based IQA methods. The DFCN-IQA outperforms other IQA methods, including the four deep learning-based methods just mentioned, on TID2013. CaHDC [4] and SNP-NIQE [28] (published in 2020) are the latest IQA methods, which are an end-to-end blind image quality predictor with the cascaded deep neural network. CaHDC jointly optimized the multilevel feature extraction, hierarchical degradation concatenation, and quality score prediction; its SROCC performance is also slightly lower than that of our approach.Figure 7
Comparison of performances on TID2013.To further test the performance of our approach, the DFCN was compared to IL-NIQE [39], HOSA [38], SNP-NIQE [28], RankIQA [40], Pseudo [41], VRPON [42], MSDD [43], NSSI [44], and DIQA (deep image quality assessor) [45] for each individual distortion in TID2013.As illustrated in Table4, our DFCN is superior to the other methods. RankIQA [40] trains networks using ranking quality data and then trains deeper networks through transfer learning; its performance is 7% worse than that of our approach. DIQA [45] combined additionally two handcrafted features with a deep convolutional neural network model. In particular, VRPON [42], MSDD [43], NSSI [44], and DIQA [45] are the latest methods, but their performance is worse than that of our approach. In short, our approach exhibited the higher accuracy on most distortions of TID2013, and its quality prediction was highly consistent with human subjective evaluation scores.Table 4
Comparison of SROCC on 24 distortions of TID2013.
Models#1#2#3#4#5#6#7#8#9#10#11#12HOSA0.850.620.780.360.900.770.810.890.870.890.930.74RankIQA0.670.620.820.360.760.730.780.800.760.860.870.70IL-NIQE0.870.810.920.510.860.750.870.810.750.830.850.28SNP-NIQE0.880.730.650.740.870.800.860.860.610.880.880.29FSI0.710.720.700.720.770.700.260.950.830.860.900.36Pseudo0.920.860.530.750.920.460.490.860.420.910.870.79BIQI0.340.200.700.180.610.020.670.890.790.780.880.55VRPON0.830.730.900.560.880.910.880.920.830.890.920.71MSDD0.650.480.780.370.780.680.800.900.820.840.920.60NSSI0.920.860.930.830.940.890.830.920.840.900.920.68DIQA0.910.750.870.730.940.840.860.920.790.890.910.86DFCN0.960.960.980.910.960.960.950.930.880.890.930.93Models#13#14#15#16#17#18#19#20#21#22#23#24HOSA0.700.190.320.230.290.110.780.530.830.850.800.90RankIQA0.810.510.620.260.610.660.610.640.800.770.620.85IL-NIQE0.520.080.130.180.010.160.690.360.830.760.680.86SNP-NIQE0.600.020.030.100.260.110.740.210.840.790.640.53FSI0.630.440.560.620.570.260.640.530.360.760.750.88Pseudo0.490.010.230.110.180.380.860.070.600.680.730.79BIQI0.550.160.100.010.420.060.260.610.550.590.760.90VRPON0.800.600.520.360.470.690.840.540.830.800.790.86MSDD0.640.210.150.210.420.120.380.620.600.680.780.90NSSI0.680.180.650.090.760.450.890.420.760.860.890.91DIQA0.810.660.410.300.69−0.150.900.660.930.940.750.91DFCN0.820.330.760.760.900.930.960.780.960.930.820.93The proposed DFCN significantly outperforms the other methods in Table4. In particular, it exhibited a relatively high performance for masked noise (#4), mean shift noise (#16), and change of color saturation (#18). This indicates that our approach can effectively address visual linear distortion. The relatively high performances for impulse noise (#6), quantization noise (#7), and local block-wise distortions (#15) show that our approach can also manage visual nonlinear distortion. It has only a slight performance bias for four distortion types.To summarize, the validation results indicate that the predicted quality of our approach is consistent with human subjective evaluation. According to the current literature, there is no IQA index that is superior to others in terms of all the individual distortion types in TID2013, but our experimental results demonstrate that the proposed DFCN outperforms other IQA metrics on the two benchmark databases: LIVE and TID2013.


### 4.4.3. Generalizability Evaluation

The most commonly encountered distortion types in TID2013 are shared with LIVE, which are JP2K, JPEG, WN, and BLUR. The SROCC and PLCC values of the cross-dataset evaluation with comparable results are displayed in Tables5 and 6. All models are trained on LIVE and tested on TID2013.Table 5
Cross-dataset evaluation on SROCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.910.870.890.820.88DIIVINE0.860.680.880.860.80BRISQUE0.910.890.890.890.88CORNIA0.910.920.800.930.89HOSA0.930.920.840.920.90dipIQ0.930.930.900.920.88WaDIQaM0.950.920.940.910.89MEON0.910.920.910.890.91DFCN0.960.930.960.940.92Table 6
Cross-dataset evaluations on PLCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.930.940.900.820.89DIIVINE0.900.700.880.860.79BRISQUE0.920.950.880.880.90CORNIA0.930.960.780.940.90HOSA0.950.950.840.920.92dipIQ0.950.970.910.930.89WaDIQaM0.950.920.940.910.89MEON0.960.960.940.900.91DFCN0.950.940.960.940.93The competing methods were selected to cover a diversity of design philosophies, containing five classic ones (CORNIA [11], IL-NIQE [39], DIIVINE [27], HOSA [38], and BRISQUE [29]) and three state-of-the-art deep learning-based ones (MEON [3], dipIQ [12], and WaDIQaM [15]). DFCN-IQA approach exhibits more powerful generalization ability compared to the deep neural network-based methods (i.e., MEON [3] and WaDIQaM-NR [15]). Unlike early deep neural networks, MEON [3] uses generalized divisive normalization (GDN) as the activation function, which is followed by four convolutional layers and two fully connected layers. The validation results indicate that the simple and highly efficient DFCN-IQA is superior to comparable methods, as it features excellent accuracy as well as strong generalization ability.Similar to other deep learning-based IQA methods, our approach has common features such as purely data-driven processes and end-to-end optimization. Moreover, we believe that the reasons for the superior performance of DFCN-IQA are as follows: first, the whole images, instead of the segments, are utilized as the inputs to reduce the label noise; second, the dual-branch learning framework enhances the quality score prediction regularized by the weights of the local patches; third, the staged optimization enables the network to decrease the risk of overfitting, resulting in a more robust model.


## 4.4.1. Performance Comparison on LIVE

We compare the proposed DFCN with 16 IQAs (8 traditional and 8 deep learning-based) on the LIVE dataset. Experimental results are displayed in Tables2 and 3, respectively. The best performance for each column is in bold. The distortion types include JPEG2000 compression distortion (JP2K), JPEG compression distortion (JPEG), white noise distortion (WN), and Gaussian blur distortion (BLUR). “ALL4” in the last column is a comprehensive value for the four distortions to the left.Table 2
Comparison of SROCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.940.970.910.95CORNIA0.910.950.950.900.92DIIVINE0.840.820.880.880.84SOM0.940.950.980.970.96BIQI0.780.880.920.830.91SNP-NIQE0.920.970.970.960.96BRISQUE0.910.910.950.940.93CNN0.950.970.970.960.96WaDIQaM0.940.960.970.940.95BIECON0.950.970.980.960.97DLIQA0.930.920.970.950.95MS-C0.970.970.980.980.96FSI0.900.960.920.960.88SGDNet0.960.960.970.980.98UNIQUE0.960.970.980.960.98IQT0.960.970.970.970.98DFCN0.950.970.990.990.98The top result is highlighted in boldface in each column.Table 3
Comparison of PLCC on LIVE.
ModelsJP2KJPEGWNBLURALL4SSIM0.950.980.980.920.96CORNIA0.910.950.950.900.92DIIVINE0.900.820.900.910.86SOM0.950.990.990.970.96BIQI0.780.930.930.830.92SNP-NIQE0.940.980.970.970.96BRISQUE0.930.950.950.930.94CNN0.950.980.980.950.96WaDIQaM0.960.940.940.940.96BIECON0.960.940.940.940.95DLIQA0.950.960.960.950.95MS-C0.960.960.960.970.96FSI0.910.930.930.970.89SGDNet0.960.960.960.970.97UNIQUE0.960.980.970.980.97IQT0.970.980.980.970.97DFCN0.950.990.990.980.97The 8 traditional competing methods are SSIM [5], CORNIA [11], BIQI (Blind Image Quality Index) [25], SOM (semantic obviousness metric) [26], DIIVINE (distortion identification-based image verity and integrity evaluation) [27], SNP-NIQE (Structure, Naturalness, and Perception quality-driven Natural Image Quality Evaluator) [28], BRISQUE (Blind/Referenceless Image Spatial QUality Evaluator) [29], and FSI (Free-energy principle and Sparse representation-based Index) [30].The 8 state-of-art deep learning-based methods are CNN (convolutional neural network for image quality assessment) [13], WaDIQaM-NR [15], SGDNet (Saliency-Guided Deep neural network) [31], DLIQA (Deep Learning-based Image Quality Assessment) [32], BIECON (Blind Image Evaluator based on a CONvolutional neural Network) [33], MS-C (Multiple Scale Concat) [34], UNIQUE (Unified No-reference Image Quality and Uncertainty Evaluator based on a deep neural network) [35], and IQT (Image Quality assessment with Transformers) [36].In the compared algorithms, CORNIA [11] is a classic learning-based method, and it was subsequently refined to SOM [26]. DLIQA [32] classified a distorted image into five levels and then estimated the quality score. DIIVINE [27] is an improved version of BIQI [25] with more advanced natural scene statistics. SNP-NIQE [28] was extended from NIQE, which extracted features of the structure, naturalness, and perception attributes. BRISQUE [29] extracted natural scene statistics features from locally normalized luminance coefficients. UNIQUE [35] used complex ResNet-34 as its backbone, and its parameters were initialized with the weights pretrained on ImageNet [37]. IQT [36] applied successfully a transformer architecture to a perceptual full-reference IQA task. Our DFCN-IQA demonstrates a state-of-the-art performance with LIVE compared to UNIQUE and IQT, despite a slight weak performance on JP2K. The excellent performance is mainly due to enhanced dual-task learning.As shown in Tables2 and 3, the overall performance of our approach is superior to that of the competing methods. In particular, it performed the best for the JPEG, WN, and GBLUR distortions. The network architecture of WaDIQaM-NR [15] is composed of 10 convolutional layers and five pooling layers. MS-C was proposed in 2020, and its basic modes are composed of parallel convolutional networks with fully connected layers. The experimental results demonstrate that our DFCN-IQA outperforms almost all other IQA methods for the four distortion types in LIVE. The comprehensive SROCC and PLCC values (“ALL4”) for the proposed DFCN-IQA method are superior to other mainstream methods on the LIVE.


## 4.4.2. Performance Comparison on TID2013

The level and number of distortions in TID2013 far exceed those in LIVE. Accordingly, the IQA method was further tested on TID2013. The experimental results are shown in Figure7. The compared methods include CaHDC [4], SSIM [5], WaDIQaM-NR [15], SGDNet [31], BIQI [25], BIECON [33], HOSA (high order statistics aggregation) [38], MS-C [34], and FSI [30]. HOSA [38] utilizes improved CORNIA feature sets; its SROCC value is about 11% lower than that of DFCN-IQA. Here, CaHDC [4], WaDIQaM-NR [15], BIECON [33], MS-C [34], and our DFCN-IQA are deep learning-based IQA methods. The DFCN-IQA outperforms other IQA methods, including the four deep learning-based methods just mentioned, on TID2013. CaHDC [4] and SNP-NIQE [28] (published in 2020) are the latest IQA methods, which are an end-to-end blind image quality predictor with the cascaded deep neural network. CaHDC jointly optimized the multilevel feature extraction, hierarchical degradation concatenation, and quality score prediction; its SROCC performance is also slightly lower than that of our approach.Figure 7
Comparison of performances on TID2013.To further test the performance of our approach, the DFCN was compared to IL-NIQE [39], HOSA [38], SNP-NIQE [28], RankIQA [40], Pseudo [41], VRPON [42], MSDD [43], NSSI [44], and DIQA (deep image quality assessor) [45] for each individual distortion in TID2013.As illustrated in Table4, our DFCN is superior to the other methods. RankIQA [40] trains networks using ranking quality data and then trains deeper networks through transfer learning; its performance is 7% worse than that of our approach. DIQA [45] combined additionally two handcrafted features with a deep convolutional neural network model. In particular, VRPON [42], MSDD [43], NSSI [44], and DIQA [45] are the latest methods, but their performance is worse than that of our approach. In short, our approach exhibited the higher accuracy on most distortions of TID2013, and its quality prediction was highly consistent with human subjective evaluation scores.Table 4
Comparison of SROCC on 24 distortions of TID2013.
Models#1#2#3#4#5#6#7#8#9#10#11#12HOSA0.850.620.780.360.900.770.810.890.870.890.930.74RankIQA0.670.620.820.360.760.730.780.800.760.860.870.70IL-NIQE0.870.810.920.510.860.750.870.810.750.830.850.28SNP-NIQE0.880.730.650.740.870.800.860.860.610.880.880.29FSI0.710.720.700.720.770.700.260.950.830.860.900.36Pseudo0.920.860.530.750.920.460.490.860.420.910.870.79BIQI0.340.200.700.180.610.020.670.890.790.780.880.55VRPON0.830.730.900.560.880.910.880.920.830.890.920.71MSDD0.650.480.780.370.780.680.800.900.820.840.920.60NSSI0.920.860.930.830.940.890.830.920.840.900.920.68DIQA0.910.750.870.730.940.840.860.920.790.890.910.86DFCN0.960.960.980.910.960.960.950.930.880.890.930.93Models#13#14#15#16#17#18#19#20#21#22#23#24HOSA0.700.190.320.230.290.110.780.530.830.850.800.90RankIQA0.810.510.620.260.610.660.610.640.800.770.620.85IL-NIQE0.520.080.130.180.010.160.690.360.830.760.680.86SNP-NIQE0.600.020.030.100.260.110.740.210.840.790.640.53FSI0.630.440.560.620.570.260.640.530.360.760.750.88Pseudo0.490.010.230.110.180.380.860.070.600.680.730.79BIQI0.550.160.100.010.420.060.260.610.550.590.760.90VRPON0.800.600.520.360.470.690.840.540.830.800.790.86MSDD0.640.210.150.210.420.120.380.620.600.680.780.90NSSI0.680.180.650.090.760.450.890.420.760.860.890.91DIQA0.810.660.410.300.69−0.150.900.660.930.940.750.91DFCN0.820.330.760.760.900.930.960.780.960.930.820.93The proposed DFCN significantly outperforms the other methods in Table4. In particular, it exhibited a relatively high performance for masked noise (#4), mean shift noise (#16), and change of color saturation (#18). This indicates that our approach can effectively address visual linear distortion. The relatively high performances for impulse noise (#6), quantization noise (#7), and local block-wise distortions (#15) show that our approach can also manage visual nonlinear distortion. It has only a slight performance bias for four distortion types.To summarize, the validation results indicate that the predicted quality of our approach is consistent with human subjective evaluation. According to the current literature, there is no IQA index that is superior to others in terms of all the individual distortion types in TID2013, but our experimental results demonstrate that the proposed DFCN outperforms other IQA metrics on the two benchmark databases: LIVE and TID2013.


## 4.4.3. Generalizability Evaluation

The most commonly encountered distortion types in TID2013 are shared with LIVE, which are JP2K, JPEG, WN, and BLUR. The SROCC and PLCC values of the cross-dataset evaluation with comparable results are displayed in Tables5 and 6. All models are trained on LIVE and tested on TID2013.Table 5
Cross-dataset evaluation on SROCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.910.870.890.820.88DIIVINE0.860.680.880.860.80BRISQUE0.910.890.890.890.88CORNIA0.910.920.800.930.89HOSA0.930.920.840.920.90dipIQ0.930.930.900.920.88WaDIQaM0.950.920.940.910.89MEON0.910.920.910.890.91DFCN0.960.930.960.940.92Table 6
Cross-dataset evaluations on PLCC with comparable results from [3].
ModelsJP2KJPEGWNBLURALL4IL-NIQE0.930.940.900.820.89DIIVINE0.900.700.880.860.79BRISQUE0.920.950.880.880.90CORNIA0.930.960.780.940.90HOSA0.950.950.840.920.92dipIQ0.950.970.910.930.89WaDIQaM0.950.920.940.910.89MEON0.960.960.940.900.91DFCN0.950.940.960.940.93The competing methods were selected to cover a diversity of design philosophies, containing five classic ones (CORNIA [11], IL-NIQE [39], DIIVINE [27], HOSA [38], and BRISQUE [29]) and three state-of-the-art deep learning-based ones (MEON [3], dipIQ [12], and WaDIQaM [15]). DFCN-IQA approach exhibits more powerful generalization ability compared to the deep neural network-based methods (i.e., MEON [3] and WaDIQaM-NR [15]). Unlike early deep neural networks, MEON [3] uses generalized divisive normalization (GDN) as the activation function, which is followed by four convolutional layers and two fully connected layers. The validation results indicate that the simple and highly efficient DFCN-IQA is superior to comparable methods, as it features excellent accuracy as well as strong generalization ability.Similar to other deep learning-based IQA methods, our approach has common features such as purely data-driven processes and end-to-end optimization. Moreover, we believe that the reasons for the superior performance of DFCN-IQA are as follows: first, the whole images, instead of the segments, are utilized as the inputs to reduce the label noise; second, the dual-branch learning framework enhances the quality score prediction regularized by the weights of the local patches; third, the staged optimization enables the network to decrease the risk of overfitting, resulting in a more robust model.


## 5. Conclusions

This study established a simple and highly efficient blind image quality predictor that exhibited the superior performance and generalization capability compared to other existing state-of-the-art IQA methods. The multiscale image inputs enrich the extracted distortion feature. A weighted scoring method enhances the mapping from the distorted image to the quality score.The experimental results verify that our model is effective for both linear and nonlinear distortions, and the test results are highly consistent with human subjective evaluation. However, there are still challenges for deep learning-based IQA with purely data-driven processes. The deep learning model is easy to overfit due to its high complexity. To prevent data overfitting and reduce the parameters, we implemented a series of measures, such as enhancing the data, introducing constraints to the loss functions, and adopting full convolution. Our DFCN can estimate image quality from coarse-to-fine grains, using two-stage method which makes the training easy.Deep neural networks with far more parameters than training data can also be reliably trained [46], but it is a long way from the current artificial neural network to produce human-like intelligence [47]. The big IQA datasets will contain more types of distortions in the future, which makes it difficult for the existing IQA metrics without retraining to achieve good performance on the specific distortion types. The types of distorted images in the real world are more complicated. Deep learning model needs big data. However, how to use big data is a research topic worth exploring [48]. We did not propose a full-fledged solution but believe that deep learning-based IQA methods will still have lots of potential in the future.



---

*Source: 1686298-2022-02-23.xml*