An improved objective metric to predict image quality using deep neural networks

Objective quality assessment of compressed images is very useful in many applications. In this paper we present an objective quality metric that is better tuned to evaluate the quality of images distorted by compression artifacts. A deep convolutional neural networks is used to extract features from a reference image and its distorted version. Selected features have both spatial and spectral characteristics providing substantial information on perceived quality. These features are extracted from numerous randomly selected patches from images and overall image quality is computed as a weighted sum of patch scores, where weights are learned during training. The model parameters are initialized based on a previous work and further trained using content from a recent JPEG XL call for proposals. The proposed model is then analyzed on both the above JPEG XL test set and images distorted by compression algorithms in the TID2013 database. Test results indicate that the new model outperforms the initial model, as well as other state-of-the-art objective quality metrics. Introduction Continuous improvements in digital imaging and video keep motivating broadcasters and service providers to supply contents of superior visual quality to their viewers, despite larger storage and transmission requirements. Compression is used to reduce such resources. The principal trade-off in compression is between perceived quality of the compressed content and transmission rate. The goal of efficient state-of-the-art compression algorithms is to achieve lower rates while maintaining the quality. Compression related artifacts such as blur, blocking, ringing and change in contrast can distort a content and reduce its perceived quality. It is therefore important for content providers to be able to anticipate the degree of annoyance caused by such distortions, and to optimize their systems accordingly. Quality assessment methods provide means of analyzing a content either subjectively or objectively. Subjective quality assessment methods employ human subjects and evaluate the quality of contents by collecting viewers ratings. While this is the most reliable form of measuring the quality of contents, the process is highly impractical in terms of time and labor costs. Objective quality assessment methods are more practical since they rely on mathematical models to evaluate the degradation and overall quality of contents. However, ensuring a good correlation between objective metrics and subjective ratings is a challenging problem. The general interest in image quality assessment (IQA) is to introduce an objective image quality metric (IQM) that is able to determine the quality of an image with high accuracy, i.e., with high correlation to the would-be perceived quality. The simplest measure to assess the quality of distorted images in a full reference (FR) framework, i.e. when the reference image is available, is a difference-based metric called the peak signal to noise ratio (PSNR). PSNR and its derivatives do not consider models based on the human visual system (HVS) and therefore often result in low correlations with subjective quality ratings. [1]. Metrics such as structural similarity index (SSIM) [2], multi-scale structural similarity index (MS-SSIM) [3], feature similarity index (FSIM) [4] and visual information fidelity (VIF) [5] use models motivated by HVS and natural scenes statistics, resulting in better correlations with viewers’ opinion. Numerous machine learning based objective quality metrics have been reported in the literature. Weighted Average Deep Image Quality Measure for FR-IQA (WaDIQaM-FR) [6] is an endto-end trained method that extracts features from reference and distorted image patches using convolutional filters. Recently, we have proposed a new model inspired by WaDIQaM-FR architecture that will be referred to as MTID2013 in the reminder of this paper [7]. Besides changes in the architecture and learning parameters when compared to those used in [6], MTID2013 incorporates both spatial and spectral information in its extracted features and consequently delivers more accurate results. To the best of the authors’ knowledge, none of the methods cited have been specially adjusted to compression artifacts, but trained using many different distortions in addition to compression, such as various types of noise, distortions due to quantization as well as transmission and sampling errors [8]. In this paper, we propose a FR-IQA based on deep convolutional neural network that is able to objectively predict the quality of distorted images suffering particularly from compression artifacts. The same architecture as in [7] is used and further trained with a new database that contains compression-based distortions only. This will make it more likely for the final model to correlates with subjective ratings when compared to the previous model, when evaluating the quality of compressed images. The rest of the paper is structured as follows. The proposed framework is described in details in the next section. Experiments and results are reported after the framework description. The paper ends by providing an overview on the performance, followed by discussions as well as future directions. Proposed Framework The proposed framework is identical to the architecture introduced in [7]. Features extracted from both a reference image and its distorted version are concatenated into a single feature vector that is passed onto the fully connected layers for regression and an objective quality score is assigned to the output. We used a Siamese network to extract features from both the reference and distorted images as in [6], whereas the design of convolutional IS&T International Symposium on Electronic Imaging 2019 Human Vision and Electronic Imaging 2019 214-1 https://doi.org/10.2352/ISSN.2470-1173.2019. 12.HVEI-214 © 2019, Society for Imaging Science and Technology