<figure><img src="image_1.png" /></figure>

HAL
open science

# NNCD-IQA: A new neural networks based compressed database for image quality assessment

Zohaib Amjad Khan, Tassnim Dardouri, Mounir Kaaniche, Gabriel Dauphin

► To cite this version:

Zohaib Amjad Khan, Tassnim Dardouri, Mounir Kaaniche, Gabriel Dauphin. NNCD-IQA: A new neural networks based compressed database for image quality assessment. Multimedia Tools and Applications, 2022, 82 (9), pp.13951-13971. 10.1007/s11042-022-13842-8. hal-03915363

HAL Id: hal-03915363

https://hal.science/hal-03915363

Submitted on 8 May 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
New Subjective Database with Learning Objective
Metrics for Quality Assessment of Neural Networks
based Compressed Images

Zohaib Amjad Khan, Tassnim Dardouri, Mounir Kaaniche, and Gabriel Dauphin

L2TI, Institut Galilée, Université Sorbonne Paris Nord, France

Abstract

Subjective and objective image quality assessment is still a challenging problem that needs to be further developed due to the recent advances in image acquisition technologies and the emergence of novel image processing techniques based on deep learning. In this paper, we focus on the quality assessment of neural networks based image compression algorithms. In this respect, a new database containing mainly neural networks based compressed images at different bitrates is firstly built. Then, subjective experiments are performed to evaluate the mean opinion scores of the obtained distorted images. Finally, an extensive evaluation and analysis of image quality assessment metrics is conducted. To this end, and in addition to conventional and machine learning metrics, neural networks based metrics will be investigated in this work after fine-tuning. The new subjective deep learning based compressed image database with its associated mean opinion scores as well as the trained models of the deep learning based metrics will be made publicly available. This will be of great interest to advance future research works devoted to the development of new image quality assessment metrics as well as the design of novel neural networks-based image compression algorithms.

Keywords: Image compression, neural networks, quality assessment,
subjective scores, learning objective metrics.

2010 MSC: 00-01, 99-00
# 1. Introduction

Due to the continuous advances of display and acquisition technologies, huge amounts of diverse visual data are generated every day, which constitute a major issue in terms of storage and transmission. In this respect, many research works have been dedicated to the design of efficient image compression methods. For instance, most of the developed algorithms are suitable for lossy compression and aims at minimizing the distortion of the reconstructed image at a given bi-trate. In this context, it becomes necessary to find appropriate quality metrics to assess the quality of the reconstructed images resulting from the employed image compression method.

While JPEG2000 [1] and more recently BPG<fnref n="1" /> (Better Portable Graphics) image compression standards have shown good performance in terms of quality of reconstruction, deep learning-based compression algorithms have attracted a great attention in the last years to improve the coding efficiency of the traditional compression schemes. An overview of image and video compression with deep learning approaches is provided in [2, 3]. More precisely, the developed algorithms are based on Neural Networks (NN) and consist of the following steps. First, a NN-based analysis stage is performed to transform the input image into a compact representation. The latter is then quantized and encoded. Finally, the inverse transform is achieved to obtain the reconstructed image. This typical architecture is referred to as auto-encoder where the network parameters are trained in an end-to-end manner [4, 5, 6, 7, 8, 9, 10]. It should be noted here that the main differences between these methods are related to the employed NN architecture and/or the retained loss function in the training phase. For instance, among the existing architectures, the Convolutional Neural Network (CNN) and Fully Connected Network (FCN) have been recently investigated for intra prediction in the context of image and video coding [11, 12]. An hybrid method, where small (resp. large) blocks are predicted using an FCN

<footnote n="1">¹https://bellard.org/bpg/</footnote>
(resp. a CNN) model, is proposed in [13]. Moreover, motivated by the different advantages of transform coding schemes, other methods have been developed to improve DCT (Discrete Cosine Transform) and DWT-based coding schemes [14, 15, 16, 17]. Indeed, a DCT-based coding scheme using a CNN is used in [14]. In [15], a DWT is first applied to the input image, and then, the generated subbands are fed into a CNN to produce the final detail coefficients. In [16], the authors propose to design a separable lifting structure based wavelet transform using a CNN. While the latter method employs the CNN for only the prediction stage, a fully nonlinear transform where prediction and update stages are performed using an FCN has been recently developed in [17]. This recent method has also been made adaptive by taking into account the input image to be encoded.

Since the developed methods are often dedicated to lossy compression, the quality assessment of the reconstructed (i.e. decoded) images becomes an important step to evaluate the performance of these deep learning based coding schemes. The quality assessment problem of compressed images have been widely investigated in the context of JPEG and JPEG2000 coding algorithms. In this context, the full-reference PSNR and SSIM (or MS-SSIM) metrics have been extensively used to evaluate the performance of conventional image compression algorithms. Moreover, other works have also been developed to design no-reference quality assessment metrics [18, 19, 20]. For example, in [19], a no-reference metric for JPEG compressed images based DCT coefficients distribution and PSNR estimation is proposed. Another method for JPEG2000 compressed images using natural scene statistics is developed in [20].

However, in the context of deep learning based image compression algorithms, the latter are often evaluated in terms of PSNR, SSIM (or MS-SSIM). In addition to this commonly used evaluation approach, there are very few works which have proposed to conduct some subjective experiments [21, 22, 23, 24]. Indeed, in [21], the authors achieve a pairwise comparison study with 10 observers to show the preference of their method compared to JPEG standard and a neural network baseline method [4]. In [22], a single-stimulus rating test with 25
observers is also performed to validate their coding approach and show its pref-
erence over JPEG and JPEG2000 compression methods as well as the neural
network baseline method [4]. A similar subjective study was performed in [23]
to show that MS-SSIM is better than MSE for optimizing an end-to-end learned
compression method. However, the latter works do not investigate the corre-
lation between the obtained Mean Opinion Scores (MOS) and the employed
objective metrics (PSNR and MS-SSIM). For this reason, Valenzise et al. have
proposed to study in [24] the accuracy of classical metrics in predicting MOS
for deep learning based compression methods. More precisely, using a double
stimuls rating test and 23 obervers, they have considered 6 references images
and 4 image compression methods which are JPEG2000, BPG as well as two
NN-based algorithms [4, 5]. It has been concluded that conventional PSNR and
SSIM metrics are not appropriate to assess the quality of deep learning-based
compressed images.

The objective of this paper is to further investigate the quality assessment (QA)
process in the context of deep learning (DL) based image compression algo-
rithms. More precisely, we first propose to build a new database where distorted
images using recent DL compression algorithms a well as their associated MOS
will be publicly available. While several databases with MOS have already been
developed for QA of JPEG and JPEG2000 compressed images, and to the best
of our knowledge, our dataset will be the first accessible database in the context
of QA of DL compressed images. For instance, compared to previous subjective
studies [23, 24], the proposed database includes more reference images with more
and newer DL compression algorithms resulting in a larger set. It is important
to note here that designing a large public subjective dataset presents a great
interest to the research community working on the development of both new
IQA alorithms as well as deep learning based compression algorithms. Based on
this database, we achieve an extensive evaluation of image quality assessment
metrics and analyse their correlation with the subjective quality scores. In this
respect, and unlike recent studies [23, 24] focusing on traditional image quality
assessment (IQA) algorithms, we propose here to investigate new emerging IQA
methods based on neural networks.

The remainder of this paper is organized as follows. In Section 2, an overview of the retained neural networks based image compression algorithms is provided. Then, the subjective test methodology is described in Section 3. Finally, the objective metrics as well as the experimental results are discussed in Section 4 and some conclusions are drawn in Section 5.

## 2. Retained neural networks-based image compression algorithms

Since the focus of this paper is on the quality assessment of deep learning compressed image, we propose to select one conventional image compression method, which is often used as a comparison method, and four neural networks based compression methods. The conventional method is the standard JPEG2000 coding scheme [25] whereas the neural networks based ones are described in the following.

### 2.1. *End-to-end learned image compression models*

This method, developed by Ballé *et al.* [5], is among the first developed end-to-end image compression methods based on deep learning. Its block diagram is shown in Fig. 1.

Thus, the method consists of a nonlinear analysis transform $g_a$, a uniform quantizer $q$ and a nonlinear synthesis transform $g_s$. Moreover, a perpetual transform $g_p$ can be used to evaluate the quality of the decoded image. Note that the latter was set to the identity in [5]. For the analysis and synthesis transforms, they are performed using three convolution layers and nonlinear activation functions. More precisely, for the analysis (resp. synthesis) stage, each convolution layer is followed by downsampling (resp. upsampling) and generalized divisive normalization (approximate inverse of the generalized divisive normalization) operations. To optimize their network and find the optimal parameters of the analysis and synthesis transforms, the authors use a Rate-Distortion (R-D) loss
<figure><img src="image_3.png" /><figcaption>Figure 1: Block diagram of the baseline end-to-end image compression algorithm [5] (TDB by Tassnim).</figcaption></figure>

function given by

$$ 
\begin{aligned}
\mathcal{L}_1 &= R + \lambda D \\
&= -E[\log_2(P_q)] + \lambda E[d(\mathbf{x}, \hat{\mathbf{x}})]
\end{aligned} 
\quad (1) 
$$

where $P_q$ is the discrete probability distribution of the quantized vector, $d(\mathbf{x}, \hat{\mathbf{x}})$ is the distortion (typically the Mean Square Error) between the original and reconstructed images, and $E[\cdot]$ represents the expectation operation approximated by average over a given training set of images. For optimization purpose via gradient descent algorithm, $P_q$ will be approximated by the density function of $\tilde{\mathbf{y}}$ (denoted by $p_{\tilde{\mathbf{y}}}$) which is obtained by replacing the quantizer with an additive i.i.d uniform noise source. Based on this approximation of the quantized coefficient distribution, and given a probability model $p_{\tilde{\mathbf{y}}}$, the loss function becomes suitable for stochastic optimization.

The above architecture [5] has been then considered as a reference model in many other deep learning based image compression algorithms [6, 26, 27]. Among them, we will retain here the method proposed in [6]. The latter aims to extend the first model [5] by integrating an hyperprior $h$ that captures the spatial dependencies in the latent representation $\mathbf{y}$. The block diagram of this architecture is shown in Fig. 2.
<figure><img src="image_4.png" /><figcaption>Figure 2: Block diagram of the end-to-end image compression algorithm with a scale hyper-prior [6] (TDB by Tassnim).</figcaption></figure>

In this extended architecture, $h_a$ and $h_s$ can be seen as an auxiliary autoencoder that aims to estimate the probability distribution $p_{\hat{y}}$ after decoding $\hat{z}$. In this respect, different methods have been developed to model this distribution. More precisely, three variants of this generic autoencoder architecture have been retained in this work. The first approach uses a non-adaptive distribution model based on piecewise linear functions and was referred in [6] to as *factorized-prior* model. The second one, designated by *hyperprior* model in [6], assumes a zero mean Gaussian distribution with standard deviation parameters $\sigma^2$. The third approach corresponds to a more recent work where authors resort to a *Gaussian mixture model*.

## 2.2. Fully connected network for lifting based image coder

While the previous approaches as well as most of the developed neural networks based compression methods are not suitable for *lossy-to-lossless* coding applications, a novel method based on lifting schemes [28] has been recently developed in [17]. In this architecture, shown in Fig. 3, the traditional predictors and update linear operators are replaced by fully connected neural network (FCNN) models. More precisely, three FCNN based prediction models, denoted by $f_j^{(HH)}$, $f_j^{(LH)}$, and $f_j^{(HL)}$, are employed to generate the three detail wavelet
<figure><img src="image_5.png" /><figcaption>Figure 3: Block diagram of the fully connected network based lifting coding scheme [17].</figcaption></figure>

subbands oriented diagonally, vertically and horizontally. These FCNN models
are learned by minimizing the energy (i.e the l₂-norm) of the detail coefficients.
Then, an FCNN based update model, designated by $f_j^{(LL)}$, is used to generate
the approximation subband. The latter model is optimized by minimizing the
quadratic error between the approximation coefficients and those obtained using
an ideal low pass filter. More details regarding this approach can be found in
[17].

**3. Subjective study**

In this section, we describe the conducted subjective experiment to build
a new database with MOS for quality assessment of deep learning compressed
images.

**3.1. Image database**

Our database has been derived from 16 reference (i.e. uncompressed) images
taken from the standard Kodak PhotoCD dataset. The latter is composed of
color pictures of size 768 × 512 with various foreground/background contents
as it can be seen from some samples shown in Fig. 5. It should be noted here
that we selected the Kodak dataset since it is widely used to validate deep
learning-based image compression algorithms.

<figure><img src="image_6.png" /><figcaption>Figure 4: Some reference images taken from the kodak dataset.</figcaption></figure>

Based on these reference images, we applied the standard JPEG2000 image
compression algorithm as well as the four deep learning-based ones, described
in Section 2, to generate the different distorted images. Moreover, and in order
to generate different distortion levels, the retained compression methods are
performed at four bitrates (i.e four quality levels). Since distortion are more
visible at low and middle bitrates, the bitrates are set to 0.1, 0.15, 0.2, 0.3 bits
per pixel (bpp). Some examples of compressed images are shown in Fig. 5.
As a result, and compared to the recent subjective studies using small datasets
(with 7 reference images and 168 distorted ones) [23, 24], a larger one with 336 images (16 reference ones and 320 distorted ones) has been built.

<figure><img src="image_7.png" /><figcaption>Figure 5: Examples of compressed images from our subjective database.</figcaption></figure>

## 3.2. Test methodology

While single and double stimulus procedures have been widely used in the literature [29], we propose in this paper to follow the second one during the subjective quality assessment step. More specifically, we have selected Double stimulus impairment scale (DSIS) Variant I method [30] where a pair of images are displayed side-by-side with one of them being the reference and the other one is the distorted. During their experiments, the observers are asked to evaluate the distorted image in comparison to the reference one using a continuous linear impairment scale from 0 to 100. The scale was also marked with five equally-spaced adjectives corresponding to the level of impairment. These five levels from best (i.e 100) to worse (i.e 0) were imperceptible, perceptible but not annoying, slightly annoying, annoying and very annoying. In our study, we observed that 7-8 seconds are sufficient to provide a score to a given test image, resulting in a duration of around 40 minutes for the whole subjective evaluation process. The total number of observers participating in the subjective study is 21. The latter correspond to some senior researchers and mainly PhD students
(with an age ranging from 24 to 45 years). Before starting the subjective tests, a brief introduction of the goal of the study followed by a short training session (of about 5 minutes) is presented to each observer. Note that the images used in the training are different from those of the constructed subjective database. Moreover, the different images are randomly displayed while ensuring that two consecutive test images do not correspond to the same content. For each single test image shown in the center of the screen, the observer provides an opinion score of the picture visual quality by moving a slider along a continuous graded quality scale. A screenshot of the interface used in our experiments is shown in Fig. 6.

<figure><img src="image_8.png" /><figcaption>Figure 6: Subjective experiment interface.</figcaption></figure>

### 3.3. Subjective scores computation and analysis

Let us denote by $S_{i,j}$ the raw score assigned by the i-th observer to the j-th image. Thus, by taking the average of the scores given by the $N = 21$ observers, we obtain the Mean Opinion Score $MOS_j$ for each image with index number $j$:

$$MOS_j = \frac{1}{N} \sum_{i=1}^{N} S_{i,j} \qquad (2)$$

Once the subjective scores are collected, a screening of observers was firstly performed for outlier detection using the method described in the ITU-R-REC-BT.500-13 [30]. Following this procedure, no outlier was detected.
<figure><img src="image_9.png" /><figcaption>Figure 7: Distribution of all MOS score.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: Standard deviation vs MOS for all images with different compression methods.</figcaption></figure>

Fig. 7 illustrates the histogram for MOS distribution in our database. From the figure, we can firstly see that the data follows a uniform distribution. Moreover, there are no values at the extremities of the MOS scale especially at the higher end, depicting that none of the compressed images is perceptually similar to the pristine one. Finally, it is also important to note that all the compression methods have similar quality range. In addition, we have analyzed the variation
between the observers’ ratings. Indeed, Fig. 8 shows the standard deviations versus the MOS. Thus, it can be observed that the images with the best and the worst scores generally have lower variations as compared to those obtained in the middle range of MOS. This is expected since it is generally easy to score images that are very bad or very good. These analysis results indicate that our new deep learning based compressed image database could be considered as representative for testing IQA metrics.

# 4. Performance evaluation

In this section, we propose to evaluate the performance of various objective quality assessment metrics on the new deep learning based compressed image database.

## 4.1. *Objective image quality assessment metrics*

Unlike the recent works devoted to quality assessment of DL-based compressed images which considered only conventional IQA metrics, we propose here to cover a wide range of metrics by investigating machine learning as well as recent deep learning based metrics. In what follows, we will briefly describe these metrics.

### Conventional metrics

These metrics include some popular metrics often used in IQA as well as some recent ones. In addition to the most commonly used Peak Signal-to-Noise ratio (PSNR) metric, we have considered the following ones:

* Structure SIMilarity (SSIM) [31]: It is a widely used metric that measures the similarity between a reference image and the tested one based on their structural information.

* Visual Information Fidelity (VIF) [32]: This measure is considered as a Human Visual System (HVS) based method and aims to quantify the loss of image information to the distortion process. Note that the pixel domain implementation of the VIF has been used.
* Visual Saliency-based Index (VSI) [33]: It is based on the extraction of the visual saliency map which is then exploited as an image feature and used to characterize the relevance of a local image region.

* Perceptual image quality assessment using a Normalized Laplacian Pyramid (PIQA-NLP) [34]: It is based on local luminance subtraction and local gain control obtained after applying the Laplacian pyramid decomposition to images.

* Haar Wavelet-Based Perceptual Similarity Index (HaarPSI) [35]: It consists in using the wavelet representation of images to assess local similarities between them, as well as the relative importance of image regions.

It should be noted here that these metrics belong to the class of Full-Reference (FR) metrics which use the reference image to assess the quality of the tested one.

## Machine Learning (ML)-based metrics

Natural Scene Statistics (NSS) models followed by training are among the ML-based metrics which are often used in IQA studies. These metrics include:

* DIIVINE [36]: It aims to extract statistical features using Discrete Wavelet Transform (DWT). Then, Support Vector Machine (SVM) followed by Support Vector Regression (SVR) stages are used to predict the quality score of the tested image.

* BLIINDS-II [37]: It relies on a statistical model of local discrete cosine transform (DCT) coefficients and employs a probabilistic predictive model to train the features and predict the image quality.

* BRISQUE [38]: Unlike the two previous metrics where statistical features are extracted from DCT and DWT domains, BRISQUE operates in the spatial domain, and then uses SVM and SVR to predict the image quality score.

* NIQE [39]: Based on spatial domain NSS features, it consists in evaluating the image quality based on a multivariate Gaussian (MVG) fitting model. While this method does not involve SVM and SVR modules, it requires a training on pristine images to generate the parameters of the MVG model.

It should also be noted here that these metrics belong to the class of No-Reference (NR) metrics where the quality of a tested image is evaluated without
using the reference image.

## Deep Learning (DL)-based metrics

Recently, and motivated by the success of neural networks, deep learning based metrics have been developed. Among them, we have considered the following ones.

* Perceptual Image-Error Assessment through Pairwise Preference (PieAPP) [40]: A pairwise-learning approach is developed to predict the perceptual error between an original image and a tested one. A deep CNN is used to train an error estimation function and produce the perceptual error score.

* Deep Image Quality Measure for FR IQA (DIQaM) [41]: The method uses a Convolutional Neural Network (CNN) for feature extraction followed by a Fully Connected Neural Network (FCNN) for regression, yielding the quality score prediction.

* Region-Adaptive Deformable Network (RADN) [42]: It consists of a modified residual block, a patch-level attention one and a reference-oriented deformable convolution block. The latter are performed on different non-overlapping patches and the final quality score is obtained using a weighted average operation.
Note that these three DL based metrics belong to the class of Full-Reference (FR) metrics.

## 4.2. *Analysis of objective quality metrics*

### Experimental setup

While the conventional IQA metrics as well as the ML-based ones can be easily tested since their implementations are publicly available, that of DL-based metrics needs more care since only the pre-trained models and the code for the test phase are available. However, such models may not be appropriate for our DL based compressed images and so should be fine-tuned. In this respect, significant efforts have been made to perform the training phase and obtain the new models. To this end, and for each DL-based metric, we have used the default setting parameters (number of epochs, learning rate, optimizer) provided in its respective reference paper. It is important to note here that the obtained
training models will be made publicly available.

Moreover, for the ML and DL based metrics, 75% of the dataset is used for training while the rest is used for testing. For instance, since our new database is built from 16 reference images, we proposed to divide them into four non-overlapping subsets where each test subset is composed of 4 reference images (i.e 80 distorted images) and the remaining 12 reference images (i.e 240 distorted images) are used for training. This means that 4-fold cross validation has been considered in our simulations.

## Evaluation criteria

In order to judge the performance of the objective metrics against our benchmark subjective scores, we have used three different criteria. They are the Pearson Linear Correlation Coefficient (PLCC), Spearman Rank-Order Correlation Coefficient (SROCC) and Kendall Rank-Order Correlation Coefficient (KROCC). Before evaluating the correlation coefficients, a five-parameter logistic function, given by (3), is applied to the predicted scores to take into account for non-linear relation between MOS and the predicted scores

$$f(x) = \beta_1 \left( \frac{1}{2} - \frac{1}{1 + e^{\beta_2(x - \beta_3)}} \right) + \beta_4 x + \beta_5 \quad (3)$$

where $\beta_1, \beta_2, \beta_3, \beta_4$ and $\beta_5$ are the five model parameters which are obtained by minimizing mean square error between the MOS and the predicted scores.

## Correlation results

Tables 1, 2 and 3 provide the correlation values between the objective scores and the subjective ones in terms of PLCC, SROCC and KROCC, respectively. Thus, different observation can be made from these tables. First, regarding the conventional IQA metrics, it can be seen that the widely used PSNR metric leads to the lowest correlation values while the recent HaarPSI one presents higher correlations. Moreover, the ML-based metrics have lower correlation values overall. For instance, while some of them (especially BLIINDS-II and NIQE) can outperform some conventional metrics (like PSNR and SSIM) for JPEG2000 compressed images, ML-based metrics are generally less performant than the conventional metrics for the DL compressed images. This suggests that
this category of metrics are not suitable for quality assessment in the context of
neural networks based image compression. This may be explained by the fact
that these ML based metrics belong to the class of NR IQA methods and so they
are trained using only the distorted images. Finally, using the DL based metrics,
better correlation values are obtained. For instance, DIQaM outperforms all the
other metrics and yields the highest correlation values.

In addition, Fig. 9 illustrate the scatter plots of all the considered IQA metrics versus the MOS. Our results show again that the conventional PIQA_NLP and HaarPSI as well as the three DL based metrics have much higher correlations with MOS compared to the remaining metrics.

Finally, the best IQA metrics have been retained to evaluate their performance
with respect to the different quality levels (i.e coding rate) of distorted images.
More precisely, in addition to the standard PSNR and SSIM metrics often used
to assess the quality of deep learning based image compression algorithms, we
have considered VIF, VSI, HaarPSI, PieAPP and DIQaM. Fig. 10 shows the
correlation values of these metrics with respect to the four quality levels. These
plots confirm the results provided in the previous tables and show that DIQaM
outperforms the other IQA metrics. Moreover, two main important observations
can be made from this figure. First, while HaarPSI and DIQaM have led to close
correlation values (overall) in Tables 1, 2 and 3, it can be seen here that DIQaM
is more suitable than HaarPSI at low birtates (i.e higher distortion levels).
Moreover, the curve obtained with DIQaM shows small variations of correlation
values compared to other curves like PSNR, SSIM and VIF. This indicates that
the performance of DIQaM is independent of the coding rate and so has the
advantage to be a more consistent metric.

4.3. *Qualitative results*

To confirm again the limitations of standards PSNR and SSIM metrics,
often used in the evaluation of deep learning based image compression algo-
rithms, Fig. 11 illustrates some reconstructed images with their associated
PSNR, SSIM, HaarPSI and DIQaM metrics as well as the MOS. For exam-
<table><caption>Table 1: Pearson Linear Correlation Coefficient of different IQA metrics with MOS.</caption><thead><tr><th>Category</th><th>Metric</th><th>JPEG2000</th><th>AE-Factor [6]</th><th>AE-Hyp-GM [6]</th><th>AE-Hyp-GMM [10]</th><th>FCNN-LS [17]</th><th>Overall</th></tr></thead><tbody><tr><td rowspan="6">Classical</td><td>PSNR</td><td>0.7856</td><td>0.7943</td><td>0.8206</td><td>0.8063</td><td>0.8478</td><td>0.8409</td></tr><tr><td>SSIM</td><td>0.8127</td><td>0.9009</td><td>0.8971</td><td>0.9134</td><td>0.8444</td><td>0.8953</td></tr><tr><td>VIF-P</td><td>0.8482</td><td>0.9018</td><td>0.8951</td><td>0.9016</td><td>0.8737</td><td>0.9061</td></tr><tr><td>VSI</td><td>0.8781</td><td>0.9205</td><td>0.9351</td><td>0.9412</td><td>0.8710</td><td>0.9048</td></tr><tr><td>PIQA_NLP</td><td>0.8206</td><td>0.8777</td><td>0.8814</td><td>0.8939</td><td>0.8710</td><td>0.9012</td></tr><tr><td>HaarPSI</td><td>0.8763</td><td>0.9166</td><td>0.9207</td><td>0.9087</td><td>0.8917</td><td>0.9292</td></tr><tr><td rowspan="4">ML-based</td><td>BRISQUE</td><td>0.7035</td><td>0.8418</td><td>0.8299</td><td>0.5683</td><td>0.7787</td><td>0.7862</td></tr><tr><td>NIQE</td><td>0.9070</td><td>0.7552</td><td>0.7908</td><td>0.6912</td><td>0.5503</td><td>0.7712</td></tr><tr><td>DIIVINE</td><td>0.5986</td><td>0.8126</td><td>0.8864</td><td>0.7162</td><td>0.8301</td><td>0.7271</td></tr><tr><td>BLIINDS-II</td><td>0.8469</td><td>0.7664</td><td>0.7973</td><td>0.6543</td><td>0.7678</td><td>0.7313</td></tr><tr><td rowspan="4">DL-based</td><td>PieApp</td><td><b>0.9103</b></td><td>0.8635</td><td>0.8875</td><td>0.8727</td><td><b>0.9418</b></td><td>0.8958</td></tr><tr><td>DIQaM</td><td>0.8720</td><td><b>0.9425</b></td><td><b>0.9388</b></td><td><b>0.9472</b></td><td>0.9338</td><td><b>0.9347</b></td></tr><tr><td>RADN</td><td>0.8690</td><td>0.8688</td><td>0.8767</td><td>0.9082</td><td>0.9395</td><td>0.8777</td></tr><tr><td>BIECON</td><td>0.8180</td><td>0.8189</td><td>0.8464</td><td>0.7566</td><td>0.7900</td><td>0.8494</td></tr></tbody></table>

ple, from the images shown in the first row of Fig. 11(a), it can be observed that the AE-Factor [6] method leads to better subjective reconstructed quality compared to JPEG2000. However, the conventional PSNR and SSIM metrics obtained with JPEG2000 are higher to those obtained with the AE-Factor [6] method. Thus, these metrics are not appropriate to show the relevance of the DL based compression method. Unlike PSNR and SSIM, DIQaM metric show more coherent results well correlated with the human perception.

## 5. Conclusion and perspectives

In this paper, a new database of deep learning-based compressed images is built for quality assessment purpose. In this respect, in addition to the JPEG2000 compression standard, four recent neural networks based coding methods have been considered while using different coding rates. Then, after performing the subjective experiments, different categories of IQA metrics, including conventional, ML and DL metrics, have been evaluated. Our experiments confirm that the standard PSNR and SSIM metrics, usually used in the
<table><caption>Table 2: Spearman Rank-order Correlation Coefficient of different IQA metrics with MOS.</caption><thead><tr><th>Metric</th><th>JPEG2000</th><th>bmshjfactor</th><th>bmshjhyper</th><th>cheng</th><th>recimycbcr</th><th>Overall</th></tr></thead><tbody><tr><td>PSNR</td><td>0.7758</td><td>0.8040</td><td>0.8058</td><td>0.8284</td><td>0.8355</td><td>0.8322</td></tr><tr><td>SSIM</td><td>0.8091</td><td>0.8983</td><td>0.8903</td><td>0.9133</td><td>0.8496</td><td>0.8908</td></tr><tr><td>VIF-P</td><td>0.8478</td><td>0.8938</td><td>0.8855</td><td>0.9049</td><td>0.8609</td><td>0.9013</td></tr><tr><td>VSI</td><td>0.8831</td><td>0.9208</td><td>0.9262</td><td>0.9304</td><td>0.8610</td><td>0.9133</td></tr><tr><td>PIQA_NLP</td><td>0.8258</td><td>0.8783</td><td>0.8750</td><td>0.8922</td><td>0.8610</td><td>0.8966</td></tr><tr><td>HaarPSI</td><td>0.8776</td><td>0.9187</td><td>0.9060</td><td>0.9071</td><td>0.8878</td><td>0.9257</td></tr><tr><td>BRISQUE</td><td>0.6843</td><td>0.8272</td><td>0.7382</td><td>0.5037</td><td>0.7603</td><td>0.7780</td></tr><tr><td>NIQE</td><td>0.9042</td><td>0.6875</td><td>0.7478</td><td>0.6456</td><td>0.4963</td><td>0.7608</td></tr><tr><td>DIIVINE</td><td>0.6320</td><td>0.7978</td><td>0.8596</td><td>0.7088</td><td>0.8206</td><td>0.7211</td></tr><tr><td>BLIINDS-II</td><td>0.8653</td><td>0.6794</td><td>0.7544</td><td>0.5419</td><td>0.7809</td><td>0.7224</td></tr><tr><td>PieApp</td><td>0.9005</td><td>0.8699</td><td>0.8801</td><td>0.8507</td><td>0.9338</td><td>0.8956</td></tr><tr><td>DIQaM</td><td>0.8483</td><td>0.9346</td><td>0.9191</td><td>0.9184</td><td>0.9206</td><td>0.9343</td></tr><tr><td>RADN</td><td>0.8487</td><td>0.8618</td><td>0.8478</td><td>0.8794</td><td>0.9199</td><td>0.8696</td></tr><tr><td>BIECON</td><td>0.8178</td><td>0.7963</td><td>0.7868</td><td>0.6816</td><td>0.7904</td><td>0.8372</td></tr></tbody></table>

context of image and video coding, are not suitable for DL based compressed images, and promising results are obtained with a recent DL based metric referred to as DIQaM. Therefore, in the future, it would be interesting to further investigate new DL based metrics for the quality assessment of neural networks-based compressed images. It is worth pointing out that our new database with the subjective scores will be made publicly available to advance the future research works of IQA community. Moreover, the trained models, obtained with our DL based compressed database, will be of a great interest to researchers working on the design of neural networks-based image compression methods and who require to evaluate their compression methods using the DL based metrics retained in this paper.
<table><caption>Table 3: Kendall Rank-Order Correlation Coefficient of different IQA metrics with MOS.</caption><thead><tr><th>Metric</th><th>JPEG2000</th><th>bmshjfactor</th><th>bmshjhyper</th><th>cheng</th><th>recimycbcr</th><th>Overall</th></tr></thead><tbody><tr><td>PSNR</td><td>0.5816</td><td>0.6071</td><td>0.6114</td><td>0.6190</td><td>0.6455</td><td>0.6384</td></tr><tr><td>SSIM</td><td>0.5995</td><td>0.7192</td><td>0.7067</td><td>0.7411</td><td>0.6514</td><td>0.7091</td></tr><tr><td>VIF-P</td><td>0.6481</td><td>0.7093</td><td>0.7156</td><td>0.7321</td><td>0.6723</td><td>0.7267</td></tr><tr><td>VSI</td><td>0.7047</td><td>0.7520</td><td>0.7623</td><td>0.7639</td><td>0.6683</td><td>0.7388</td></tr><tr><td>PIQA_NLP</td><td>0.6313</td><td>0.6954</td><td>0.6908</td><td>0.7054</td><td>0.6683</td><td>0.7162</td></tr><tr><td>HaarPSI</td><td>0.7017</td><td>0.7550</td><td>0.7474</td><td>0.7282</td><td>0.7051</td><td>0.7629</td></tr><tr><td>BRISQUE</td><td>0.5317</td><td>0.6500</td><td>0.5625</td><td>0.3917</td><td>0.5958</td><td>0.5854</td></tr><tr><td>NIQE</td><td>0.7612</td><td>0.5625</td><td>0.6208</td><td>0.4917</td><td>0.3958</td><td>0.5698</td></tr><tr><td>DIIVINE</td><td>0.4773</td><td>0.6250</td><td>0.6917</td><td>0.5584</td><td>0.6125</td><td>0.5447</td></tr><tr><td>BLIINDS-II</td><td>0.7030</td><td>0.5125</td><td>0.5792</td><td>0.4042</td><td>0.6458</td><td>0.5458</td></tr><tr><td>PieApp</td><td>0.7570</td><td>0.7042</td><td>0.7208</td><td>0.6750</td><td>0.8250</td><td>0.7231</td></tr><tr><td>DIQaM</td><td>0.6737</td><td>0.8000</td><td>0.7792</td><td>0.7750</td><td>0.7917</td><td>0.7821</td></tr><tr><td>RADN</td><td>0.6862</td><td>0.7000</td><td>0.6875</td><td>0.7417</td><td>0.7917</td><td>0.7051</td></tr><tr><td>BIECON</td><td>0.6487</td><td>0.6208</td><td>0.6250</td><td>0.5375</td><td>0.6417</td><td>0.6590</td></tr></tbody></table>

References

[1] D. Taubman, High performance scalable image compression with EBCOT, IEEE Transactions on Image Processing 9 (7) (2000) 1158–1170.

[2] D. Liu, Y. Li, J. Lin, H. Li, F. Wu, Deep learning-based video coding: A review and a case study, ACM Computing Surveys 53 (1) (2020) 1–35.

[3] S. Ma, X. Zhang, C. Jia, Z. Zhao, S. Wang, S. Wang, Image and video compression with neural networks: A review, IEEE Transactions on Circuits and Systems for Video Technology 30 (6) (2020) 1683–1698.

[4] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, M. Covell, Full resolution image compression with recurrent neural networks, in: IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, Hawaiʻi, USA, 2017, pp. 5306–5314.
365

[5] J. Ballé, V. Laparra, E. P. Simoncelli, End-to-end optimized image compression, in: International Conference on Learning Representations, Toulon, France, 2017, pp. 1-27.

370

[6] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, N. Johnston, Variational image compression with a scale hyperprior, in: International Conference on Learning Representations, Vancouver, Canada, 2018, pp. 1-47.

[7] O. Rippel, L. Bourdev, Real-time adaptive image compression, in: International Conference on Machine Learning, Sydney, Australia, 2017, pp. 1-9.

375

[8] M. Li, W. Zuo, S. Gu, D. Zhao, D. Zhang, Learning convolutional networks for content-weighted image compression, in: Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 3214-3223.

380

[9] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, V. G. Luc, Generative adversarial networks for extreme learned image compression, in: International Conference on Learning Representations, New Orleans, Louisiana, USA, 2019, pp. 1-31.

[10] Z. Cheng, H. Sun, M. Takeuchi, J. Katto, Learned image compression with discretized gaussian mixture likelihoods and attention modules, in: IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 7936-7945.

385

[11] I. Schiopu, A. Munteanu, Macro-pixel prediction based on convolutional neural networks for lossless compression of light field images, in: International Conference on Image Processing, Athens, Greece, 2018, pp. 445-449.

[12] J. Li, B. Li, J. Xu, R. Xiong, W. Gao, Fully connected network-based intra prediction for image coding, IEEE Transactions on Image Processing 27 (7) (2018) 3236-3247.

390
[13] T. Dumas, A. Roumy, C. Guillemot, Context-adaptive neural network-based prediction for image compression, IEEE Transactions on Image Processing 29 (1) (2019) 679-693.

395

[14] D. Liu, H. Ma, Z. Xiong, F. Wu, CNN-based DCT-like transform for image compression, in: International Conference on Multimedia Modeling, Bangkok, Thailand, 2018, pp. 61-72.

400

[15] E. Ahanonu, M. Marcellin, A. Bilgin, Lossless image compression using reversible integer wavelet transforms and convolutional neural networks, in: Data Compression Conference, Snowbird, UT, USA, 2018, p. 1.

[16] H. Ma, D. Liu, R. Xiong, F. Wu, iWave: CNN-based wavelet-like transform for image compression, IEEE Transactions on Multimedia 22 (7) (2020) 1667-1679.

405

[17] T. Dardouri, M. Kaaniche, A. Benazza-Benyahia, J.-C. Pesquet, Optimized lifting scheme based on a dynamical fully connected network for image coding, in: International Conference on Image Processing, Abu Dhabi, United Arab Emirates, 2020, pp. 1-5.

410

[18] Z. Wang, H. R. Sheikh, A. C. Bovik, No-reference perceptual quality assessment of jpeg compressed images, in: IEEE International Conference on Image Processing, Rochester, NY, USA, 2002, pp. 477-480.

[19] Z. Wang, W. Wang, Z. Li, J. Wang, W. Lin, No-reference image quality assessment for compressed images based on dct coefficient distribution and psnr estimation, in: Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, Hollywood, CA, USA, 2012, pp. 1-4.

415

[20] H. R. Sheikh, A. C. Bovik, L. Cormack, No-reference quality assessment using natural scene statistics: Jpeg2000, IEEE Transactions on Image Processing 14 (11) (2005) 1918-1927.
420

[21] D. Minnen, G. Toderici, M. Covell, T. Chinen, N. Johnston, J. Shor, S. J. Hwang, D. Vincent, S. Singh, Spatially adaptive image compression using a tiled deep network, in: IEEE International Conference on Image Processing, Beijing, China, 2017, pp. 1–5.

425

[22] L. Theis, W. Shi, A. Cunningham, F. Huszar, Lossy image compression with compressive autoencoders, in: International Conference on Learning Representation, Toulon, France, 2017, pp. 1–19.

[23] Z. Cheng, P. Akyazi, H. Sun, J. Katto, T. Ebrahimi, Perceptual quality study on deep learning based image compression, in: IEEE International Conference on Image Processing, Taipei, Taiwan, 2019, pp. 719–723.

430

[24] G. Valenzise, A. Purica, V. Hulusic, M. Cagnazzo, Quality assessment of deep-learning-based image compression, in: International Workshop on Multimedia Signal Processing, Vancouver, BC, Canada, 2018, pp. 1–6.

[25] D. Taubman, M. Marcellin, JPEG2000: Image Compression Fundamentals, Standards and Practice, Kluwer Academic Publishers, Norwell, MA, USA, 2002.

435

[26] J. Lee, S. Cho, S.-K. Beack, Context adaptive entropy model for end-to-end optimized image compression, in: International Conference on Learning Representations, New Orleans, Louisiana, USA, 2019, pp. 1–20.

440

[27] H. Sun, Z. Cheng, M. Takeuchi, J. Katto, End-to-end learned image compression with fixed point weight quantization, in: International Conference on Image Processing, Abu Dhabi, United Arab Emirates, 2020, pp. 1–5.

[28] W. Sweldens, The lifting scheme: A custom-design construction of biorthogonal wavelets, Applied and Computational Harmonic Analysis 3 (2) (1996) 186–200.

445

[29] P. Korshunov, P. Hanhart, T. Richter, A. Artusi, R. Mantiuk, T. Ebrahimi, Subjective quality assessment database of hdr images compressed with jpeg
xt, in: 2015 seventh international workshop on quality of multimedia ex-
perience (QoMEX), IEEE, 2015, pp. 1–6.

[30] Methodology for the subjective assessment of the quality of television pictures, Recommendation, ITU-R BT.500-13 (Jan 2012).

450

[31] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE transactions on image processing 13 (4) (2004) 600–612.

[32] H. R. Sheikh, A. C. Bovik, Image information and visual quality, IEEE Transactions on image processing 15 (2) (2006) 430–444.

455

[33] L. Zhang, Y. Shen, H. Li, Vsi: A visual saliency-induced index for per-
ceptual image quality assessment, IEEE Transactions on Image processing
23 (10) (2014) 4270–4281.

460

[34] V. Laparra, J. Ballé, A. Berardino, E. P. Simoncelli, Perceptual image quality assessment using a normalized laplacian pyramid, Electronic Imaging 2016 (16) (2016) 1–6.

[35] R. Reisenhofer, S. Bosse, G. Kutyniok, T. Wiegand, A haar wavelet-based perceptual similarity index for image quality assessment, Signal Processing: Image Communication 61 (2018) 33–43.

465

[36] A. K. Moorthy, A. C. Bovik, Blind image quality assessment: From nat-
ural scene statistics to perceptual quality, IEEE Transactions on Image
Processing 20 (12) (2011) 3350 – 3364.

[37] M. A. Saad, A. C. Bovik, C. Charrier, Blind image quality assessment: A
naturalscene statistics approach in the dct domain, IEEE Transactions on
Image Processing 21 (8) (2012) 3339 – 3352.

470

[38] A. Mittal, A. K. Moorthy, A. C. Bovik, No-reference image quality assess-
mentin the spatial domain, IEEE Transactions on Image Processing 21 (12)
(2012) 4695 – 4708.
[39] A. Mittal, R. Soundararajan, A. C. Bovik, Making a completely blind image quality analyzer, IEEE Signal Processing Letters 20 (3) (2013) 909–912.

475

[40] E. Prashnani, H. Cai, Y. Mostofi, P. Sen, Pieapp: Perceptual image-error assessment through pairwise preference, in: IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 1808–1817.

[41] S. Bosse, D. Maniry, K.-R. Muller, T. Wiegand, W. Samek, Deep neural networks for no-reference and full-reference image quality assessment, IEEE Transactions on Image Processing 27 (1) (2018) 206–219.

480

[42] S. Shi, Q. Bai, M. Cao, W. Xia, J. Wang, Y. Chen, Y. Yang, Region-adaptive deformable network for image quality assessment, in: IEEE Conference on Computer Vision and Pattern Recognition, OnLine, 2018.
<figure><img src="image_11.png" /><figcaption>(a) PSNR</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) SSIM</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) VIFP</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(d) BRISQUE</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(e) NIQE</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(f) FSIM</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(g) BLIINDS</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(h) DIIVINE</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>(i) VSI</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>(j) HaarPSI</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>(k) NLP-DIST</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>(l) PieApp</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>(m) RADN</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(n) IQA_CNN</figcaption></figure>

<caption>Figure 9: Scatter plots of the different IQA metrics versus the MOS.</caption>
<figure><img src="image_25.png" /><figcaption>Figure 10: Performance of IQA metrics with respect to the compression quality level.</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>Figure 11: Some examples of reconstructed images with IQA metrics as well as the MOS.</figcaption></figure>