Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop
Authors: Weixia Zhang
Dingquan Li
Xiongkuo Min
Guangtao Zhai
Guodong Guo
Xiaokang Yang
Kede Ma
Venue: NeurIPS 2022 Accept (2022)

Abstract: No-reference image quality assessment (NR-IQA) aims to quantify how humans perceive visual distortions of digital images without access to their undistorted references. NR-IQA models are extensively studied in computational vision, and are widely used for performance evaluation and perceptual optimization of man-made vision systems. Here we make one of the first attempts to examine the perceptual robustness of NR-IQA models. Under a Lagrangian formulation, we identify insightful connections of the proposed perceptual attack to previous beautiful ideas in computer vision and machine learning. We test one knowledge-driven and three data-driven NR-IQA methods under four full-reference IQA models (as approximations to human perception of just-noticeable differences). Through carefully designed psychophysical experiments, we find that all four NR-IQA models are vulnerable to the proposed perceptual attack. More interestingly, we observe that the generated counterexamples are not transferable, manifesting themselves as distinct design flows of respective NR-IQA methods. Source code are available at https://github.com/zwx8981/PerceptualAttack_BIQA.

Perceptual Attacks of No-Reference Image Quality Models with Human-in-the-Loop
Weixia Zhang
Dingquan Li
Xiongkuo Min
Guangtao Zhai
Guodong Guo
Xiaokang Yang!
Kede Ma
1 MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University
Network Intelligence Research Department, Peng Cheng Laboratory 3 Department of Computer Science and Electrical Engineering, West Virginia University
4 Department of Computer Science, City University of Hong Kong
{zwx8981, minxiongkuo, zhaiguangtao, xs edu en, 1idq01@pcl..... guodong - guo@mail. wvu. edu, kede www.mmaiiittttt tyu. edu.hk
Abstract
No-reference image quality assessment (NR-IQA) aims to quantify how humans perceive visual distortions of digital images without access to their undistorted references. NR-IQA models are extensively studied in computational vision, and are widely used for performance evaluation and perceptual optimization of man-made vision systems. Here we make one the first attempts to examine the perceptual robustness of NR-IQA models. Under Lagrangian formulation, we identify insightful connections of the proposed perceptual attack to previous beautiful ideas in computer vision and machine learning. We test one knowledge- driven and three data-driven NR-IQA methods under four full-reference IQA models (as approximations to human perception of just-noticeable differences). Through carefully designed psychophysical experiments, we find that all four NR- IQA models are vulnerable the proposed perceptual attack. More interestingly, we observe that the generated counterexamples are not transfcrable, manifesting themselves distinct design flows of respective NR-IQA methods. Source code are available at https: //g thub. com http://///weeeeeeaooooaaaaaaaaa
Introduction
Over the past decade, deep neural networks (DNNs) have revolutionized a wide range of computer vision and machine learning applications. For example, ResNet [1] reduced the object recognition error rate on ImageNet [2] to 3.56%, surpassing the performance of informed humans. A similar story is written in the field of image quality assessment (IQA), where DNN-base IQA models [3, 4, 5, 6] exhibit very high correlations with human opinions of perceptual quality on various subject-rated datasets [7, 8, 9, 5]. IQA models be roughly classified into full-reference IQA (FR-IQA) and no-reference IQA (NR-IQA) ones depending on the availability of the pristine undistorted image as reference. Representative FR-IQA methods include the Minkowski distance (i.e., the e,, -,-norm induced metric), the structural similarity (SSIM) index [10], the learned perceptual image patch similarity (LPIPS) method [3], and the deep image structure and texture similarity (DISTS) model [4], which are widely used for measuring signal fidelity and quality in various vision applications. NR-IQA
"Correpooding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

models [11] are extensively studied in computational vision, which mimic the human ability to judge the perceptual quality of test image without comparison to any reference image. NR-IQA plays an indispensable role in the design and optimization of real-world image processing algorithms.
Despite the remarkable achievements of DNN-based models, recent work has identified their vulnera- bility to adversarial perturbations [12. 13]. For example, in natural image classification, visually indistinguishable perturbation added natural image would mislead "top-performing" classifiers. This imperceptible perturbation is called below the just-noticeable difference (JND) of human per- ception in psychophysies, and is less relevant in image classification [14]. This because for the majority of practical adversarial attacks [15] considered in image classification (e.g., the foc -norm constrained attack), the allowable perturbations. even if above JNDs, do not alter the image semanties: they are preseergs, In the context of IQA, it becomes highly nontrivial to craft tabel-preserving attacks for two main reasons. First, for adversarial perturbations that are above JNDs, they are highly likely to be perceived as some form of visual distortions that lead to quality degradation (see Fig. 1). Second, computational prediction of JNDs for natural images [16] remains an open research problem as it depends the combination of the image content and the perturbation type, constrained by the psychophysical experimental conditions (e.g., the maximum time of visual inspection).
(b)
Figure 1: Visual comparison of (a) natural photographic image and (b) computer generated example from (a) by attacking BRISQUE [17] using the projected gradient method, under an E0o- norm constraint with radius of 8/255. It is clear that the semantic information of the image remains intact, while its perceptual quality degrades due to the introduced "mosquito" noise.
In this work, we take initial steps to examine the perceptual robustness of NR-IQA models. Our main contributions are threefold.
We propose a two-step perceptual attack for NR-IQA methods with human-in-the-lood First, we define the objective of the perceptual attack as a Lagrangian function of an FR-IQA model (as the "perceptual" constraint) and the NR-IQA model (to be examined). By varying the Lagrange multiplier, we generate series of perturbed images of different distortion visibility. Second, we ask human observers in a well-controlled psychophysical experiment to determine whether the perturbation of each image discriminable. The output of our perceptual attack is leading to the most significant change in quality prediction.
counterexample, that is below the JND while
We draw connections between the proposed perceptual attack and conventional wisdom in literature, including the Carlini-Wagner attack [18] computer vision, maximum a posterior (MAP) estimation in statistical signal processing, maximum differentiation (MAD) competition [19] in computational vision, and eigen-distortion neuroscience.
distortion analysis [20] in computational
We conduct an extensive experiment to examine four NR-IQA models, the knowledge-driver BRISQUE [17], the shallow learning-based based CORNIA [21], well the deep learning- based Ma19 [22] and UNIQUE [6] under four FR-IQA models, the Chebyshev distance (i.e., the -orm induced metric), SSIM, LPIPS, and DISTS (as approximations to human perception of JNDs). We arrive at several important observations, among which the most interesting one is that the proposed perceptual attack succeeds in fooling all four NR-IQA models, but the generated counterexamples are not transferable, manifesting themselves as distinct design flows of respective NR-IQA methods.

---

2 Related Work
In this section, we give review of NR-IQA models, and summarize representative adversarial attacks in classification, and discuss them in a broader context of "analysis by synthesis" [23, 24].
2.1 NR-IQA Models
Early NR-IQA models were designed to deal with specific distortion types, e.g., JPEG compres- sion [25] and JPEG2200 compression [26]. In the past decade, research on purposse NR-IQA become popular with the proposal of a variety of quality-awarr features based on natural scene statis- ties (NSS) [17, 11, 27J. Unsupervised feature learning such as codebook construction [21] was also explored. Since the work of [28], DNN-based methods began to revolutionize the field of NR-IQA. Towards developing more accurate NR-IQA models, recent research includes joint learning from multiple databases [6], active learning for improved generalizability [29], patch-to-picte to--ttuur learning for local quality prediction [30], and continual learning for handling streaming visual data [31].
2.2 Adversarial Attacks in Classification
Machine learning models are, for a long time, known to be vulnerable to adversarial examples [12, 32, 33, 34]: data samples that have been modified very slightly but enough to falsify a machine learning model. Most frequently, adversarial examples automatically generated by projected gradient-basee based methods. Szegedy et al. [13] used a box-constrained L-BFGS method to discover the prevalence of adversarial examples in DNN-based classifiers. Goodfellow al. [35] introduced fast gradient sign method (FGSM), which modifies samples along the steepest descent direction under the -orm constraint. Kurakin et al. [36] provided extension to FGSM, which iteratively generates adversarial examples with a small step size. Madry et al. [37] formulated adversarial training as robust optimization [38], where adversarial example generation corresponds to solving the inner maximization problem by projected gradient descent (PGD). The authors recommended a practical trick construct more transferable adversarial examples: adding small amount of uniform noise to each pixel of the initial image as a form of dequantization. Moosavi et al. [39] defined the adversarial attack as the process to find the minimal perturbation that leads to erroneous model behavior. Close to the present work, Carlini and Wagner [18] considered a Lagrangian relaxation of the nn-norm constrained optimization problem to search for adversarial examples. Laidlaw al. [40] switched to more perceptual FR-IQA model, LPIPS [3], under which adversarial training gives improved robustness to unseen attacks. Here, we consider the Lagrangian formulation in a different context (i.e., NQ-IQA). for a different purpose (i.e., encouraging generating samples below JNDs), and under different FR-IQA models (i.e., the Chebyshev distance, SSIM, LPIPS, and DISTS).
We conclude this section by putting adversarial attacks in a broader context of "analysis by synthesis", which is a core idea in the pattern theory by [24]. Analysis by synthesis suggests test a machine learning model in generative rather than discriminative ways, which is well demonstrated in the field of texture modeling [41]. Adversarial attacks can be seen as form of analysis by synthesis, where the machine learning model is tested on the automatically generated worst-case data samples, which are less likely to be included in the fixed and excessively reused test sets [42]. In the context of computational vision and neuroscience, the MAD competition [19], the eigen-distorie distortit analysis [20], and the controversial stimuli synthesis [43] all fall into the category of analysis by synthesis. As one of our main contributions, we will make insightful connections of the proposed perceptual attack to these methods.
3 Perceptual Attacks on NR-IQA Models
We first formulate the perceptual attack in NR-IQA, and then put it in proper context by making connections to previously related techniques from different engincering fields.
3

---

3.1 Problem Formulation
An adversarial attack aims to modify a benign data sample zo to x² such that the prediction of a machine learning model f.(-), parameterized by vector w, undesirably deviates from the ground- truth label с = fzo), where f(-) denotes the underlying true hypothesis. This problem can be generally formulated
3 arg max [(Io(), s.t. D(r, S T
(1)
where L(.-)) denotes the loss function for particular machine learning task, D(s-) is a signal fidelity metric to define the feasible set of perturbations, and 7 constrains the maximum magnitude of all possible perturbations. In image classification, one may adopt the cross- cross-entropy function as -(..)) (or simply the negative logit of the ground-truth category for untargeted attacks and the logit difference of the specified and the ground-truth categories for targeted attacks). lp ep-norm induced metrics are frequently used to implement D(:;)) We refer readers to [44, 45, 46, 47, 48, 49] for instantiations of Eq. (1) for attacking other vision tasks.
It is tempting to reuse Eq. (1) to instantiate the adversarial attack on NR-IQA models, where fa : IR + RR takes an image x E IR-M as input, and computes a real number as the quality estimate. Without loss of generality, we assume larger (x) indicates higher predicted quality. The true perceptual quality, (1), referred to as the mean opinion score (MOS) of æ; can be collected via a standard psychophysical experiment. As regression task, we may specify L(-, -) as some discrepancy function between the quality prediction of the perturbed image, fu, and the MOS of the initial image (zo), e.g., the mean squared error (MSE), (fw (z) - f(zo))² D(, can be implemented by any "perceptual" FR-IQA model.
We point out three caveats of this constrained optimization. First, existing FR-IQA models only provide a rough account for human perception of image quality. For example, the a-level set of D(...) w.r.t. zig, i.e., (zr zo) ax. may contain two images of drastically different quality, Computational models that are specifically designed for measuring JNDs [16, 50, 51, 52, 53, 54, 55), are only tested under simple perturbations (e.g., noise shaping and compression), and may have their own adversarial examples. As a consequence, the feasible set D(z, xg) < T is little likely to have the quality-preservie property. Second, working with an imperfect FR-IQA model requires setting a content-dependens threshold T(zo) to ensure a quality-preservinn
feasible set. However, this is a highly non-trivial task, and may result in trivial solution (e.g., T(xa) O due the inaceuracy of .). Third, solving the constrained optimization problem using PGD (even with careful step size scheduling) appears to converge very slowly (due to the high nonlinearity and nonconvexity of D). In addition, PGD tends to make the constraint tight with equality and thus is less likely to promote images that are below JNDs. All these motivate to consider Lagrangian relaxation of Problem (1):
x= arg max- D(z, + (x) f
(2)
where À > O is the Lagrange multiplier. An immediate advantage of Eq. (2) is that it allows simultaneous maximization of the discrepancy term measured by the MSE and minimization of the FR-IQA term as the fidelity constraint, which encourages perturbed images to be below JNDs. An alternative way of viewing Eq. (2) is that we perturb zo in the quality increase and decrease directions specified by the NR-IQA model fw. respectively, under the constraint of D(.-)) and we choose to optimize along the direction that leads to the best discrepancy- fidelity trade-off. Empirically, we find that, if xo is of poor quality, it is easier to expose counterexamples in the direction of quality improvement, and vice versa. Meanwhile, it is noteworthy that our discrepancy term relies on the MOS of the initial image, fzu), which permits naturally occurring failure examples. This is the case when f. makes a poor quality prediction of ro. leading to large discrepancy and high fidelity. Nevertheless, ((n) can be replaced by (xo) if they are close.
3.2 Perceptually Imperceptible Counterexample Generation
We describe a variant of the steepest ascent method to solve Problem (2). Given the initial image Tp. we first add to each pixel some noise sampled uniformly from discrete set (-1/255, 0, 1 1/55 to trigger the optimization [37]. We then compute the steepest ascent direction g of the objective w.r.t. 2, and take step of y along the direction of g. The pixel values of the intermediate image

---

Algorithm 1 Perceptually Imperceptible Counterexample Generation
Require: An NR-IQA model (-). an FR-IQA model D(:,-) as the fidelity measure, of K
hyperparameter values { and the step size
Input: An initial image zo
Output: A perceptually imperceptible counterexample x²
1: for i 1 K do
2: I + 20 + where e randomly sampled from (-1/255,
3:
while the maximum iteration is not reached do
4:
< X;
5:
Compute the objective value J D(x, zo) ((f - f
6:
Compute the steepest ascent direction +- arg max, 1 VzJ" j ||ll|p 1}
7:
Update the image x x + 4
8:
Clamp x into [0, 1]
9:
Yi -
10:
Quantize y, to a 24-bit color image
11: Identify the perceptually imperceptible counterexample x² from tw in psychophysical experiment
clamped into the valid range, i.e., [0, 1], whenever necessary. After the maximum number of iterations is reached, we quantize it to obtain 24-bit color image as the output.
By varying the Lagrange multiplier A, we are able to generate sequence of images with different distortion visibility. We next design a psychophysical experiment to identify the perceptually imperceptible counterexample, giving careful treatment to two subtleties. First, the distortion visibility of the generated image generally reduces with the decrease of "(=",,o) and the decrease of À (see Eq. (2)). but there exceptions due to the high dimensionality and non-convexity of the optimization problem. Therefore, standard psychophysica, staircase methods [56] that assume monotonicity w.r.t. the stimulus intensity may not be straightforwardly applied. Second, ideally, we would like to measure the nottinotiee distortion?, but practically, it is difficult to collect such human thresholds in reliable way unless excessive instructions given. This is especially true when Ty has already been severely distorted (e.g.. blurred or compressed), and is no easy task to determine whether the added perturbation further degrades the image quality.
We thus choose the yes-no task for screening perturbed images below JNDs [57]. Specifically, human participants undergo a series of trials, each consisting of pair of the perturbed image (corresponding to a certain X) and the initial image. For each trial, they must judge whether the two images look identical. A perturbed image is said to be below the JND if participants fail to distinguish it from the initial image 75% of the time. We identify the perceptually imperceptible counterexample as the perturbed image that 1) is below the JND and 2) causes the largest change in quality prediction. More details about the psychophysical experiment can be found in Sec. 4.1, and the procedure of counterexample generation of IQA models is summarized in Algorithm 1.
3.3 Connections to Previous Work
In this subsection, we make insightful connections to several beautiful ideas in signal processing, computational vision, and machine learning.
Carlini- Wagner attack [18] has the closest relationship the proposed perceptual attack on NR- IQA models with three differences. First, Carlini and Wagner chose to work with the Lagrangian formulation mainly from the perspective of computational convenience, while we arrive at the same formulation from different percepiual perspective: no computational models exist to reliably compute INDs. Second, for classification, there are look for adversarial examples, where C' is the number of categories. For NR-IQA regression, there are two such directions. Third, the Carlini- Wagner attack suffices to employ the binary search to find the desired Lagrange multiplier due the label-preserving property of the majority perturbations in image classification. contrast, the proposed perceptual attack may have to rely on carefully
directions for the Carlini- Carlini-Wagner attack to
In some situations, the visible differences may not be perceived as distortions by the human visual system (e.g.. image enhancement).
5

---

(a)
(b)
(c)
(d)
(c)
Figure 2: "Enhanced"' versions from the initial image (a) by maximizing BRISQUE (b), CORNIA
(c), Ma19 (d), and UNIQUE (e), respectively.
designed psychophysical experiments to determine the value of in NR-IQA, which is common practice in computational vision.
MAP estimation [58, 59, 60, 61] seeks estimate a latent image x² given a corrupted observation Zo by maximizing the posterior probability p(x|2o):
x² = arg max (oo = arg max log p(rola>) + A Alogp(z))
(3)
where (orp((r))a ax) is the data fidelity term implemented by an FR-IQA model (e.g., the MSE) and log p(z) is the image prior term to measure the naturalness of the image. A better prior leads to improved estimation performance. The added hyperparameter À is often manually tuned, where A == 1 makes Eg. (3) hold. It is widely acknowledged that ideal NR-IQA model must rely solely on knowledge of the appearance of natural undistorted images. This suggests that the NR-IQA method should embody an image prior model, and perhaps even that the quality predictions should be monotonically related to probability densities [22]. With such conceptual equivalence between NR-IQA methods and natural image priors, we that the proposed perceptual attack differs only from MAP estimation in the choice of X. Specifically, our attack sets a À to produce the image below the JND relative to Tg while causing the most significant change of f.) In contrast, MAP estimation tests p(æ) using a À that leads to the best estimation performance. More importantly, we shall come a conclusion: in order for NR-IQA models to work in MAP estimation, they must first survive the proposed perceptual attack as an easier task. Taking a step further, we may set À = +oc, which corresponds to the maximization of (.) alone as a way of performing image enhancement. Not surprisingly, all state-of NR-IQA models fail this task, generating images with annoying distortions but higher predicted quality scores (see Fig. 2).
MAD competition [19] is an efficient methodology for comparing computational methods of percep- tually discriminable quantities (e.g.. image quality). In the context of comparing two IQA models, f.))) and fz(.), MAD automatically synthesizes an image pair (x", yy") that are likely to falsify at least one IQA model in competition by solving
(x", yn) arg max f1(z) f1)),, s.t. fa(x) = fz(v) = f2(2o),
(4)
a.y
where zo is the initial image and fz(2z) specifies quality level. By doing so, obtain pair of counterexamples, on which (.) and fz(-) hold contradictory opinions: f.) predicts x² to have much better quality than y², while fz(-) treats them as images of identical quality. Similar as the proposed perceptual attack, psychophysical testing on (x", is required to declare which model is the winner. In contrast to our attack that is constrained in the set of images that look identical to ro according to human perception (approximated by an FR-IQA model), MAD operates at the level set of another IQA model fe(-) that contains images of the same predicted quality as zo. Moreover, MAD generates the best-quality and worst-quality images in terms of fi f.(.) to compare the relative performance of two IQA models, while we only test the perceptual robustness of a single NR-IQA model.
Eigen-distortion analysis [20] is computational method for comparing perceptual image repre- sentations. f : R.M + R-V. It computes the eigenvectors of the Fisher information matrix3 with the largest and smallest eigenvalues to account for the model-predicted most- and least-noticeable distortion directions, respectively. A psychophysical experiment based on standard staircase method
Asssumngg the additive white Gaussian noise in the response space, we can compute the Fisher information 2j øs where dr
matrix by Oz Ge
Øf is the Jacobian matrix.

---

is necessary to measure the ratio of the visibility thresholds induced by the two directions. When N (i.e.. f(+) can be an NR-IQA method). the nost-noticeable direction reduces to the gradient of w.r.t. the input image 0f/Or. Correspondingly, the least-noticeable direction (with an eigenvalue of zero) can be any M-dim unit vector living in the M 1-dim subspace orthogonal to Of f//rr. /Or. Taking consecutive small steps along least-noticeable directions creates a predicted imperceptible perturbation, which is subject to human verification. In this sense, the proposed perceptual attack aims to tackle a "dual" problem, generating true imperceptible perturbation to maximize the prediction discrepancy.
4 Experiments
In this section, we first set up the experiments, including descriptions of NR-IQA and FR-IQA models, details of the psychophysical experiment, and new evaluation metrics for measuring the perceptual robustness of NR-IQA models. We then present the quantitative and qualitative results accompanied by an in-depth analysis.
4.1 Experimental Setups
Choice of NR-IQA Models. We test four NR-IQA models that are believed to be representative in the field: BRISQUE [17], CORNIA [21]. Ma19 [22]. and UNIQUE [6]. BRISQUE is knowledge- driven model, extracting NSS from mean-subtracted contrastnnormalizer pixel values. CORNIA [21] relies on unsupervised learning of a visual codebook from image patches, followed by soft-assinnment coding and max pooling, to obtain image representations. As feature engineering not involved, CORNIA can be seen as a shallow learning-based method. Ma19 [22] is a four-layer DNN with generalized divisive normalization [62] as nonlinear activation, which is trained from a large number of corrupted image pairs without reliance on MOSs. UNIQUE [6] trains variant of ResNet-34 [1] on multiple IQA datasets to handle both synthetic and realistic camera distortions. We use the training codes provided by the original authors to re- re-train BRISQUE and CORNIA on LIVE [7], Ma19 [22] on our own collected dataset', and UNIQUE on six human-rated IQA databases [7, 63, 64, 9, 5, 65]. There is no overlap between all training sets and the initial images used to generate counterexamples.
As suggested by the Video Quality Experts Group [66], a four- nour-parameter logistic function can be adopted to compensate for the prediction nonlinearity, making different NR-IQA more comparable:
31 32
q fz(
+ 82,
(5)
f 22 12,1
1 exp
where indicates the function composition operation. 31 to B, are fitting parameters, where 3n and 32 determine the maximum and minimum mapping values. Empirically, we find that, for different NR-IQA models, the estimated 31 and 82 can be quite different. Thus, choose to manually enforce
31 = 10 and B2 = O. We consider the learned q()) as part of the NR-IQA model.
Choice of FR-IQA Models. We consider four FR-IQA models to approximate the perceptual distance between the initial and perturbed images: the Chebyshev distance, SSIM [10], LPIPS [3], and DISTS [4]. The Chebyshev distance constrains the maximum pixel difference within an -bal.. SSIM [10] is arguably the most successful "perceptual" metric that compares luminance, contrast, and structure, separately. LPIPS [3] computes the Euclidean distance between deep representations of two images, which shows reasonable effectiveness in explaining image quality. DISTS [4] the first FR-IQA method that unifies structure and texture similarity, and is competitive in perceptual optimization of various image processing tasks [67]. We apply the Chebyshev distance to each color channel, and take the maximum distance across three channels. We enable SSIM to be aware of color information by treating the color-to- togayyscall conversion as a fixed differentiable front-end, which allows the gradient to be back- proppparp to the input color image. Both LPIPS and DISTS are based on variants of VGG-16 16 [68].
Details of the Psychophysical Experiment. We collect twelve images as initializations from the publicly available LIVE IQA database [7] (see Fig. 3), with four basic distortions (i.e., JPEG
"We the same pristine-quale images in [22], but generate the training set with only four types of distortions overlapping with LIVE, i.e., white Gaussian noise, Gaussian blur. JPEG compression, and JPEG2000 compression.
In our implementation, negate SSIM to make it distance measure.
7

---

Figure 3: Twelve initial images from the LIVE database [7]. Images are cropped for improved visibility.
compression, JPEG2000 compression, Gaussian blur, and additive white Gaussian noise) in the field of IQA. In addition, we ensure that the selected images cover broad quality range with discriminable perceptual quality. For each of sixteen combinations of NR-IQA and FR-IQA models, and each of the twelve initial images, we set À to 32 values, and optimize the objective in Eq. (2) to generate 32 perturbed images. For the Chebyshev distance, we use the steepest ascent direction in &oo -norm, and for the remaining three FR-IQA models, we choose the steepest ascent direction in f--norm, namely the gradient direction. We set the step size to 10 and the maximum number of iterations to 200, respectively. As suggested in the BT. 500 recommendations [69], we carry out the experiments in an indoor office environment with a normal lighting condition (with approximately 200 lux) and without reflecting ceiling walls and floors. The peak luminance of the displayed images is mapped to 200 cd/ cd/m"?. We recruit fifteen human subjects (with normal or corrected prrrected-t-nonnee to-moalal vision) to participate in the psychophysical experiment, viewing the image pairs from fixed distance of twice the screen height. A training session is performed to familiarize each subject with the study. For each yes-no trial, subjects are shown (for one second each with half-second gray screen between images, and in randomized order) a perturbed image and the corresponding initial image, and then asked to determine whether the two images look visually different. All image pairs are displayed on a 24" LCD monitor at resolution of 1, 920 080. To avoid the fatigue effect, subjects are required to take a break during the experiment. In total, we generate 4 x 4 x 12 = 192 counterexamples to test the robustness of NR-IQA models.
Evaluation Metrics. Unlike image classification, it is generally difficult (or even pointless) for a regression task to define the success of adversarial attack a per-image basis. As image quality is a relative perceptual quantity, we adopt the Spearman rank-order correlation coefficient (SRCC) to measure the prediction monotonicity between model predictions and MOSs of 24 images (12 initial images and 12 corresponding counterexamples). In addition, we define the average ratio of maximum allowable change in quality prediction to actual change over counterexamples in logarithmic scale:
S
max (31 - (m (32
log
(6)
S
(z,) fw (x7))
where S is the number of initial images and x; denotes the i-th one. A larger R means better stability (i.e., robustness), but not necessarily better quality prediction performance as the MOS is not involved in the computation.
4.2 Main Results
We summarize the SRCC and average ratio results of NR-IQA models under intra-model attacks (i.e.. using counterexamples generated by the proposed attack with different FR-IQA models) and
8

---

Table 1: SRCC and average ratio results of NR-IQA models under mntra-model attacks
Metric
SRCC
A defined in Eq. (6)
FR-IQA
Chebyshev
SSIM
LPIPS
DISTS
Chebyshev
SSIM
LPIPS
DISTS
BRISQUE
0.0959
0.2023
0.1221
0.1308
0,4116
0.3824
0.4412
0.7392
CORNIA
0.1998
0.0663
0.2092
0.0139
0,9141
0.4181
0.3682
0.2197
Ma19
0.2441
0. 0.1447
0.0959
0,2005
0.6188
0.3914
0.3943
0.3828
UNIQUE -0.0994
-0.1064
-0.1813
-0.1726
0.2059
0.1539
0.0927
0.1229
Table SRCC and average ratio results of NR-IQA models under inter-model attacks. The number in the parentheses shows the original performance of the NR-IQA model
Attack
BRISQUE (0.9231)
CORNIA(0.9650)
Under Attack
CORNIA
Ma19
UNIQUE
BRISQUE
Ma19
UNIQUE
Chebyshev
0.8230
0.6103
0.7777
0.8596
0,9730
0,9660
SSIM
0.8684
0.9091
0.9276
0.9538
0,9790
0,9555
LPIPS
0.7725
0.8806
0.8806
0.9154
0,9451
0,9451
SRCC
DISTS
0.7254
0.8806
0.8806
0.8997
0.9660
0.9625
Attack
Ma19 (0.9091)
UNIQUE(0.9301)
Under Attack
BRISQUE
CORNIA
UNIQUE
BRISQUE
CORNIA
Ma19
Chebyshev
0.7725
0.8387
0.8422
0.9381
0,9224
0,9242
SSIM
0.8771
0,9102
0.9137
0.9346
0,9137
0.9276
LPIPS
0.8718
0.8387
0.8893
0.9276
0.9346
0,9346
DISTS
0.8387
0.8422
0.8858
0.9276
0.8945
0.9259
Attack
BRISQUE (+00)
CORNIA (4oo)
Under Attack
CORNIA
Ma19
UNIQUE
BRISQUE
Ma19
UNIQUE
R. defined in Eg. (6)
Chehyshev
2.1064
2.7651
2.4192
3.2428
4,3930
4.2005
SSIM
2.4097
3,1472
3,2371
3,3378
4.1283
4,2050
LPIPS
1.7400
3.1070
3.0490
3,5316
4,2963
4,6841
DISTS
1.8017
3.1368
2.9193
3,8798
3,8258
4.0432
Attack
Ma19 (++00)
UNIQUE
Under Attack
BRISQUE
CORNIA
UNIQUE
BRISQUE
CORNIA
Ma19
Chebyshev
2,4957
3.0544
3.1594
4,0762
3.6509
3,9139
SSIM
3.6393
4.3902
4,4728
4.6850
2.9966
4,2564
LPIPS
2.9033
3.0309
3,5103
4.4247
3.2835
4,9144
DISTS
3.6089
3.4737
3.5533
4.8580
2.6109
4.1454
inter-model attacks (i.e., using counterexamples originally spotted falsify another NR-IQA model) in Tables and 2, respectively. The results without perceptual attacks are also presented as reference. From the tables, we have a number of insightful observations.
First, the intra-attacks are capable of falsifying the corresponding NR-IQA models, no matter which FR-IQA model is adopted. This is evidenced by a catastrophic SRCC drop relative the original performance. We then conclude that none of the evaluated NR-IQA design philosophies (i.e., NSS- based, codebook-basd based, and DNN-based approaches), are inherently perceptually robust. Second, compared with intra-model attacks, NR-IQA methods appear much more robust to inter-model attacks. This manifests the poor transferability of perceptually imperceptible counterexamples. which fail to craft black-box attacks in NR-IQA. On the positive side, the generated examples by proposed perceptual attack are informative to reveal distinct design flows of different NR-IQA models, and may point out promising ways to improve model to combine the best aspects of multiple models. Third, when switching to the average ratio (R defined in Eq. (6)), we find that UNIQUE is the least stable model consistently across all FR-IQA methods, suggesting that the overparameterization of UNIQUE creates an abundant space for overfitting the current IQA datasets. Although the observations have been obtained from the expensive psychophysica experiment only twelve images, we empirically find these to be consistent across a wide range of images with different content and distortion complexities.
To compare the perturbations generated for different NR-IQA models in a more intuitive way, we visualize the absolute residual images (i.e., (|r xx' ) in Fig. 4, where DISTS is used as the image fidelity measure in Eq. (2). The primary observation is that the difference perturbations provides useful clues how they extract quality-award features. Specifically, perturbations for BRISQUE [17]

---

(a)
(b)
(c)
(d)
(c)
Figure Perturbations added to the initial image (a) by attacking BRISQUE (b), CORNIA (c), Ma19 (d), and UNIQUE (e), respectively.
mainly emerge in smooth and textured regions, where manipulation of individual and product of locally normalized luminances is much easier, way of cheating the built NSS model. The learned codebook in CORNIA [21] contains many Dirac delta functions of different locations and edge filters of different orientations, for which the selected texture and edge patches from maximum response and thus are used for quality computation. This may explain the perturbations in Fig. 4 (c) concentrated along the edges and in strong textures. The blocking perturbations in Fig. (d) appear primarily on the objects (e.g.. the lighthouse and the fence) and along strong edges (e.g.. the image borders due zero padding in DNNs). We believe this arises from the spatial pyramid pooling [70] layer used in Ma19 [22]. When the pyramid pooling is replaced with global average pooling, the perturbations by Ma19 resemble those Fig. (e) by UNIQUE. In addition, compare to BRISQUE and CORNIA that only accept grayscale images, the perturbations by Ma19 and UNIQUE occur in all color channels (not shown). Finally, nearly all pixel perturbations are less than 4/255, justifying the effectiveness of our psychophysical experiment to identify counterexamples that are below JNDs.
image give the
5 Conclusion
We have described perceptual attack NR-IQA models, based which their perceptual robustness has been thoroughly evaluated. The proposed attack has close connections to previous techniques. We found that neither the conventional knowledge-driver NR-IQA models nor the modern DNN-based methods are inherently robust to perceptually imperceptible perturbations. Moreover, the generated counterexamples by one NR-IQA model do not transfer in an efficient way to falsify other models, which contain valuable information to expose the design flows of respective models. We believe with the exploration of perceptual attacks of NR-IQA models, NR-IQA research may enter new and explosive development stage.
In the future, plan to develop perceptually robust NR-IQA methods. From the model construction perspective, we will identify and combine robust building blocks of different NR-IQA methods. From the optimization perspective, Meanwhile, we may draw inspiration from the connections between the proposed perceptual attack and previous analysis by synthesis techniques, and develop robust regularizers [71] facilitate training robust NR-IQA methods.
may directly employ adversarial training suggested by [37].
Acknowledgments and Disclosure of Funding
The authors would like to thank Chengguang Zhu for coordinating the psychophysical experiment and all subjects for participation. This work was supported in part by the Hong Kong RGC ECS Grant 21213821, the National Natural Science Foundation of China under Grants 62071407 and 61901262, and Shanghai Municipal Science and Technology Major Project (202ISHZDZX0102),
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognition, pages 248- 255, 2009.
[3] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[4] Keyan Ding, Kede Ma. Shiqi Wang. and Eero P. Simoncelli, Image quality assessment: Unifying structure and texture similarity. IEEE Transactions Pattern Analysis and Machine Intelligence, 445)22567-2581, May 2022.
[5] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe. Kon1Q-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE Transactions Image Processing, 29:4041-4056, Jan. 2020.
[6] Weixia Zhang, Kede Ma. Guangtao Zhai, and Xiaokang Yang. Uncertainty aware blind image quality assessment the laboratory and wild. IEEE Transactions on Image Processing, 30:3474 3486, Mar. 2021.
[7] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing, 443443334403345, Nov. 2006.
[8] Ponomarenko Nikolay. Jin Lina, Ieremeiev Oleg, Lukin Vladimir, Egiazarian Karen, Astola Jaakko, Vozel Benoit, Chehdi Kacem, Carli Marco, Battisti Federica, and C.-C. Jay Kuo. Image database TID2013: Peculiarities, results and perspectives. Signal Processing: Image Communication, 30:57-77, Jan. 2015.
[9] Deepti Ghadiyaram and Alan C. Bovik. Massive online crowdsourced study of subjective and objective picture quality. IEEE Transactions on Image Processing, 25(1):372-387, Jan. 2016.
[10] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4)6000 612, Apr. 2004.
[11] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20(3):209-212, Mar. 2013.
[12] Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, and Deepak Verma. Adversarial classification. In ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 99-108, 2004.
[13] Christian Szegedy, Wajeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representattons, 2014.
[14] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances Neural Information Processing Systems, pages 8312 8323, 2018
[15] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wicland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR. abs/1902.06705, 2019.
[16] Chun-Hsien Chou and Yun-Chin Li. A perceptually tuned subband image coder based on the measure of just-noticeaall-diiteee distortion profile. IEEE Transactions on Circuits and Systems for Video Technology. 5(6):467-476, Dec. 1995.
[17] Anish Mittal, Anush Krishna Moorthy, and Alan C Bovik. No-reference image quality assessment in the spatial domain. IEEE Transactions Image Processing, 21(12))4695-4708, Dec. 2012.
[18] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pages 39-57 39-57, 2017.
[19] Zhou Wang and Eero P. Simoncelli. Maximum diffcrentiation (MAD) competition: A methodology for comparing computational models of perceptual quantities. Journal of Vision, 8112):8.18.133, Sep. 2008.
[20] Alexander Berardino, Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. Eigen-distortion of hierarchical representations. In Advances in Neural Information Processing Systems, pages 3531-354 1531-3540, 2017.
11

---

[21] Peng Ye, Jayant Kumar, Le Kang, and David Doermann. Unsupervised feature learning framework for no-reference image quality assessment. In IEEE Conference on Computer Vision and Puttern Recognition, pages 1098-1105, 2012.
[22] Kede Ma. Xuelin Liu, Yuming Fang. and Eero P. Simoncelli. Blind image quality assessment by learning from multiple annotators. In IEEE International Conference on Imaging Processing, pages 2344-2348. 2019.
[23] David Mumford. Pattern theory: A unifying perspective. In First European Congress of Mathematics. pages 187-224, 1994.
[24] Ulf Grenander and Michael Miller. Pattern Theory: From Representation Inference. Oxford University Press, 2007.
[25] Zhou Wang, Hamid R Sheikh, and Alan C Bovik. No-reference perceptual quality assessment of JPEG compressed images. IEEE International Conference on Image Processing, pages 477-480, 2002.
[26] Pina Marziliano, Frederic Dufaux, Stefan Winkler, and Touradj Ebrahimi. Perceptual blur and ringing metrics: Application JPEG2000. Signal Processing: Image Communication, 19(2):633-72,, Feb. 2004.
[27] Deepti Ghadiyaram and Alan Bovik. Perceptual quality prediction authentically distorted images using a bag of features approach. Journal of Vision, 17(1):32-32, Jan. 2017.
[28] Le Kang, Peng Ye, Yi Li, and David Doermann. Convolutional neural networks for no-reference image quality assessment, IEEE Conference on Computer Vision and Pattern Recognition, pages 1733-1740, 2014.
[29] Zhihua Wang, Haotao Wang, Tianlong Chen, Zhangyang Wang, and Kede Ma. Troubleshooting blind image quality models in the wild. In IEEE Conference on Computer Vision and Puttern Recognition, pages 16256-16265, 2021.
[30] Zhenqiang Ying. Haoran Niu, Praful Gupta, Dhruv Mahajan, Deepti Ghadiyaram, and Alan C Bovik. From patches to pictures (PaQ-2-PiQQ) Mapping the perceptual space of picture quality. In IEEE Conference on Computer Vision and Puttern Recognition, pages 3572-3582, 2020.
[31] Weixia Zhang, Dingquan Li, Chao Ma. Guangtao Zhai, Xiaokang Yang, and Kede Ma. Continual learning for blind image quality assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, appear, 2022.
[32] Daniel Lowd and Christopher Meek. Adversarial Icarning. In ACM SIGKDD International Conference on Knowledge Discovery in Data Mining, pages 641 647, 2005.
[33] Battista Biggio, Giorgio Fumera, and Fabio Roli. Multiple classifier systems for robust classifier design in adversarial environments. International Journal of Machine Learning and Cybernetics, 1:27-41, Oct. 2010.
[34] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šindie, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 387-402. Springer, 2013.
[35] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[36] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In International Conference on Learning Representations Workshop. 2017.
[37] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[38] Aharon Ben-Tal Laurent El Ghaoui, and Arkadi Nemirovski. Robust Optimization. Princeton University Press, 2009.
[39] Seyed-Mohsen Moosavi-Dezfoole Alhussein Fawzi, and Pascal Frossard. DeepFool: A simple and accurate method to fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2574222,, 2016.
[40] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In International Conference on Learning Representations, 2021.
12

---

[41] Bela Julesz. Visual pattern discrimination. IRE Transactions on Information Theory, 8(2):84-92, Feb. 1962.
[42] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize ImageNet? In International Conference on Machine Learning, pages 5389-5400, 2019.
[43] Tal Golan, Prashant C Raju, and Nikolaus Kriegeskorte. Controversial stimuli: Pitting neural networks against each other as models human cognition. Proceedings of the National Academy of Sciences. 117(4))229330- 2930-299377720 2020.
[44] Chaowei Xiao, Ruizhi Deng, Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In Europeon Conference on Computer Vision, pages 217-234, 2018.
[45] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In JEEE International Conference on Computer Vision, pages 1369 1378, 2017.
[46] Yinpeng Dong, Hang Su, Banyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu. Efficient decision- based black-box adversarial attacks on face recognition. In IEEE Conference on Computer Vision and Puttern Recognition, pages 7714-7722, 2019.
[47] Jiahang Wang. Sheng Jin, Wentao Liu, Weizhong Liu, Chen Qian, and Ping Luo. When human pose estimation meets robustness: Adversarial algorithms and henchmarks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 11855-11864, 2021.
[48] Shuai Jia, Chao Ma, Yibing Song, and Xiaokang Yang. Robust tracking against adversarial attacks. In European Conference on Computer Vision, pages 69-84, 2020.
[49] He Wang. Feixiang He, Zhexi Peng, Tianjia Shao, Yong-Liang Yang, Kun Zhou, and David Hogg- Understanding the robustness of skeleton-based action recognition under adversarial attack. In IEFE Conference on Computer Vision and Pattern Recognition, pages 14656-14665, 2021.
[50] Xiaokang Yang, Weisi Lin, Zhongkang Lu, Ee Ping Ong, and Susu Yao. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication, 20(7):662 680, Aug. 2005.
[51] Anmin Liu, Weisi Lin, Manoranjan Paul, Chenwei Deng, and Fan Zhang. Just noticcable difference for images with decomposition model for separating edge and textured regions. IEEE Transactions on Circuits and Systems for Video Technology. 20(1 30 1668 Nov. 2010.
[52] Jinjian Wu, Guangming Shi, Weisi Lin, Anmin Liu, and Fei Qi. Just noticeable difference estimation for images with free-energy principle. IEFE Transactions on Multimedia, 15(7):1705-1710 Nov. 2013.
[53] Haiqiang Wang, Ioannis Katsavounidis, Jiantong Zhou, Jeonghoon Park, Shawmin Lei, Xin Zhou, Man-On Pun, Xin Jin, Ronggang Wang. Xu Wang. Yun Zhang. Jiwu Huang. Sam Kwong, and C.-C. Jay Kuo. VideoSet: A large-scale compressed video quality dataset based on JND measurement, Journal of Visual Commanication and Image Representation, 46:292-30 -02, Jul. 2017.
[54] Huanhua Liu, Yun Zhang, Huan Zhang, Chunling Fan, Sam Kwong, C-C Jay Kuo, and Xiaoping Fan. Deep learning-based picture-wise just noticeable distortion prediction model for image compression. IEFE Transactions Image Processing, 29:641-656 1656, Aug. 2019.
[55] Xuelin Shen, Zhangkai Ni, Wenhan Yang. Xinfeng Zhang, Shiqi Wang, and Sam Kwong. Just noticcable distortion profile inference; A pateh-level structural visibility learning approach. IEEE Transactions on Image Processing, 30:26-35, Nov. 2020.
[56] Tom Cornsweet. The stairease-method in psychophysics. The American Journal of Psychology. 75(3):485-491, 1962.
[57] Wilson P Tanner Jr and John A Swets. A decision decision-making theory of visual detection. Psychological Review, 61(6):401-4p) 409, 1954.
[58] Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, -0)154) 268. Nov. 1992.
[59] Stefan Roth and Michael J Black. Fields of Experts: A framework for learning image priors. In IEEE Conference Computer Vision and Pattern Recognition, pages 860-867, 2005.
13

---

[60] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In International Conference Computer Vision, pages 479-486, 2011.
[61] Lei Zhang and Wangmeng Zuo. Image restoration: From sparse and low-rank priors to deep priors. IEEE Signal Processing Magazine, 34(5):172-179, Sept. 2017.
[62] Johannes Ballé, Valero Laparra, and Bero P. Simoncelli. End-to-end optimized image compression. In International Conference Learning Representations, 2017.
[63] Eric C Larson and Damon M Chandler. Most apparent distortion: Full-reference image quality assessment and the role of strategy. Journal of Electronic Imaging, 19(1):1211, Jan. 2010.
[64] Alexandre Ciancio, A. L. N. T. Targino da Costa, E. A. B. da Silva, Amir Said, Ramin Samadani, and Pere Obrador. No-reference blur assessment of digital pictures based on multifeature classifiers. IEEE Transactions on Image Processing. 20(1):64-1) 75, Jan. 2011.
[65] Hanhe Lin, Vlad Hosu, and Dietmar Saupe. KADID-I0K: A large-scale artificially distorted IQA database. In International Conference on Quality of Multimedia Experience, pages 1-3, 2019.
[66] VQEG. Final report from the video quality experts group on the validation of objective models of video quality assessment, 2000.
[67] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli. Comparison of full-reference image qual- ity models for optimization of image processing systems. International Journal of Computer Vision, 129(4))1288-1281 Jan. 2021.
[68] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[69] ITU-R BT,500-14. Methodologies for the subjective assessment of quality of television images, 2019.
[70] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):100--1161, 2015.
[71] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theo- retically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482, 2019.
14


Reviews:
[{"review_id": "wNPjyy810Qf", "reviewer": "Reviewer_qeax", "summary": "This work presents a detailed analysis of the vulnerability of commonly used no-reference image quality assessment (NR-IQA) metrics to perceptual attacks. To this end, the authors include four NR-IQA metrics, BRISQUE, CORNIA, Ma19, and UNIQUE, into their analysis. The possibility of perceptual attacks is explored with four full reference IQA metrics, Chebyshev distance, SSIM, LPIPS, and DISTS. Based on the study, the authors have come up with several important observations that shed light on the design flows of the specific NR-IQA methods.", "questions": "1. Based on the experimental observations, can the authors provide some suggestions on some easy tricks that one could adopt to mitigate the vulnerability of perceptual quality metrics?\n\n2. What about generating attacks that could simultaneously affect multiple NR-IQA methods? If that is not possible, can one rely on multiple metrics and hope for improved/complete resistance to perceptual attacks?\n", "limitations": "Has been addressed to a satisfactory level.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- First of its kind work to study the perceptual attack vulnerability of multiple NR-IQA methods\n- The findings in the works point to the design flows of the existing NR-IQA methods while also giving insights into the need for developing NR-IQA methods that can well resist perceptual attacks.\n\nWeaknesses\n\n- The possibility of attacks that can simultaneously affect multiple NR-IQA metrics is not investigated\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mqFGNaTiJAF", "reviewer": "Reviewer_1irv", "summary": "This paper builds upon work from several fields (adversarial attacks, MAD competition, and Eigen-distortion analysis) which are analysis by synthesis methods used to generate small magnitude perturbations (by some pixel measure) that cause large change in model response (either in classification for adversarial attacks, or perceptual distance in MAD and Eigen-distortion).  The paper extends this to the field of no-reference image quality metrics, and compares their contribution with these other methods.  They successfully use this method to synthesize images that change the NR-IQA quality score significantly but are below human detection (with humans in the loop).  They use these images to quantify model to model performance, but also to elucidate how the construction of specific models leads to specific failure cases, which demonstrates one of the key values of analysis-by-synthesis.", "questions": "Table 1 is incomprehensible on first read and should be reorganized for more clarity.  It shouldn't require reading several pages past the table to fully digest what it is communicating.  \n\nDoes the performance of UNIQUE actually improve in the face of the Brisque attacks over naive performance? (Same for Cornia under Unique attacks). What does that imply?\n\nWhy did the authors retrain the networks (at all), and why each of them on different datasets? \n\nIn addition to the four FR-IQA models, it would be interesting to compare to pixel MSE as an isotropic (in the pixel space) baseline. This will tell you the most about the NR-IQA model in the specific (as all pixel directs are equally penalized), whereas the other results show you a combination of the effects of the FR-IQA model used as well as the NR-IQA model, which is still useful but requires more effort to properly interpret. \n\nMore details about the exact lux of the illumination and the brightness of the displayed images on the displayed monitor would be useful.\n\nFor your derived value R, is there a theoretical upper bound that would provide satisfactory robustness?\n\nNot a question, but this is a perfect encapsulation of the power of analysis by synthesis : \n\"The primary observation is that the difference in perturbations provides useful clues on how they extract quality-aware features.\"  The results presented in the final figure are insightful and very interesting to ponder.\n", "limitations": "Yes the author's carefully place their work in the context of many other works and carefully point on the limitations and advantages of their contribution with respect to this work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very strong paper that builds nicely on several previous works, but meaningfully advances the work and extends it to an application that was not previously accessible. They also derive key insights from application of their method that could lead to followup work to improve NR-IQA models. This paper fits nicely within the NeuRIPS audience sweet spot, mixing knowledge and techniques from perceptual science, signal and image processing and machine learning.\n\nThere are a few details of the human experiments that should be elucidated further, these will be included in questions below.  I would encourage the authors to work in luminances when quantifying perceptual quality (in human experiments and in IQA models) and not fixed code values which are less generalizable to different displays and viewing environments.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iT1Pm_rHgc9", "reviewer": "Reviewer_4XSx", "summary": "This paper aims to investigate the perceptual robustness of NR-IQA models. It proposes a Lagrangian relaxation of the existing adversarial attack formula. Then it gives the algorithm for perceptually imperceptible counterexample generation. The experimental results show some findings.", "questions": "1. Why this problem is important?\n\n2. Are there any suggestions that can be concluded from the findings to defend against the adversarial attack?\n\n3. How do you make sure that the perturbation is below the just-noticeable difference? The examples computed by MAP estimation shown in Fig. 2 are clearly different from the original image. The examples of the proposed method are not given in the paper. So it is hard to judge whether the perturbation is noticeable or not.\n\n4. What is the relationship between the four values in Italics and the value in parentheses for each group?\n", "limitations": "The limitations are not discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThe problem is novel.\nThe problem and its relationship with adversarial attacks in classification are well described. The Lagrangian relaxation and the algorithm for perceptually imperceptible counterexample generation are easy to follow.\n\n\nWeaknesses:\nThe motivations to investigate the perceptual robustness of NR-IQA models should be given. Why this problem is important?\nThe findings are a bit general. What are the insights for the application of NR-IQA metrics? Are there any suggestions that can be concluded from the findings to defend against the adversarial attack?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "clRhX0Q2AQm", "reviewer": "Reviewer_Hmqo", "summary": "This submission proposes a Lagrangian formulation to study the perceptual attack of no-reference image quality. By using the Lagrangian relaxation between the prediction error of NR-IQA and the perceptual constraints with FR-IQA, the perceptually imperceptible counterexamples are generated. With the psychophysical experiments, the analysis of the perceptual attack of NR-IQA is conducted.", "questions": "1）What is the basis for choosing these 6 images and their corresponding distortions？", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1)\tperceptual attack of NR-IQA is an interesting problem, which would benefit to the image quality assessment community.\n2)\tThe visualization of the perturbations to the different NR-IQA methods shows the different characteristics, which is also interesting.\n\nWeaknesses:\n1)\tIt states the connections with the existing methods, while it is better to validate the connections by experiments. \n2)\tIt lacks comparisons, making it hard to judge its advancement.\n3)\tIt is better to provide the statistic for the relationship between the Lagrange multiplier and the perceptual quality. \n4)\tThe original performance of R in Table 1 is positive infinity, and the following values are not informative. Equation 6 can be modified by adding a regular term to the denominator to make the original performance a positive real number.\n5)\tSSIM is a higher the better metric, and it should be stated in the text whether (1-SSIM) or -SSIM is used as the D\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept