Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper builds upon work from several fields (adversarial attacks, MAD competition, and Eigen-distortion analysis) which are analysis by synthesis methods used to generate small magnitude perturbations (by some pixel measure) that cause large change in model response (either in classification for adversarial attacks, or perceptual distance in MAD and Eigen-distortion).  The paper extends this to the field of no-reference image quality metrics, and compares their contribution with these other methods.  They successfully use this method to synthesize images that change the NR-IQA quality score significantly but are below human detection (with humans in the loop).  They use these images to quantify model to model performance, but also to elucidate how the construction of specific models leads to specific failure cases, which demonstrates one of the key values of analysis-by-synthesis.
strengths_and_weaknesses: This is a very strong paper that builds nicely on several previous works, but meaningfully advances the work and extends it to an application that was not previously accessible. They also derive key insights from application of their method that could lead to followup work to improve NR-IQA models. This paper fits nicely within the NeuRIPS audience sweet spot, mixing knowledge and techniques from perceptual science, signal and image processing and machine learning.

There are a few details of the human experiments that should be elucidated further, these will be included in questions below.  I would encourage the authors to work in luminances when quantifying perceptual quality (in human experiments and in IQA models) and not fixed code values which are less generalizable to different displays and viewing environments.


questions: Table 1 is incomprehensible on first read and should be reorganized for more clarity.  It shouldn't require reading several pages past the table to fully digest what it is communicating.  

Does the performance of UNIQUE actually improve in the face of the Brisque attacks over naive performance? (Same for Cornia under Unique attacks). What does that imply?

Why did the authors retrain the networks (at all), and why each of them on different datasets? 

In addition to the four FR-IQA models, it would be interesting to compare to pixel MSE as an isotropic (in the pixel space) baseline. This will tell you the most about the NR-IQA model in the specific (as all pixel directs are equally penalized), whereas the other results show you a combination of the effects of the FR-IQA model used as well as the NR-IQA model, which is still useful but requires more effort to properly interpret. 

More details about the exact lux of the illumination and the brightness of the displayed images on the displayed monitor would be useful.

For your derived value R, is there a theoretical upper bound that would provide satisfactory robustness?

Not a question, but this is a perfect encapsulation of the power of analysis by synthesis : 
"The primary observation is that the difference in perturbations provides useful clues on how they extract quality-aware features."  The results presented in the final figure are insightful and very interesting to ponder.

limitations: Yes the author's carefully place their work in the context of many other works and carefully point on the limitations and advantages of their contribution with respect to this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a detailed analysis of the vulnerability of commonly used no-reference image quality assessment (NR-IQA) metrics to perceptual attacks. To this end, the authors include four NR-IQA metrics, BRISQUE, CORNIA, Ma19, and UNIQUE, into their analysis. The possibility of perceptual attacks is explored with four full reference IQA metrics, Chebyshev distance, SSIM, LPIPS, and DISTS. Based on the study, the authors have come up with several important observations that shed light on the design flows of the specific NR-IQA methods.
strengths_and_weaknesses: Strengths

- First of its kind work to study the perceptual attack vulnerability of multiple NR-IQA methods
- The findings in the works point to the design flows of the existing NR-IQA methods while also giving insights into the need for developing NR-IQA methods that can well resist perceptual attacks.

Weaknesses

- The possibility of attacks that can simultaneously affect multiple NR-IQA metrics is not investigated

questions: 1. Based on the experimental observations, can the authors provide some suggestions on some easy tricks that one could adopt to mitigate the vulnerability of perceptual quality metrics?

2. What about generating attacks that could simultaneously affect multiple NR-IQA methods? If that is not possible, can one rely on multiple metrics and hope for improved/complete resistance to perceptual attacks?

limitations: Has been addressed to a satisfactory level.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission proposes a Lagrangian formulation to study the perceptual attack of no-reference image quality. By using the Lagrangian relaxation between the prediction error of NR-IQA and the perceptual constraints with FR-IQA, the perceptually imperceptible counterexamples are generated. With the psychophysical experiments, the analysis of the perceptual attack of NR-IQA is conducted.
strengths_and_weaknesses: Strengths:
1)	perceptual attack of NR-IQA is an interesting problem, which would benefit to the image quality assessment community.
2)	The visualization of the perturbations to the different NR-IQA methods shows the different characteristics, which is also interesting.

Weaknesses:
1)	It states the connections with the existing methods, while it is better to validate the connections by experiments. 
2)	It lacks comparisons, making it hard to judge its advancement.
3)	It is better to provide the statistic for the relationship between the Lagrange multiplier and the perceptual quality. 
4)	The original performance of R in Table 1 is positive infinity, and the following values are not informative. Equation 6 can be modified by adding a regular term to the denominator to make the original performance a positive real number.
5)	SSIM is a higher the better metric, and it should be stated in the text whether (1-SSIM) or -SSIM is used as the D

questions: 1）What is the basis for choosing these 6 images and their corresponding distortions？
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to investigate the perceptual robustness of NR-IQA models. It proposes a Lagrangian relaxation of the existing adversarial attack formula. Then it gives the algorithm for perceptually imperceptible counterexample generation. The experimental results show some findings.
strengths_and_weaknesses: Strengths: 
The problem is novel.
The problem and its relationship with adversarial attacks in classification are well described. The Lagrangian relaxation and the algorithm for perceptually imperceptible counterexample generation are easy to follow.


Weaknesses:
The motivations to investigate the perceptual robustness of NR-IQA models should be given. Why this problem is important?
The findings are a bit general. What are the insights for the application of NR-IQA metrics? Are there any suggestions that can be concluded from the findings to defend against the adversarial attack?

questions: 1. Why this problem is important?

2. Are there any suggestions that can be concluded from the findings to defend against the adversarial attack?

3. How do you make sure that the perturbation is below the just-noticeable difference? The examples computed by MAP estimation shown in Fig. 2 are clearly different from the original image. The examples of the proposed method are not given in the paper. So it is hard to judge whether the perturbation is noticeable or not.

4. What is the relationship between the four values in Italics and the value in parentheses for each group?

limitations: The limitations are not discussed in the paper.
review_title: null

Meta-review: metareview: All the reviewers are in agreement in their recommendations to accept the paper. The topic of the paper brings along many different sub-fields, including adversarial attacks and image quality assessment, and should be interesting to several folks in the community. There are several constructive comments by the reviewers that I’d encourage the authors to address, especially the ones asking for clearly state the motivation of this line of research, as well synthesis of ideas from the study that would enable ideas for better defense against such attacks.
confidence: Certain
Decision: Accept