Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The submission aims to analyze deep neural network (DNN) features in terms of how well they measure the perceptual severity of image distortions. It proposes to characterize each DNN feature in terms of two well known properties of the human visual system: a) sensitivity to changes in visual frequency and b) orientation selectivity. Both properties are evaluated with respect to the known human Contrast Sensitivity Function (CSF) and measured empirically from the feature’s response to (oriented) sinusoidal gratings. The results are quantified by a composite score termed Perceptual Efficacy (PE).
In a set of comprehensive experiments (several pre-trained DNNs, several layers per DNN and two different datasets of distorted images with human perceptual quality annotations) it is demonstrated that feature representation consisting of a layer’s features with high PE better agree with human perceptual quality judgments than low PE feature representations from the same layer.

I believe the submission convincingly demonstrates a statistical association between the proposed PE score of a DNN feature and human perceptual quality assessments.
Though, it remains unclear whether the characteristics captured by the PE score are necessary or sufficient to explain the success of DNNs to guide image generation tasks by providing a perceptual loss function.
For that I believe it is necessary to demonstrate that the present empirical results can be used to improve results of an image generation task, e.g. super-resolution. 
Furthermore the limits of the PE score could be explored by hand-crafting image representations with maximal PE score and comparing their usefulness in guiding e.g. a super-resolution task compared to a pre-trained DNN.

Thus, overall I believe the submission reports interesting initial results but falls short of showing that they capture general properties that can be transferred to improving perceptual loss functions. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes an analysis of convolutional neural networks (CNNs) features the basis for making perceptual quality comparisons. The analysis is based on the proposed Perceptual Efficacy (PE) Score that measures spatial frequency and orientation selectivity of CNN features. The hypothesis put forward by the authors is that a CNN features with high PE score can be used to formulate a perceptual loss (Eq. 1) that correlates well with human image quality judgement. The authors use a dataset of human image quality judgements to assess their hypothesis. 

One issue I see with the hypothesis as stated is that in the definition of frequency selective features, the authors make use of
the Contrast Sensitivity Function (CSF) which quantifies the dependency of human perceptual characteristics on frequency. So in the definition of the PE score we have embedded knowledge of human perceptual sensitivity. Is it therefore surprising  that we see correlation between the high PE features and human judgements of quality? 

Experimental results: The scatterplot presented in Figure 4 does not say to me what the authors claim it should. I do not see a significant difference between the low-PE features and the high-PE features in terms of their correlation with human image quality judgement (as measures in this case by the DMOS). I also find the large table of number in Table 1 to be rather
impenetrable. I would recommend an alternative method of presentation to make the desired point.  

Clarity: There are many undefined terms and acronyms (eg. SISR, HVS are not defined, while DMOS and SROCC are not described). Also, the description of visual masking in Sec. 4.3 was confusing and difficult to follow. Otherwise the writing was reasonably clear.

Impact and significance: Overall, the findings of the paper are not terribly surprising and as discuss above, given the use of the CSF (quantifying human perceptual characteristics) in the definition of the CNN Perceptual Efficacy (PE) score, it would seem rather surprising that a correlations would not be found. As a result of this as well as the rather narrow nature of the study involved, I am inclined to think that the impact potential of this paper would be rather low. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I thank the authors for their detailed response. Some of my questions have been addressed in the rebuttal, but as far as I can tell very few modifications have been made to the paper: mainly an additional super-resolution experiment in the appendix, which goes in a good direction, but currently comes across as quite preliminary. So I think the paper still has most of the weaknesses I mentioned and thus it is not quite fit for publication. But I do encourage the authors to strengthen the experiments (for instance by addressing the points I raised, or by some other means) and resubmit elsewhere.

---

The paper proposes an approach to analyzing the properties of "perceptual metrics" used in deep learning image generation methods. "Perceptual metrics" are computed by measuring distances between images not in the pixel space, but i na feature space of a pre-trained cNN. The proposed analysis method, inspired by studies of human perception, is based on measuring the response of these features to sinusoidal gratings of varying frequency or orientation. Based on these responses, the paper proposes a "Perceptual Efficacy Score" that should measure the importance of certain feature in the feature maps for the performance of a perceptual metric. Experiments show that indeed distances measured between features with high score better correlate with human judgement of image similarity than distances between features with a lower score.

I find the paper quite interesting, but lean towards rejection at this point. This i smainly because the experiments seem somewhat anecdotal and incomplete, see below for further details.

Pros:
1) Application of methods from psychology/neuroscience to artificial neural networks is an interesting avenue of work. Moreover, better unsderstanding of "perceptual metrics" is of wide interest for various image processing applications.
2) The proposed score seems to indeed correlate quite well with the importance of features for human judgement of image similarity.
3) Presentation is mainly clear.

Cons:
1) Experiments are not very exhaustive and at times a bit confusing. For instance:
1a) Results are sometimes presented in a confusing way. In Figure 4 first of all it is not quite clear what points correspond to I guess each point is an image) and, second, it is not very obvious that the correlation is higher i none of the plots. In tables 1 and 2 it is confusing that different percentiles for H and L are used for different networks/layers. Is this based on some tuning? Then the tuning process should be clearly explained. Moreover, it might be useful to report the full curves of performance as a function of the percentage of features used.
1b) There are no baselines and there is not much justification of computing the "Perceptual Efficacy" score the way it is computed. What if one uses only the orientation-based score? Or only the frequency-based? What if one selects the most relevant features in a data-driven way (based on correlation on a training set)? What if one selects subsets of features randomly? 
1c) While the method is inspired by methods used for studying natural vision systems, there is no connection to human experiments. It would be interesting to see a comparison of frequency and orientation tuning of features in a CNN to human cells (as I understand, the latter should be available in prior works?).
1d) It would be great to see the selected features be used not only for offline image similarity assessment, but also for training image processing models - in the end, this has been the main use of "perceptual metrics". Do they lead to improved results?
1e) Since the paper is about (subjective) image quality, it might be useful to show some qualitative results, potentially in the appendix if space is an issue.

2) There are some issues with the presentation:
2a) I had a hard time understanding what exactly "Contrast Sensitivity Function" and "contrast masking" are.
2b) Minor issues:
- In the abstract: "trained object detection deep CNNs" - I guess image classification is meant
- Beginning of Section 2: "Section. 2", "convolution layer as collection channels"
- Section 4: "corresponds the the peak:
- Section 5.2 "Berkeley-Adobpe"
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In the article the authors propose to measure quality of CNN-features by quantifying the orientation tuning and spatial frequency sensitivity of the features. The underlying hypothesis is that properties of features in the human visual cortex are also indicators for quality in CNNs. The authors devise an experiment similar to experiments performed on mammals to check which features are active under which types of basic patterns. Afterwards, a loss-function is devised that uses proportions of the best or worst features according to the metrics and it is shown that features that have high values on the metrics also lead to good performance.
-----------------------------------------------------------------------------------------
I have a problem understanding some of the metrics used. In the introduction, the following claim is made:
"The first attribute is sensitivity to spatial frequencies at which there is minimal contrast masking in human visual perception". To my understanding, the metric to measure this is (2) in 4.2. a_m^k is to my understanding the average response of the feature when given an image with orientation-frequency f. therefore, the derivative should be "the change of activation under change of frequency". I feel unable to connect this with the initial hypothesis, as it does not mention change of frequencies. i would have expected the correlation between CSF(f) and a^k_m(f), 
did you have a reason why you did not chose correlation?

Similarly, for mu_2 in (3) you are using the maximum value. Is there a reason not to use the mean? in that case mu_2 would be the variance, which would be a natural measure for orientation selectivity.

------------------
Is there a way to make Table 1 more pleasing for the human eye wrt the discussion of the results?
review_title: Official Blind Review #3

Meta-review: This paper aims to analyze CNN representations in terms of how well they measure the perceptual severity of image distortions.  In particularly, (a) sensitivity to changes in visual frequency and (b) orientation selectivity was used. Although the reviewers agree that this paper presents some interesting initial findings with a promising direction, the majority of the reviewers (three out of four) find that the paper is incomplete, raising concerns in terms of experimental settings and results. Multiple reviewers explicitly asked for additional experiments to confirm whether the presented empirical results can be used to improve results of an image generation. Responding to the reviews, the authors added a super-resolution experiment in the appendix, which the reviewers believe is the right direction but is still preliminary.

Overall, we believe the paper reports interesting findings but it will require a series of additional work to make it ready for the publication.
Decision: Reject