Analysis and Interpretation of Deep CNN Representations as Perceptual Quality Features
Authors: Taimoor Tariq
Munchurl Kim
Venue:  (2020)
Keywords: interpretation
perceptual quality
perceptual loss
image-restoration.

Abstract: Pre-trained Deep Convolutional Neural Network (CNN) features have popularly been used as full-reference perceptual quality features for CNN based image quality assessment, super-resolution, image restoration and a variety of image-to-image translation problems. In this paper, to get more insight, we link basic human visual perception to characteristics of learned deep CNN representations as a novel and first attempt to interpret them. We characterize the frequency and orientation tuning of channels in trained object detection deep CNNs (e.g., VGG-16) by applying grating stimuli of different spatial frequencies and orientations as input. We observe that the behavior of CNN channels as spatial frequency and orientation selective filters can be used to link basic human visual perception models to their characteristics. Doing so, we develop a theory to get more insight into deep CNN representations as perceptual quality features. We conclude that sensitivity to spatial frequencies that have lower contrast masking thresholds in human visual perception and a definite and strong orientation selectivity are important attributes of deep CNN channels that deliver better perceptual quality features. 

Under review as a conference paper at ICLR 2020
ANALYSIS AND INTERPRETATION OF DEEP CNN REP- RESENTATIONS AS PERCEPTUAL QUALITY FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Pre-trained Deep Convolutional Neural Network (CNN) features have popularly been used as full-reference perceptual quality features for CNN based image quality assessment, super-resolution, image restoration and a variety of image- to-image translation problems. In this paper, get more insight, human visual perception to characteristics of learned deep CNN representations novel and first attempt interpret them. We characterize the frequency and orientation tuning of channels in trained image classification deep CNNs (e.g.. VGG-16) by applying grating stimuli of different spatial frequencies and orienta- tions as input. We observe that the behavior of CNN channels as spatial frequency and orientation selective filters can be used link basic human visual perception models to their characteristics. Doing so, we develop theory to get more in- sight into deep CNN representations as perceptual quality features. We conclude that sensitivity to spatial frequencies that have lower contrast masking thresholds in human visual perception and a definite and strong orientation selectivity are important attributes of deep CNN channels that deliver better perceptual quality features.
link basic
INTRODUCTION
Quantifying human perception of image quality has been a subject of significant research for quite some time. Full-reference objective metries such as the PSNR (Peak Signal Noise Ratio) and SSIM (Structural Similarity Index) (Wang et al.] (2004)), being fair metrics of distortion between two images, are not a satisfactory metrics to measure differences in perceptual quality. Considering the recent interest in the applications of deep CNNs in perception-oriented problems such as super- resolution, retterestoo,tion, crame-inerpolation and style-transfer etc, research into effective loss metrics that quantify perceptual quality and help train CNNs in delivering better perceptual quality has become paramount.
The perceptual loss proposed by Johnson al. (2066) was of the first demonstrate how ef- fective the feature representations of pre-trained image classification CNNs could be as features of full-reference perceptual quality, especially when incorporated into loss functions for image restora- tion. The perceptual loss is now popularly adopted in many image restoration problems such as super-resolution, style transfer, denoising etc. (Ledig et al. (2017) Wang al. (2018) Gatys et al. 2016)). Zhang et al. (2018) and Blau & Michaeli (2018) further demonstrate how effective deep CNN representations can be as features of perceptual quality, but without any analysis into their characteristics. More recently. [Mchreez et al. (2018) proposed variation of the perceptual loss called the contextual loss. which still employs deep CNN features as perceptual quality features but uses an approximation of the KL-ddiergencee to quantify distance. The contextual loss has been demonstrated to be quite effective maintaining natural image statistics during SISR (Single-Image Resolutoo). The recent PIRM Super- Resolution Challenge Report (Blau et al. (2018)) clearly iterates that the perceptual loss and the contextual loss are the most widely used loss functions for CNN based perceptual image Super-Resolution.
Nevertheless, like most applications of deep learning, there has been little or no effort to understand and interpret the role of deep CNN representations effective perceptual quality features. This is quite understandable, as it is difficult to find a direction to approach this problem from. Neural networks are non-linear, which makes tractable analysis tricky. Furthermore, human perception of

---

Under review as a conference paper at ICLR 2020
quality is also something that is still not understood completely. Most of our basic understanding of human visual perception of quality is in the frequency domain, with models such as the Contrast Sensitivity Function (CSF) (de Faria et al. (1998)) To make connection between deep CNN features and human perception, it is important to realize that deep CNN channels are essentially complex spatial frequency and orientation selective filters.
We stimulate pre-trained image classification CNNs with sinusoidal grating stimuli, record the re- sponse in the form of mean activation of each channel as function of spatial frequency/orieenation of input grating, thus quantifying the frequency and orientation selectivity of different channels. This approach makes it significantly easier to establish connection between perception models such as the CSF with learned deep feature representations. We hypothesize that two attributes are im- portant for deep CNN channels that are good perceptual quality features. The attributes are based on visual masking in human visual perception, which refers to human ability to perceive distor- tions and contrast visual stimulus. The first attribute is sensitivity to spatial frequencies which there is minimal contrast masking in human visual perception (Nadenau al. (2000), making the CNN channel sensitive to highly perceivable distortions. The second attribute being definite and strong orientation selectivity, which helps the channel respond better image regions with less pat- tern complexity, where there is less masking for distortions from a perceptual standpoint (Wu et al. (017))
We verify our hypothesis by designing an Objective Quality Assessment (OQA) experiment (Sheikh) al. (2006)) OQA experiments correlate the performance of any quality metric with human per- ception of quality, which is an accepted and standard experimental technique. We group the set of channels in different CNN layers into subsets on the basis of our hypothesis and demonstrate that the group which has channels with our described attributes, delivers a much better as a set of per- ceptual quality features. We repeat our experiment across multiple layers of many pre-trained image classification networks such as the VGG-16 (Simonyan & Zisserman (2014)), AlexNet (Krizhevsky Hinton 2012)), ShuffleNet (Zhang et al. (2017)) and SqueezeNet (landola al. (2017)).
2 DEEP CNN REPRESENTATIONS AS PERCEPTUAL QUALITY FEATURES
The main motivation behind using pre-trained image classification deep CNN representations as perceptual quality features is that instead of distance measure between two images being good FR metric, computing distance after non-linear transformation of images into a high dimensional manifold, might result in better perceptual quality measure. The high dimensional manifold in this case is the manifold of pre-trained CNN features. The general form for the perceptual loss (Johnson) al. 2016)) is given by Eq. (1)
M
4
Σ m=11
1 100h (Lour) - Φ " (IGT)
M
H
Where *Tp is the feature map corresponding the "m channel in the tk layer which 'M' number of total channels with feature map dimensions WH-W". As mentioned before, applying pre- trained deep CNN representations as perceptual quality features has proven be quite effective in FR-IQA methods (Bosse et al. (2017)), image restoration (Wang et al. (2018)) and style transfer (Gatys et al. (2166) problems, as iterated by Blau et al. (2018).
However, little else is known of the ability and characteristics of deep CNN representations as per- ceptual quality features. In this work, using basic human perception models, we aim get more insight into the role of pre-trained deep representations as perceptual quality features.
3 PROBLEM FORMULATION
Section. 2 iterates the motivation and wide spread use of pre-trained deep CNN representations as features of full-reference perceptual quality. However, there has been no effort to explain and interpret the role of deep representations as perceptual features. We consider CNN convolution layer as collection channels which deliver perceptual quality features. For example, the relu3 layer of the VGG-16 has 256 channels. Are all of the channels equally effective in delivering good
2

---

Under review as a conference paper at ICLR 2020
Spatisl Frequenyy
Orientation
150
800
100
10
20
30
20
30
Convl
RelU3
123
Conv1
ReLU3
VGG-16
VGG-16
Spatial Frequency Tuning
Input Gratings (0-30 cyc/deg)
Input Gratings (-90 deg)
Orientation Tuning
Experimental Setup. The network is stimulated by gratings of varying spatial frequency. The responses of different feature maps are recorded as activation vs spatial frequency data. To quantify orientation tuning, the network is stimulated by gratings of fixed spatial frequency and varying orientations to record mean activation vs orientation data.
Figure 1:
perceptual quality features? Are some channels better than others and so, what attributes make them better?
The problem in question is important in explaining the role deep CNN representations as percep- tual quality features, but it is somewhat difficult to approach because of the 'black-box' nature of neural networks. In section. 4.1, we will introduce a methodology to quantify the spatial frequency and orientation tuning of channels in pre-trained CNNs. Using this formulation, and explain deep CNN features as perceptual quality features by making use basic human visual perception models, which rely on spatial frequency and orientation characteristics of input stimuli. In essence, the formulation in Section. 4.1 will act as a bridge to link attributes of deep representa- tions and basic visual perception.
will interpret
OrientationTuniin
Spetiel Frequency Tuning
RetU1
1942
ReLU1
2.40
RetUz_2
ReLu2
ReLU3
ReLuB 2
xe
Relu4.2
Relu4
Retus_2
ReLUs_2
(a) Spatial Frequency Tuning
(b) Orientation Tuning
Figure 2: Characterizing spatial frequency and orientation tuning in channels across different layers of the pre-trained VGG-16.
3

---

Under review as a conference paper at ICLR 2020
4 A PSYCHOVISUAL APPROACH
4.1 FREQUENCYY/RRIENAATOO TUNING QUANTIFICATION
Our experimental method is inspired by the grating stimulus experiments used by neuro-scientists to study human visual perception characteristics (Kulikowski et al. (1982)). We aim to quantify both the spatial frequency and orientation tuning of different channels in the pre-trained CNN.
To quantify the spatial frrquency tuming, generate concentric sinusoidal gratings of fixed con- trast and varying spatial frequencies (cycles per degree), use them to stimulate pre-trained image classification CNNs and record the responses of the feature maps in the form of mean activation ver- sus spatial frequency for each channel. Fig. illustrates the overall scheme of measuring the spatial frequency responses of channels in various convolution layers of the trained VGG-16 network. The reason we are using a concentric pattern is to eliminate the factor of orientation selectivity from this part of the analysis. Some concentrie grating stimulus patterns are shown input to the trained VGG-16 16 network in Fig. (.()).
To quantify orientation selectivity low contrast masking thresholds, we stimulate the pre-trained network with linear pattern sinusoidal gratings with different orientations. The gratings have fixed spatial frequency, which corresponds sample grating patterns are shown in Fig. 1. Sample observations of orientation selectivity for channels in different layers of the pre- pre-trained VGG-16 are shown in Fig. 2.(b).
the peak of the Contrast Sensitivity Function (CSF). Some
4.2 VISUAL FREQUENCY SENSITIVITY
In this section, we will use the spatial frequency selectivity quantification in section. 4.1 to intro- duce the concept of visual frequency sensitivity. Human perception of images is largely dependent on attributes of input stimulus. significant proportion of neuro-science research advocates the role of the early visual cortex as spatial frequency analyzer (Maffei & Fiorentini (1973)). Hu- man perception characteristics are dependent on spatial frequency and one the most significant models that quantifies this characteristic is called the Contrast Sensitivity Function (CSF). The CSF depicts human ability to perceive contrast changes as a function of spatial frequency. The spatial frequencies where the CSF has higher value, correspond lower contrast masking thresholds in perception (ease in perceiving distortions and contrast changes). In essence, this corresponds to a higher probability of perceiving distortions at high CSF valued spatial frequencies.
Considering the presented analysis on the spatial frequency selective behavior of deep feature maps. Our hypothesis is that the deep representations that are more sensitive high CSF valued spatial frequencies, can be better features of perceptual quality. Fig. 3 plots mean activation of two channels versus spatial frequency of the input grating. Feature Map-2 can be seen to have a higher sensitivity compared to Feature Map-1 I at high CSF valued spatial frequencies, making Feature Map-2 more sensitive to distortions corresponding low contrast masking threshold regions in input images.
We model this attribute quantitatively as μ1 defined in Eq. 2
Ja of
M4 (ks, m) - E,, CSF()))
where 'k' is the index for the convolution layer, 'm' is the feature map index in each convolution layer, "CSF" is the contrast sensitivity function (CSF), "a" is the mean activation of the feature map and 'f" is the spatial frequency in cycles per degree. Y4 quantifies the average sensitivity of a CNN channel weighted by the CSF over different spatial frequencies. The channels having higher μl values should deliver better perceptual features according our hypothesis, because they can be more sensitive to visually perceivable distortions in input images.
4.3 ORIENTATION SELECTIVITY
In addition to the underlying spatial frequency, orientation also plays an important part in human per- ception of visual stimulus. Neuro-scient research indicates that the HVS (Human Visual System) highly adapted to extract repeated patterns for visual content representation (Wu et al. (2017). The complexity of visual pattern has an effect in its perception. If pattern is regular, the visual
4

---

Under review as a conference paper at ICLR 2020
Visual Frequency Sensitivity
150
A 2 a [More Sensitive)
Mean Activation
100
Feature Map Fasture Map ?
Human CSF
50
10
15
20
Spátial Frequency (cycldeg)
Figure 3: Two different feature maps may have different sensitivities to important visual frequencies.
masking for such a pattern is weak, and distortions are easily perceivable. For complex and irregular image patches, the visual system presents stronger masking effect.
We have quantified orientation selectivity of different channels a pre-trained image classification CNN (VGG-16) in Fig. 2.(b). We observe that significant proportion of channels show definite orientation selective tuning, such as the ones represented in Fig. 2(b) Fig. 2(b)(b), Fig. 2(b)() etc. There channels should in theory be more sensitive responding to simple patterns. However, quite a few channels show weaker orientation sensitivity such as the ones represented in Fig. 2(b)(c), Fig. kkb) Fig. ((b)(n) and Fig. (0b)(o) etc. We hypothesize the channels that show strong and definite orientation selective tuning, respond better to regular image patterns, which have lower masking thresholds, making these channels deliver better perceptual quality features.
Suppose that within some layer 'K, ag be the mean activation of feature map corresponding to channel 'm' to the input grating of orientation '0', Let the maximum mean activation be am maxg ag We model our orientation selectivity attribute for a channel as μ22 in Eq. (3).
(2 m.)
- (a. -²m)²
(3)
Considering our hypothesis, channels with higher #2 should deliver relatively better features of perceptual quality.
4.4 PERCEPTUAL EFFICACY SCORE (PE)
Based on our defined attributes, we devise a quantification for the efficacy of channels in pre-trained deep CNNs deliver good features for perceptual quality, called the Perceptual Efficacy (PE). The perceptual efficacy of a channel with index 'm' in layer 'k" is defined as the product of normalized #2 and P2.
141 (ki m) 122 m)
PF(k, Em (k, Em ())
EXPERIMENTAL SETUP
We devise experimental methodology verify hypotheses that deep CNN representations that have higher PE are better perceptual quality features. Let F3 be the set of all channels within a layer's ' of pre-trained CNN (e.g VGG-16).
Fr = ( 4
D
(5)
We constitute subsets of channels from Fs based on the quantification of our proposed attributes (PE). For example, if there are 128 channels in the relu2 22 layer of the VGG-16, we can group the top 15% (19 channels) of the total 128 with the highest PE as
1- { / 1 T

---

Under review as a conference paper at ICLR 2020
Similarly, the bottom 15% channels with the lowest PE can be represented
L-15 = { D T
where H-r.L--x E& and e (0, 100). For x 100, the subsets become the complete set of channels Fg.
To validate our hypotheses, is necessary to demonstrate that subsets containing higher PE val- ued channels deliver better perceptual quality features compared to subsets with lower PE valued channels.
5.1 OBJECTIVE QUALITY ASSESSMENT (OQA) TESTS
OQA tests correlate the performance of any quality metric, with human subjective assessment and perception of quality (Sheikh et al. (200)). Human assessment of perceptual image quality is quan- tified using the Differential Mean Opinion Score (DMOS) over images with varying levels of distor- tion. DMOS is a quantitative representation of how human observers are able to perceive perceptual differences between natural and distorted images. Metrics that have higher correlation with DMOS scores after regression, measured using statistical indicators such the RMSE (Root Mean Square Error), LCC (Linear Correlation Coefficient) and the SROCC (Spearman Rank Order Correlation Coefficient), are regarded as better quality metrics (Sheikh et al. (2006) The higher the correlation metries (LCC and SROCC), the better a metric correlates with human ability to perceive perceptual differences.
In our problem setting, we will use Eq. with the different subsets of channels, as defined in Section 5. We demonstrate that for with Eq. 1, within different CNN layers, channels having higher PE, give much better correlation with DMOS compared to channels with lower PE. Essentially, demonstrate that CNN channels with our pre-deseribed attributes indeed better perceptual quallity features.
We images and DMOS scores from both the LIVE image quality dataset (Sheikh al. (2006) and multiple distortion dataset (Jayaraman et al.] (2012) which collectively include images with Gaussian Blur, JPEG compression, JPEG2000, White Noise as well as images which have been corrupted with multiple types of distortions (such as white noise, Gaussian blur and camera noise) simultaneously.
We will repeat our experiment accross multiple layers of several pre-trained image classification CNNs such AlexNet, ShuffleNet, SqueezeNet and VGG-16.
Datapoints
Datapoints
Fitted Curve
Fitted Curve
80
80
DMOS
60
DMOS
60
40
40
20
20
8
2
4
Metric Score
Metric Score
(a) H-10
(b) L-10
Figure 4: Correlation of metric scores in Eq. with human subjective DMOS for the 'fire2 ReLU exp2x2' layer of the 'quuezzNer'. can be seen that the metric in Eq. with the channel subset H-10 has a much better correlation with DMOS, compared to Eq. 1 with the L-10 subset of channels.

---

Under review as a conference paper at ICLR 2020
Table I: Objective Quality Assessment Test. The correlation of metric scores delivered by Eq. (for different feature subsets) with human subjective assessment of perceptual quality, quantified by DMOS.
Feature Set
Network
Layer
RMSE
LCC
SROCC
9.8366
0.8146
0.8028
H-10
8.8286
0.8538
0.8486
ReLU2 2
L-10
12.3114
0.6878
0.6806
L-990
10.5863
0.7813
0.7739
VGG-16
9.8149
0.8155
0.8076
H-2
8.8183
0.8542
0.8476
ReLU4_1
L-2
10.2338
0.7874
0.7863
L-80
9.8485
0.8141
0,8070
F
9.7580
0.8179
0.8155
H-10
9.1514
0.8419
0.8368
ReLU1
L-10
12,8110
0.6553
0.6562
L-70
10.3186
0.7936
0.7931
AlexNet
8.8015
0.8548
0.8605
H-5
8.5467
0.8637
0.8651
ReLU4
L-5
9.8927
0.8122
0.8197
L-50
9.0697
0,8450
0.8507
11.2791
0.7468
0.7397
fire2_
H-10
10.8632
0.7679
0.7625
ReLU-
L-10
12.6927
0.6632
0.6614
exp_3x3
L-50
IT.6555
0.7264
0.7199
SqueezeNet
11.4191
0.7394
0.7314
fire6.
H-5
11,8710
0.7142
0.7017
ReLU
L-5
12.6857
0.6637
0.6540
exp_3x3
L-50
12.0600
0.7063
0.6988
F
11.0810
0.7570
0.7519
H-10
9.9055
0.8117
0.8002
node7
L-10
14.2481
0.5424
0.5583
L-70
11.6409
0.7272
0.7232
ShuffleNet
9.1354
0.8425
0.8421
H-10
8.8577
0.8528
0.8477
node17
L-10
11.5070
0.7346
0.7407
L-70
9.2306
0.8389
0.8414
Table 2: 2AFC Similarity Test. image triplets.
How well metric decisions conform with human assessment of
SqueezeNet
ResNet18
(fire2
ReLU
3x333333
(Res4a ReLU)
F
H-10
L-10
L-75
H-2
L-2
L-80
60.23
62.85
56.08
59.83
60.69
62.53
60.10
60.21
VGG-16
AlexNet
(ReLU3
2)
(ReLU4)
F
H-5
L-5
L-50
F
H-2
L-2
L-75
59.97
60.86
58.28
59.41
64.62
63.38
61.30
62.35

---

Under review as a conference paper at ICLR 2020
5.2 2AFC SIMILARITY TESTS
In the 2AFC test, two distorted images are shown to an observer and he/she is asked to rate which one is closer the ground truth, in perceptual appearance. This process is repeated for multiple image triplets and observers per-triplet to construct a data-set called the Berkley-Adobe Perceptual Patch Similarity Data-set (BAPPS) Zhang et al. (2018)) ). Objective metries such as the one in Eq. I are thereafter evaluated to see how well they conform to the pair-wise human judgment. For example, in an image triplet, let Xo and X1 be two distorted versions of the ground truth image Xg that are shown to 5 human observers, 4 of which judge to be closer to Xg. as opposed to being
closer to Xg. If an objective metric evaluates Xo to closer to the ground truth, it will get an 80% credit which in the opposite case would be 20%.
The BAPPS data-set contains images with distortions such as super-resolution, frame-interpolation and deblurring, which do not have subjective DMOS data-sets available online. Therefore, secondary experiment, we perform a 2AFC test on super-resolution, came-interpolation and video- deblurred frame images in the BAPPS data-set with Eq. I for different channel subsets defined in Section. 5. In order to verify our hypothesis, we will show that subsets that contain channels with higher PE, deliver better perceptual quality features.
RESULTS AND DISCUSSIONS
Table I quantifies the correlation of Eq. I with DMOS for different subsets of channels, constructed on the basis of our described attributes, as explained Section 5. Table validates our hypothesis that within CNN layer, channels which have higher PE (Eq. 4) deliver better perceptual quality features. must be reminded that higher LCC and SROCC indicate better correlation of Eq. with human ability discern perceptual differences (DMOS). can be observed that very small proportions (2%- 10% of total) of channels with the highest PE, deliver better perceptual quality features compared
much higher proportion (50%-90% of total) of channels having lower PE. Furthermore, in a majority of cases, small proportion of channels that have our described attributes (higher PE), perform even better than the complete set of channels in the layer. This implies that our proposed attributes are indeed important characteristics that make learned deep CNN representations good perceptual quality features.
Table. 2 shows the results of our secondary 2AFC similarity test on the super-resolution, frame- interpolation and video-deblurring distorted images in the BAPPS data-set. It can been seen that yet again, similar to the conclusion in the primary QQA experiment, the subsets with channels having higher PE are better perceptual quality features compared to even much larger subsets having channels with lower PE.
7 FUTURE WORK
We have proposed a model to explain and interpret which channels in pre-trained image classification CNNs deliver better perceptual quality features. The model may be used improve the use of deep representations as perceptual quality features by helping in feature selection for IQA methods such as (Bosse al. (2017) and maybe designing channel attentive mechanism to improve the perceptual loss (Johnson et al. (2016)). The model may also be reference for learning better perceptual quality feature representations which may benefit a wide variety of applications. Furthermore, the model may be enhanced to include more psychophysical factors such as eccentricity etc. Another possible application may be CNN-based image compression where prior knowledge of the potential efficacy of different channels may help efficient perceptual compression of redundant image data.
CONCLUSIONS
Deep CNN representations of pre-trained image classifications CNNs have been popularly used as perceptual quality features for perception orientated applications such as CNN based quality assess- ment, image/video super-resolution and many image-to-image translation problems. In this paper, as a novel and first effort, we have linked basic human visual perception models to pre-trained deep CNN representations in order explain and interpret them perceptual quality features. Based on
8

---

Under review as a conference paper at ICLR 2020
masking characteristics in human visual perception, formulate attributes of channels in different layers of pre-trained networks, and experimentally demonstrate that the attributes are important char- acteristics that make some deep CNN representations better perceptual quality features compared to others.
REFERENCES
Recommendation itu-r bt.500-11 methodology for the subjective assessment of the quality of tele- vision pictures. 2002.
Yochai Blau and Tomer Michaeli. The perception-distortion tradeoff. IEEE CVPR, 2018
Yochai Blau, Roey Mechrez, Radu Timofte, Tomer Michaeli, and Lihi Zelnik-Manor. 2018 pirm challenge perceptual image super-resolution. 2018.
Sebastian Bosse, Dominique Maniry, Klaus-Robees Rohert Muller, Thomas Wiegand, and Wojciech Samek. Deep neural networks for no-reference and full-reference image quality assessment. IEEE Trans- actions on Image Processing, 27:206-219, 2017.
Jacqueline Mendonça Lopes de Faria, Osamu Katsumi, Masashi Arai, and Tekehiko Hirose. Objec- tive measurement of contrast sensitivity function using contrast sweep visual evoked responses. The British journal of ophthalmology, 82 2:168-73, 1998.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2413,2423, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770- 778, 2015.
Forrest N. landola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ilmb model size. ArXiv, abs/1602.07360, 2017.
Dinesh Jayaraman, Anish Mittal, Anush K. Moorthy, and Alan C. Bovik. Objective quality sessment of multiply distorted images. 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR), pp. 1693-1697, 2012.
Justin Johnson, Alexandre Alahi, and Fei-Fei. Perceptual losses for real-time style transfer and super-resolution, 2016.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646-1654, 2016.
Sutskever I. Krizhevsky, A. and E Hinton. Imagenet classication with deep convolutional neural networks. NIPS, 2012.
Janus J. Kulikowski, S. Marveelja, and P. O. Bishop. Theory of spatial position and spatial frequency relations in the receptive fields of simple cells in the visual cortex. Biological Cybernetics, 43: 187-198, 1982.
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution us- ing generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105-114, 2017.
Chao Ma, Chih- Yuan Yang, Xiaokang Yang, and Ming-Hsuan Yang. Learning no-reference quality metric for single-image super-resolution. CoRR, Abs/1612.05880. 2016.
Lamberto Maffei and Adriana Fiorentini. The visual cortex as a spatial frequency analyser. Vision research, 13 7:1255667, 1973.

---

Under review as a conference paper at ICLR 2020
Roey Mechrez, Itamar Talmi, Firas Shama, and Lihi Zelnik-Manor Maintaining natural image statistics with the contextual loss. In ACCV, 2018.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Processing Letters, 20:209-212, 2013.
Mareus J. Nadenau, Stephan M. Winkler, David Alleysson, and Murat Kunt. Human vision models for perceptually optimized image processing review. 2000.
Mark Sandler, Andrew F Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. 2018 IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4510 2018.
Hamid R. Sheikh, Muhammad F. Sabir, and Alan C. Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing, 15: 3440-3451, 2006.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, (bs//140.11. 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2014.
Xintao Wang, Ke Yu, S. Wu, Jinjin Gu. Yihao Liu, Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang. Esrgan: Enhanced super-resolution generative adversarial networks. 2018.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Ecro P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions Image Processing, 13(4): 600-612, 2004.
Jinjian Wu, Leida Li, Weisheng Dong, Guangming Shi, Weisi Lin, and C.-C. Jay Kuo. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing, 26:2682-2693, 2017.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. IEEE CVPR, 2018.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufftenet: An extremely efficient con- volutional neural network for mobile devices. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6848 0488-6856.228 2017.
A GRATING GENERATION
In this section, present details behind the generation of sinusoidal gratings of different spatial frequencies.
The contrast sensitivity function is expressed on the domain of spatial frequency in cycles per degree (cyc/deg). The cycles per degree express the number of sine cycles captured by the observer per unit degree of observation. Obviously, the distance of viewing and dimensions of the screen play important part in this measurement.
We essentially generate gratings in the computer simulation in cycles per pixel. Let the display screen being used the experiment have a height '/' inches and resolution 'r' pixels per inch. The optimal viewing distance in psychovisual experiments should satisfy function called the PVD (42) )2002). The PVD is function that expresses the optimal ratio of viewing distance to the height of the display screen. The optimal viewing distance 'd" for the screen with height "/² can be calculated using the PVD.
The transformation between cycles/degree and cycles/pixel is
cycles cycles degrees pizel degree pixel
(8)
10

---

Under review as a conference paper at ICLR 2020
Where
pizels
180
(9)
degree
TT x xx
Therefore,
cycles
cycles
d d
(10)
pizel
degree
180
We have tested with number of different display systems of SD, 2K and 4K resolutions. Consider- ing that the PVD takes the viewing angle into account, the changes in the resultant spatial frequencies of the gratings are small and insignificant. Therefore, it can be concluded that the choice of display system has a negligible effect on the experiment.
For the generation of grating with fixed spatial frequency and varying orientation, the experimental setup is the same.
B ADDITIONAL NETWORKS
We demonstrate the validity of our hypothesis for other pre-trained image classification CNNs as well. These networks include:
GoogleNet (Szegedy et al. 2014))
MobileNet-v2 (Sandler et al. (2018)
R8 (He et al. (2015))
can be seen in Table 3 that our hypothesis regarding important attributes is valid for these ad- ditional CNNs well. Small proportions of channels (H-(5-10)) with higher PE (Eq. 4) deliver much better perceptual quality features compared to a much higher proportion of channels with lower PE (H-(45-80) and the complete of channels in the layer (F), A scatter plot of some correlations, shown side by side is depicted in Fig. 5.
C ADDITIONAL EXPERIMENT
We have included an additional experiment with the aim to demonstrate that our work and experi- mental analysis techniques (Objective Quality Experiments) generalizable practical imaging problems. The experiment will be an x4 CNN based super-resolution task. However, a little back- ground on the perception-distortio trade-off is necessary before proceeding.
C. PERCEPTION DERCEPTIONDDISTORTION TRADE-OFF
It is a known fact that traditional distortion metrics such as PSNR and SSIM are not well correlated with perceptual quality and the distortion measured by PSNR and SSIM for degraded image dicates the net deviation from its reference image. Many different images of different perceptual qualities may have a same distortion with respect given reference. Perceptual quality, on the other hand, is no-reference quantification of how natural an image appears (conformity with natural im- age statistics). This is why objective distortion measures such as PSNR and SSIM do not necessarily account for perceptual quality. Keeping view the recent interest in perception-oriented imaging applications, to quantify no-reference perceptual quality, a combination of the metrics such as NIQE (Mittal al. (2013)) and NRQM (Ma al. (2016)) has been used perceptual indicator (PI) in recent works (Blau et al. (2018)). The lower the Perceptual Index (Eq. 1), the better the perceptual quality of an image.
- = (10- +
(II)
The work of Blau & Michaeli (018)) demonstrates that distortion and perceptual quality are in a trade-off relation and this trade-off is the correct measure for quantifying the perceptual efficacy of image restoration algorithms explained in (Blau al. (2018)). Consider (12)) standard

---

Under review as a conference paper at ICLR 2020
Table 3: Objective Quality Assessment Test. The correlation of metric scores delivered by Eq. (for different feature subsets) with human subjective assessment of perceptual quality, quantified by DMOS.
Network
Layer
Feature Set
RMSE
LCC
SROCC
F
9.2730
0.8370
0.8351
conv2_
H-5
9.1360
0.8425
0.8364
ReLU_
L-5
12.6595
0.6654
0.6674
3x3
L-80
9.6636
0.8218
0.8203
GoogleNet
F
10.2264
0.7977
0.8061
inception Re-ReLU
H-5
9.8592
0.8137
0.8201
L-5
10.8882
0.7667
0.7750
3x3
L-45
10.0326
0.8063
0.8163
F
11.9441
0.7099
0.7017
block1
H-10
11.6059
0.7292
0.7256
expand_
I-10
13.7130
0.5884
0.5825
ReLU
L-70
12.7912
0.6566
0.6505
MobileNet-v2
F
10.1957
0.7991
0.8063
block3
H-10
9.2423
0.8385
0.8459
expand_
L-10
13.2810
0.6219
0.6223
ReLU
L-70
10.7877
0.7716
0. 0.7804
F
10.8622
0.7680
0.7702
Res2a
H-10
10.0841
0.8040
0.7898
ReLU
L-10
11.6195
0.7284
0. 0.7339
I,75
11.2807
0.7467
0.7549
ResNet-18
F
9.1073
0.8436
0.8611
Res4a.
H-5
9,2559
0.8379
0.8509
ReLU
L-5
10.1132
0.8028
0.8072
L-75
9.3484
0.8344
0.8518
loss function for CNN based SR, which is defined as a combination a pixel-wise loss (41) and the perceptual loss (lp).
Lp = a 4 + a)) a). 1p
(12)
Increasing a in (12( will lead to a decrease in distortion (increasing SSIM), which according to the perception-distortion trade-off, should result in images with lower perceptual quality (higher Pl). Similarly, decreasing a will give more weight to the perceptual loss, which would result in images with better perceptual quality (lower PI), but lower SSIM.
C.2 X4 RESOLLUION EXPERIMENT SETUP
In this experiment, we have used the well known VDSR (Kim et al. (2016) network trained on images from the DIV2K data-set. The network was separately trained on two loss functions. The first being the standard perceptual loss in (12) and the second slight modification 13) based on our analysis.
Tipu a 1 (1 a))
(13)
where
lpu
wht |b (Ious) t (Tar)
(14)
M
W H.
m=1
and
(1.k))
(15)
Σn Pi (k, m)
(q.(13) presents version of perceptual loss in which the channels are weighted (Eq.14) 14) per defined visual frequency sensitivity (theoretical importance If our hypothesis is correct, the loss function in Eq.(13 should be able to perform better com- pared to (1)) in terms of providing a better cneccption-distortin trade-off for super-resolution. The results should consequently serve an additional experiment to reinforce our hypothesis and
channel based a defined attribute).
12

---

Under review as a conference paper at ICLR 2020
100
100
Datapoints
Datapoints
Fitted Curve
Fitted Curve
80
80
DMOS
60
DMOS
60
40
40
20
20
-3
(Scor))
-6
-5
log(SCore)
3
-2
-1
(a) H-10
L-10
100
100
Datapoints
Datapoints
Fitted Curve
Fitted Curve
80
80
DMOS
60
DMOS
60
40
40
20
20
2
8
10
log(Score)
(og(rc
(c) H-5
(d)
100
Datapoints
Datapoints
Filted Curve
Fitted Curve
80
DMOS
60
DMOS
60
40
40
20
20
-6
-5
(Scor))
3
-1
0
-10
-5
log(Score)
0
(c) H-10
(f) L-10
Figure 5: Correlation of metric scores in Eq. with human subjective DMOS shows that the metric Eq. with the channel subset has much better correlation with DMOS compared to Eq. with the L subset of channels. Each pair shown side by side is for different network.
demonstrate the transfer-ability of objective quality assessment experiment conclusions to practical applications. The results should also demonstrate the ability of an independent attribute indi- cator of the perceptual abilities pre- trained channels. The results are not intended to deliver state of the art results, just verify the generalization of our analysis practical problems.
13

---

Under review as a conference paper at ICLR 2020
Perception-Distoriin
3.8
Better Perceptual Quality
4
4.2
4.4
P
4.6
4.8
5.2
0.74
0.75
0.76
0.77
0.78
0.79
0.8
SSIM
Less Distortion
Figure 6: (Eq. 13) improves the perception -istootionn trade-off compared to the perceptual loss (Eq. 12) when used for training with different a settings for an x4 SR with the VDSR on the DIV2K data-set.
C.3 RESULTS
The ReLU4 layer of a pre-trained VGG-16 was used for perceptual loss terms in Eq.(12) and (13() The experiment was repeated for different values to analyze the perception-distorte trade-off delivered by both loss functions. The SSIM and PI reported in Fig.6 are averaged over all the 100 images in the test-set.
can clearly seen Fig.6 that Lyv in Eq.(13) improves the perception distortion trade-off com- pared to Lp in (12)) The improvement is evident from the fact that the perception-distortie curve delivered by Lpo is higher compared to the one delivered by Lp for various (different points on the curve correspond to different in (q.) and (13)) Therefore, at the same level of distortion (SSIM), Lepo is able to generate images with better perceptual quality compared to Lpp
The result reinforces the validity of the experimental results presented in the main paper and demon- strate that they have consequence for practical applications such as super-resolution. The results of this experiment also iterate the fact that conclusions derived from Objective Quality Assess- ment(OQA) experiments have a consequence for practical image generation problems.
14


Reviews:
[{"review_id": "rkgFHuWGtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the article the authors propose to measure quality of CNN-features by quantifying the orientation tuning and spatial frequency sensitivity of the features. The underlying hypothesis is that properties of features in the human visual cortex are also indicators for quality in CNNs. The authors devise an experiment similar to experiments performed on mammals to check which features are active under which types of basic patterns. Afterwards, a loss-function is devised that uses proportions of the best or worst features according to the metrics and it is shown that features that have high values on the metrics also lead to good performance.\n-----------------------------------------------------------------------------------------\nI have a problem understanding some of the metrics used. In the introduction, the following claim is made:\n\"The first attribute is sensitivity to spatial frequencies at which there is minimal contrast masking in human visual perception\". To my understanding, the metric to measure this is (2) in 4.2. a_m^k is to my understanding the average response of the feature when given an image with orientation-frequency f. therefore, the derivative should be \"the change of activation under change of frequency\". I feel unable to connect this with the initial hypothesis, as it does not mention change of frequencies. i would have expected the correlation between CSF(f) and a^k_m(f), \ndid you have a reason why you did not chose correlation?\n\nSimilarly, for mu_2 in (3) you are using the maximum value. Is there a reason not to use the mean? in that case mu_2 would be the variance, which would be a natural measure for orientation selectivity.\n\n------------------\nIs there a way to make Table 1 more pleasing for the human eye wrt the discussion of the results?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "rJltQvNAFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I thank the authors for their detailed response. Some of my questions have been addressed in the rebuttal, but as far as I can tell very few modifications have been made to the paper: mainly an additional super-resolution experiment in the appendix, which goes in a good direction, but currently comes across as quite preliminary. So I think the paper still has most of the weaknesses I mentioned and thus it is not quite fit for publication. But I do encourage the authors to strengthen the experiments (for instance by addressing the points I raised, or by some other means) and resubmit elsewhere.\n\n---\n\nThe paper proposes an approach to analyzing the properties of \"perceptual metrics\" used in deep learning image generation methods. \"Perceptual metrics\" are computed by measuring distances between images not in the pixel space, but i na feature space of a pre-trained cNN. The proposed analysis method, inspired by studies of human perception, is based on measuring the response of these features to sinusoidal gratings of varying frequency or orientation. Based on these responses, the paper proposes a \"Perceptual Efficacy Score\" that should measure the importance of certain feature in the feature maps for the performance of a perceptual metric. Experiments show that indeed distances measured between features with high score better correlate with human judgement of image similarity than distances between features with a lower score.\n\nI find the paper quite interesting, but lean towards rejection at this point. This i smainly because the experiments seem somewhat anecdotal and incomplete, see below for further details.\n\nPros:\n1) Application of methods from psychology/neuroscience to artificial neural networks is an interesting avenue of work. Moreover, better unsderstanding of \"perceptual metrics\" is of wide interest for various image processing applications.\n2) The proposed score seems to indeed correlate quite well with the importance of features for human judgement of image similarity.\n3) Presentation is mainly clear.\n\nCons:\n1) Experiments are not very exhaustive and at times a bit confusing. For instance:\n1a) Results are sometimes presented in a confusing way. In Figure 4 first of all it is not quite clear what points correspond to I guess each point is an image) and, second, it is not very obvious that the correlation is higher i none of the plots. In tables 1 and 2 it is confusing that different percentiles for H and L are used for different networks/layers. Is this based on some tuning? Then the tuning process should be clearly explained. Moreover, it might be useful to report the full curves of performance as a function of the percentage of features used.\n1b) There are no baselines and there is not much justification of computing the \"Perceptual Efficacy\" score the way it is computed. What if one uses only the orientation-based score? Or only the frequency-based? What if one selects the most relevant features in a data-driven way (based on correlation on a training set)? What if one selects subsets of features randomly? \n1c) While the method is inspired by methods used for studying natural vision systems, there is no connection to human experiments. It would be interesting to see a comparison of frequency and orientation tuning of features in a CNN to human cells (as I understand, the latter should be available in prior works?).\n1d) It would be great to see the selected features be used not only for offline image similarity assessment, but also for training image processing models - in the end, this has been the main use of \"perceptual metrics\". Do they lead to improved results?\n1e) Since the paper is about (subjective) image quality, it might be useful to show some qualitative results, potentially in the appendix if space is an issue.\n\n2) There are some issues with the presentation:\n2a) I had a hard time understanding what exactly \"Contrast Sensitivity Function\" and \"contrast masking\" are.\n2b) Minor issues:\n- In the abstract: \"trained object detection deep CNNs\" - I guess image classification is meant\n- Beginning of Section 2: \"Section. 2\", \"convolution layer as collection channels\"\n- Section 4: \"corresponds the the peak:\n- Section 5.2 \"Berkeley-Adobpe\"", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylO0Xs75H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an analysis of convolutional neural networks (CNNs) features the basis for making perceptual quality comparisons. The analysis is based on the proposed Perceptual Efficacy (PE) Score that measures spatial frequency and orientation selectivity of CNN features. The hypothesis put forward by the authors is that a CNN features with high PE score can be used to formulate a perceptual loss (Eq. 1) that correlates well with human image quality judgement. The authors use a dataset of human image quality judgements to assess their hypothesis. \n\nOne issue I see with the hypothesis as stated is that in the definition of frequency selective features, the authors make use of\nthe Contrast Sensitivity Function (CSF) which quantifies the dependency of human perceptual characteristics on frequency. So in the definition of the PE score we have embedded knowledge of human perceptual sensitivity. Is it therefore surprising  that we see correlation between the high PE features and human judgements of quality? \n\nExperimental results: The scatterplot presented in Figure 4 does not say to me what the authors claim it should. I do not see a significant difference between the low-PE features and the high-PE features in terms of their correlation with human image quality judgement (as measures in this case by the DMOS). I also find the large table of number in Table 1 to be rather\nimpenetrable. I would recommend an alternative method of presentation to make the desired point.  \n\nClarity: There are many undefined terms and acronyms (eg. SISR, HVS are not defined, while DMOS and SROCC are not described). Also, the description of visual masking in Sec. 4.3 was confusing and difficult to follow. Otherwise the writing was reasonably clear.\n\nImpact and significance: Overall, the findings of the paper are not terribly surprising and as discuss above, given the use of the CSF (quantifying human perceptual characteristics) in the definition of the CNN Perceptual Efficacy (PE) score, it would seem rather surprising that a correlations would not be found. As a result of this as well as the rather narrow nature of the study involved, I am inclined to think that the impact potential of this paper would be rather low. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxhCSGC9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission aims to analyze deep neural network (DNN) features in terms of how well they measure the perceptual severity of image distortions. It proposes to characterize each DNN feature in terms of two well known properties of the human visual system: a) sensitivity to changes in visual frequency and b) orientation selectivity. Both properties are evaluated with respect to the known human Contrast Sensitivity Function (CSF) and measured empirically from the feature’s response to (oriented) sinusoidal gratings. The results are quantified by a composite score termed Perceptual Efficacy (PE).\nIn a set of comprehensive experiments (several pre-trained DNNs, several layers per DNN and two different datasets of distorted images with human perceptual quality annotations) it is demonstrated that feature representation consisting of a layer’s features with high PE better agree with human perceptual quality judgments than low PE feature representations from the same layer.\n\nI believe the submission convincingly demonstrates a statistical association between the proposed PE score of a DNN feature and human perceptual quality assessments.\nThough, it remains unclear whether the characteristics captured by the PE score are necessary or sufficient to explain the success of DNNs to guide image generation tasks by providing a perceptual loss function.\nFor that I believe it is necessary to demonstrate that the present empirical results can be used to improve results of an image generation task, e.g. super-resolution. \nFurthermore the limits of the PE score could be explored by hand-crafting image representations with maximal PE score and comparing their usefulness in guiding e.g. a super-resolution task compared to a pre-trained DNN.\n\nThus, overall I believe the submission reports interesting initial results but falls short of showing that they capture general properties that can be transferred to improving perceptual loss functions. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to analyze CNN representations in terms of how well they measure the perceptual severity of image distortions.  In particularly, (a) sensitivity to changes in visual frequency and (b) orientation selectivity was used. Although the reviewers agree that this paper presents some interesting initial findings with a promising direction, the majority of the reviewers (three out of four) find that the paper is incomplete, raising concerns in terms of experimental settings and results. Multiple reviewers explicitly asked for additional experiments to confirm whether the presented empirical results can be used to improve results of an image generation. Responding to the reviews, the authors added a super-resolution experiment in the appendix, which the reviewers believe is the right direction but is still preliminary.

Overall, we believe the paper reports interesting findings but it will require a series of additional work to make it ready for the publication.
Decision: Reject