Analysis and Interpretation of Deep CNN Representations as Perceptual Quality Features

1 INTRODUCTION . Quantifying human perception of image quality has been a subject of significant research for quite some time . Full-reference objective metrics such as the PSNR ( Peak Signal to Noise Ratio ) and SSIM ( Structural Similarity Index ) ( Wang et al . ( 2004 ) ) , being fair metrics of distortion between two images , are not a satisfactory metrics to measure differences in perceptual quality . Considering the recent interest in the applications of deep CNNs in perception-oriented problems such as superresolution , image-restoration , frame-interpolation and style-transfer etc , research into effective loss metrics that quantify perceptual quality and help train CNNs in delivering better perceptual quality has become paramount . The perceptual loss proposed by Johnson et al . ( 2016 ) was one of the first to demonstrate how effective the feature representations of pre-trained image classification CNNs could be as features of full-reference perceptual quality , especially when incorporated into loss functions for image restoration . The perceptual loss is now popularly adopted in many image restoration problems such as super-resolution , style transfer , denoising etc . ( Ledig et al . ( 2017 ) , Wang et al . ( 2018 ) , Gatys et al . ( 2016 ) ) . Zhang et al . ( 2018 ) and Blau & Michaeli ( 2018 ) further demonstrate how effective deep CNN representations can be as features of perceptual quality , but without any analysis into their characteristics . More recently , Mechrez et al . ( 2018 ) proposed a variation of the perceptual loss called the contextual loss , which still employs deep CNN features as perceptual quality features but uses an approximation of the KL-divergence to quantify distance . The contextual loss has been demonstrated to be quite effective in maintaining natural image statistics during SISR ( Single-Image Super-Resolution ) . The recent PIRM Super-Resolution Challenge Report ( Blau et al . ( 2018 ) ) clearly iterates that the perceptual loss and the contextual loss are the most widely used loss functions for CNN based perceptual image Super-Resolution . Nevertheless , like most applications of deep learning , there has been little or no effort to understand and interpret the role of deep CNN representations as effective perceptual quality features . This is quite understandable , as it is difficult to find a direction to approach this problem from . Neural networks are non-linear , which makes a tractable analysis tricky . Furthermore , human perception of quality is also something that is still not understood completely . Most of our basic understanding of human visual perception of quality is in the frequency domain , with models such as the Contrast Sensitivity Function ( CSF ) ( de Faria et al . ( 1998 ) ) . To make a connection between deep CNN features and human perception , it is important to realize that deep CNN channels are essentially complex spatial frequency and orientation selective filters . We stimulate pre-trained image classification CNNs with sinusoidal grating stimuli , record the response in the form of mean activation of each channel as function of spatial frequency/orientation of input grating , thus quantifying the frequency and orientation selectivity of different channels . This approach makes it significantly easier to establish a connection between perception models such as the CSF with learned deep feature representations . We hypothesize that two attributes are important for deep CNN channels that are good perceptual quality features . The attributes are based on visual masking in human visual perception , which refers to human ability to perceive distortions and contrast in visual stimulus . The first attribute is sensitivity to spatial frequencies at which there is minimal contrast masking in human visual perception ( Nadenau et al . ( 2000 ) ) , making the CNN channel sensitive to highly perceivable distortions . The second attribute being a definite and strong orientation selectivity , which helps the channel respond better to image regions with less pattern complexity , where there is less masking for distortions from a perceptual standpoint ( Wu et al . ( 2017 ) ) . We verify our hypothesis by designing an Objective Quality Assessment ( OQA ) experiment ( Sheikh et al . ( 2006 ) ) . OQA experiments correlate the performance of any quality metric with human perception of quality , which is an accepted and standard experimental technique . We group the set of channels in different CNN layers into subsets on the basis of our hypothesis and demonstrate that the group which has channels with our described attributes , delivers a much better as a set of perceptual quality features . We repeat our experiment across multiple layers of many pre-trained image classification networks such as the VGG-16 ( Simonyan & Zisserman ( 2014 ) ) , AlexNet ( Krizhevsky & Hinton ( 2012 ) ) , ShuffleNet ( Zhang et al . ( 2017 ) ) and SqueezeNet ( Iandola et al . ( 2017 ) ) . 2 DEEP CNN REPRESENTATIONS AS PERCEPTUAL QUALITY FEATURES . The main motivation behind using pre-trained image classification deep CNN representations as perceptual quality features is that instead of a distance measure between two images being a good FR metric , computing distance after non-linear transformation of images into a high dimensional manifold , might result in a better perceptual quality measure . The high dimensional manifold in this case is the manifold of pre-trained CNN features . The general form for the perceptual loss ( Johnson et al . ( 2016 ) ) is given by Eq . ( 1 ) lp = 1 M ·W ·H M∑ m=1 ‖Φkm ( Iout ) − Φkm ( IGT ) ‖22 ( 1 ) Where ’ Φkm ’ is the feature map corresponding to the ’ m th ’ channel in the ’ kth ’ layer which as ’ M ’ number of total channels with feature map dimensions ’ H·W ’ . As mentioned before , applying pretrained deep CNN representations as perceptual quality features has proven to be quite effective in FR-IQA methods ( Bosse et al . ( 2017 ) ) , image restoration ( Wang et al . ( 2018 ) ) and style transfer ( Gatys et al . ( 2016 ) ) problems , as iterated by Blau et al . ( 2018 ) . However , little else is known of the ability and characteristics of deep CNN representations as perceptual quality features . In this work , using basic human perception models , we aim to get more insight into the role of pre-trained deep representations as perceptual quality features . 3 PROBLEM FORMULATION . Section . 2 iterates the motivation and wide spread use of pre-trained deep CNN representations as features of full-reference perceptual quality . However , there has been no effort to explain and interpret the role of deep representations as perceptual features . We consider a CNN convolution layer as collection of channels which deliver perceptual quality features . For example , the relu3 2 layer of the VGG-16 has 256 channels . Are all of the channels equally effective in delivering good perceptual quality features ? Are some channels better than others and if so , what attributes make them better ? The problem in question is important in explaining the role of deep CNN representations as perceptual quality features , but it is somewhat difficult to approach because of the ’ black-box ’ nature of neural networks . In section . 4.1 , we will introduce a methodology to quantify the spatial frequency and orientation tuning of channels in pre-trained CNNs . Using this formulation , we will interpret and explain deep CNN features as perceptual quality features by making use of basic human visual perception models , which rely on spatial frequency and orientation characteristics of input stimuli . In essence , the formulation in Section . 4.1 will act as a bridge to link attributes of deep representations and basic visual perception . 4 A PSYCHOVISUAL APPROACH . 4.1 FREQUENCY/ORIENTATION TUNING QUANTIFICATION . Our experimental method is inspired by the grating stimulus experiments used by neuro-scientists to study human visual perception characteristics ( Kulikowski et al . ( 1982 ) ) . We aim to quantify both the spatial frequency and orientation tuning of different channels in the pre-trained CNN . To quantify the spatial frequency tuning , we generate concentric sinusoidal gratings of a fixed contrast and varying spatial frequencies ( cycles per degree ) , use them to stimulate pre-trained image classification CNNs and record the responses of the feature maps in the form of mean activation versus spatial frequency for each channel . Fig . 1 illustrates the overall scheme of measuring the spatial frequency responses of channels in various convolution layers of the trained VGG-16 network . The reason we are using a concentric pattern is to eliminate the factor of orientation selectivity from this part of the analysis . Some concentric grating stimulus patterns are shown as input to the trained VGG-16 network in Fig . 2. ( a ) . To quantify orientation selectivity at low contrast masking thresholds , we stimulate the pre-trained network with linear pattern sinusoidal gratings with different orientations . The gratings have a fixed spatial frequency , which corresponds to the peak of the Contrast Sensitivity Function ( CSF ) . Some sample grating patterns are shown in Fig . 1 . Sample observations of orientation selectivity for channels in different layers of the pre-trained VGG-16 are shown in Fig . 2 . ( b ) . 4.2 VISUAL FREQUENCY SENSITIVITY . In this section , we will use the spatial frequency selectivity quantification in section . 4.1 to introduce the concept of visual frequency sensitivity . Human perception of images is largely dependent on attributes of input stimulus . A significant proportion of neuro-science research advocates the role of the early visual cortex as a spatial frequency analyzer ( Maffei & Fiorentini ( 1973 ) ) . Human perception characteristics are dependent on spatial frequency and one of the most significant models that quantifies this characteristic is called the Contrast Sensitivity Function ( CSF ) . The CSF depicts human ability to perceive contrast changes as a function of spatial frequency . The spatial frequencies where the CSF has a higher value , correspond to lower contrast masking thresholds in perception ( ease in perceiving distortions and contrast changes ) . In essence , this corresponds to a higher probability of perceiving distortions at high CSF valued spatial frequencies . Considering the presented analysis on the spatial frequency selective behavior of deep feature maps . Our hypothesis is that the deep representations that are more sensitive to high CSF valued spatial frequencies , can be better features of perceptual quality . Fig . 3 plots mean activation of two channels versus spatial frequency of the input grating . Feature Map-2 can be seen to have a higher sensitivity compared to Feature Map-1 at high CSF valued spatial frequencies , making Feature Map-2 more sensitive to distortions corresponding to low contrast masking threshold regions in input images . We model this attribute quantitatively as µ1 defined in Eq . 2 µ1 ( k , m ) = ∑ f CSF ( f ) . ∣∣∣∂akm ∂f ∣∣∣ ( 2 ) where ’ k ’ is the index for the convolution layer , ’ m ’ is the feature map index in each convolution layer , ’ CSF ’ is the contrast sensitivity function ( CSF ) , ’ a ’ is the mean activation of the feature map and ’ f ’ is the spatial frequency in cycles per degree . µ1 quantifies the average sensitivity of a CNN channel weighted by the CSF over different spatial frequencies . The channels having higher µ1 values should deliver better perceptual features according to our hypothesis , because they can be more sensitive to visually perceivable distortions in input images .

Summary: This paper proposes an analysis of convolutional neural networks (CNNs) features the basis for making perceptual quality comparisons. The analysis is based on the proposed Perceptual Efficacy (PE) Score that measures spatial frequency and orientation selectivity of CNN features. The hypothesis put forward by the authors is that a CNN features with high PE score can be used to formulate a perceptual loss (Eq. 1) that correlates well with human image quality judgement. The authors use a dataset of human image quality judgements to assess their hypothesis.